For the last year or so I’ve been working on building a software application to help marketers allocate their marketing spend. This software is statistics and data-science powered and my partner and I have spent more hours than I’d like to admit struggling to squash bugs, achieve model convergence, and generally answer the question “why on earth could that be happening?” In this post I’ll discuss the history of the lab book and how it’s used generally before discussing how to use it for data science and software engineering projects and providing an example lab book template.

The tools and techniques of data science and advanced analytics can be used to solve many problems. In some cases – self-driving cars, face recognition, machine translation – those technologies make tasks possible that previously were impossible to automate. That is an amazing, transformative accomplishment. But I want to sing a paean to a mundane but important aspect of data science – the ability to intelligently put lists of things in a better order. For many organizations, once you have found some insights, and are into the realm of putting data products into production, the most substantial value can be found by identifying inefficient processes and making them efficient. Twenty or thirty years ago, that efficiency-gain might have been addressed by converting a paper-based process to a computer-based process. But now, prioritization – putting things in the right order – can be what it takes to make an impact.