Introduction
I recently read an excellent paper from 2009 by William Noble of the University of Washington, shared online by Chirag Gupta on advice for bioinformatics students on organising their research. Many have noted that working with LLMs and agents is more akin to biology than mathematics or physics, and I was struck by how much this computational biology article was relevant to working with LLMs in industry. I noted too that almost all of the advice is still relevant over 15 years later.
The paper can be found here, but I will pick out some key points and add some of my thoughts.
Organisation, Documentation, and Reproducibility
Good processes will speed up your work and can be a breakthrough all on their own.
A core idea is that your work should be organised in such a way that some future person should be able to look at your work and very quickly understand it. This is especially important in industry where you may not publicly disseminate and publish your work, but co-workers or your possible future replacement may need to build upon it. More important still - that future person may be you in a few months. The same idea holds for commenting code well.
Everything you do, you will need to re-do, either for publication or production. Science and engineering at the forefront of technology unfolds in a messy way and you will very likely have to tidy up work to make it useful to others. This is especially true for experiments on agents, as their behaviour can display unexpected behaviours with even small changes in input, and very complex behaviour can emerge during repeated tool use. Documenting and organising work clearly makes the process of tidying it up easier.
File structure is a powerful organisational tool. Engineers do this well for their own purposes in software development. Scientists need something a little different. Noble suggests that the project name should be at the root of the structure, and the next level should be the standard categories such as “code”, “data”, “plots”, “results”. Below that, folders for experiments should be named chronologically, as the content and structure of an experiment for a given project can change dramatically over time. Organising this chronologically helps to position the reader in time and to easily find the most recent experiment, which we all hope is the most well developed. I have slowly converged on this method myself too. Below this chronological level, folders and files can be organised as appropriate.
Noble makes a big point of keeping a good lab notebook. This may not be common practice among those from a maths or physics background, and many seem to abandon this practice either after high school or once they finish with lab modules. Biologists and chemists tend to adhere to this practice much more rigorously. I found it much more useful to take more detailed chronological notes on experiments when I started working on models of biological neural networks in my postdoc, and I find the same is true for working with LLM-based agents. Good results often require many attempts at finicky experiments and well maintained notes are essential for making sense of them, which was not so much the case when working on more traditional mechanistic mathematical models.
Furthermore, writing is thinking, and in this age, when standards of literacy are declining, students are using ChatGPT to write essays, managers are using it to write emails, and absolutely everyone is using it to write LinkedIn posts, the clarity of thought and insight that one gains from writing can be invaluable, and will no doubt help those who do write to stand out in the future.
Noble is quite prescriptive about what should go in a lab notebook, and his advice is similar to what many learn in high school - aims, methods, observations, results, and ideas for future work. He also advises one to do this for failed experiments too - good advice, even if it is annoying to do in practice.
The paper spends some time discussing the use of version control. Anyone working in industry will no doubt be familiar with using Github or similar and will likely not need to be convinced of its merits. Even if they do, engineers or managers will shame them into using it anyway. Noble also spends some time sketching out how to organise a single experiment and suggests some design patterns for code. I would highly recommend the interested reader to go through the rest of the paper, no matter what stage of their career.