My creative software design process
One of the things I'm happiest about since slowly recovering from burnout is leveraging my slightly manic creative energy and obsessive note taking to do software design. I spend a lot of time in grasshopper-mode, bouncing from idea to idea, weaving threads together, writing everything down in my sketchbooks, trying to capture as much details as I can so that I will be able to retrace my steps. I usually fill one 60 page sketchbook every month or so. This habit came from when I was learning how to draw, where I kept sketchbooks both for my sketches and for various notes about the engineering I was doing.
Later on (often months later), I'll review these sketches and try to extract actual design patterns and software features that are directly related to what I want to build. Again, I document everything precisely, although I'll usually try to do this work digitally in my obsidian vault.
After filling a sketchbook page (the spread I'm sharing is from June 3rd), I'll sit down during my daily/weekly coffee shop session and start annotating it with a different colored pen. I'll go over the old notes, make new ones and connecting things with pretty arrows. I'll start transposing these ideas into the relevant notes in my vault, often creating new ones in the process. Once I feel that I am through making a filled-out square once I've properly digitized the analog content.
As part of my obsidian workflow, which I modify as things get messy, but is always optimized for fun, I started keeping index notes for sketchbooks (I used to keep indexes on the front cover (a technique inspired by Lion Kimbro, but never ended up really using them). I create a new note for each sketchbook and cross-link all the relevant subjects and notes to their individual pages.
For software design and project ideas, I used to use Markdown files (trying to follow the RFC workflow I was using professionally) and GitHub issues, but it felt too disconnected. Now I do the work in my Obsidian vault. I usually start with my daily note and refactor my rambling writing into individual notes. These notes will then later on be linked from the relevant Github issues. That way I can trace a freature from GitHub pull request to Obsidian notes all the way back to the original analog sketchbook.
Example: brainstorming a LLM context manager
Today's spread is about building a prompt context manager for my LLM framework Geppetto (something I've already made a really rough stand-in proxy for, and have already been iterating on since probably february). I first created a dedicated "IDEA" note to capture the brainstorming digitally. I go over the notes from june, group them into coherent units (this is where the abundance of notes really helps, there's nothing worse than just having a few keywords such as "pipeline templates" with no context). I transpose these ideas into the vault, linking them to existing ideas, potentially separating them out into their own projects, creating Zettelkasten claims, adding literature references.
In today's session I ended up gathering over a dozen context manager ideas, 3 new tool ideas, features for existing projects, and refined a few maturing concepts, all from this one sketchbook page.
Here's the final annotated page:
What's the end goal?
Will I build all this? Hopefully, but not as they are now. I constantly rework these ideas with the aim to unify them, to create simpler, denser abstractions that enable them to be built almost trivially.
As you can see in my vault and GitHub, hundreds if not thousands of hours of thinking have gone into this already, aside from writing code. It's really fun, and the emerging ecosystem lets me do cool stuff with just a few hours of coding. It needs patience and accepting that I'll never be able to work through every thought, as worthwhile as it might be. I've already spent around 10 hours digitizing this sketchbook, and I'm only on page 8 of 60!
The process produces a constant stream of ideas I can incorporate into my growing personal knowledge management system and software projects. It is a lot of work but leveraging my creative energy in this way has been hugely beneficial for recovering from burnout (and hopefully creating some useful software).