🐛 10 Debugging Tips for embedded and distributed systems 👇
Level up your printf game and keep an actual event log. This will not tell you what the bug is, but where it is.
Make logging frictionless. Make verbosity level configurable and make it easy to quickly toggle whole sections on and off.
Log structured data and make it queryable. Bonus point for being able to reproduce the bug using the log itself.
Think laterally and leverage all senses. I love using my peripheral vision or my ears. Trigger visual events and sounds as an alternative to printf.
Keep your code testable. Writing a unit test shouldn't require refactoring and writing mocks for 3h.
Write down what you think is going on in plain English. I call it rubber-writing.
Grab a colleague and a whiteboard and explain the bug to them. Speaking is different than writing.
Draw what you think is causing the bug. Use timing diagrams, causal diagrams, and class diagrams. Use colors and stickers. Write on top of previous diagrams as patterns emerge. A drawing is worth 1000 log lines.
Record your debugging session with a screen recording tool. You'll thank me if your bug hunt spans more than a single session.
Think hard and then take a walk.