#1 - Tape everything down.

Nothing is worse than ripping off connectors and wires, creating short circuits because your table is metal, having oscilloscope probes fling across the board touching random pins. Tape everything down to a piece of cardboard. Go even crazier and create custom rigs with MDF, additional plugs, buttons, switches, relays, lightbulbs.

#2 Get a scope

Having an oscilloscope completely changes the way you approach a bug / hardware design. I bought mine in 2003 and have never needed another one since, but I know there are plenty cheap and portable ones out there these days. I do have a small saelae probe that will do in a pinch, and is useful for digital communication problems, but it doesn't replace a real scope. You want to put a couple of probes here and there, turn 2 buttons, and get to look at your system behaving "for real". If you are serious about embedded, get a scope that fits like a glove. It's the same as getting a good chef's knife, you don't want a tool that irritates you.

#3 Learn gdb commands

In gdb, you can set breakpoints and watchpoints. Then, you can attach command sequences to each breakpoint watchpoints, kind of like a printf debugging that doesn't require recompile/reflashing. The downside is that it often completely messes up the timing of your firmware, depending on the chip / probe / debug protocol combination.

For example:

break main.rs:30
commands 1
print x
print y
c 
end

will add a breakpoint at main.rs:30, and then print out x, y before continuining.

#4 - Be creative in your "debugging input/output" channels

You don't need to just do printf debugging over the debugging interface or the serial interface. You can toggle GPIO pins, use LEDs. Other creative ideas I've used in the past:

• Rotate a galvo to show how much memory is full and attach a little gauge to it.
• Output MIDI notes on a scale because the serial interface was used for MIDI -> if the melody changes, something weird happened.
• Attach a piezo speaker and use variations in pitch to recognize timing issues / jitter. In general using audio for debugging is great because it frees up your eyes to look at logs or at the scope.
• Buy or build a collection of "physical probes", devices you can attach to a 3V / 5V pin that real world actions. For example, a solenoid that can be used to hit a bell, flip a button or slap you in the face when a certain debug pin is set.
• Lego Mindstorms is great to build little contraptions that control your device, for example press buttons or move sliders around in an automated fashion.