Testing Smarter, Not Harder: Using LLMs for Better Unit Tests
(This is part of a longer series of short essays I'm writing as part of a marketing writing course. This is the first article I think is meaty enough to be published here. I will rework most of my material once the course is over.)
Unit tests enable bold refactors and protect against subtle errors introduced by Large Language Models. However, while unit testing strengthens codebases for the long haul, they drain developer bandwidth with rote work.
In this article, we'll explore tactics to leverage LLMs as a unit testing sidekick - taking care of the grunt work so you can focus on the good stuff.
Craft test blueprints
Following the funnel method, we leverage LLMs in crafting unit tests by transforming the code we want into an actionable unit testing plan.
- Summarize the code as bullet points, to ensure the language model understands what needs to be tested. Experiment with providing only your code's API (transcript) or its actual implementation (transcript).
- Prompt the LLM to look for edge cases, common scenarios, complex combinations of parameters. Don't hesitate to provide your own edge cases too. Don't forget to regenerate often (transcript).
- Don't have the model output any code at this point. We are actually interested in transforming the formal input into something higher up the language ladder, as it provides denser information in less tokens.
Once you have a testing plan that you are happy with, it is time to take a step back (or a short walk around the block) and make sure that you have taken everything into account.
Reviewing the test blueprints
Reviewing the test blueprint is pivotal to ensure it captures the essence and intricacies of your target code. This blueprint will be used for actual code generation; it is crucial to ensure it is concise, correct and exhaustive.
- Begin by manually scrutinizing each test, checking that it indeed makes sense. Don't hesitate to manually edit the blueprint.
- When faced with complex tests, don't hesitate to ask the LLM for clarifications on their purpose and expected outcomes; this not only aids comprehension but also doubles as invaluable documentation. It will also enhance the LLM's ability to generate proper code (transcript).
Ready to bring the blueprint to life? Dive into the next section: creating a testing skeleton.
Implementing the tests
After crafting a strong list of unit tests, it's time to let the model write the actual code.
- Set a consistent style by generating a list of unit test signatures and brief descriptions (remember how self-attention works).
- Guide the model by providing a template example that shows which framework, checks, and mock objects to use. This will ensure the upcoming tests align with existing code. (It's often beneficial to have the model produce table-driven tests for uniformity).
- Always ask for clear error messages and possible reasons for test failure, making failed tests easy to decode.
- Don't ask the model to implement all tests at once. Instead, iterate through the signatures and use the same template each time to ensure consistency.
- Make sure the template implementation is spotless. This will improve the quality of future implementations (transcript, note the LLM-esque off by one in the key iteration).
With straightforward tests and clear error info, there's no need to spend hours sifting through test code when issues arise.
The future of unit testing
LLMs' convenience means that even when faced with tight deadlines, there's always time for thorough tests: a short investment of 10-15 minutes can ensure foundational coverage. This makes me pretty confident saying that large language models will revolutionize unit testing.
Remember, the aim isn't to let LLMs fully take the reins, but to utilize them as invaluable colleagues.