the scapegoat dev

I want my software to be visionary - the go go golems ecosystem

https://s.mj.run/c5n61SnhpX8 https://s.mj.run/Vzq5-c2jnAs retro mainframe flower zombie head, flower fountain retro mainframe, exposed electronics, tentacular book shapes, extremely detailed, white on black background, dark background, night scene, ink and brush watercolor by moebius, dark black and white watercolor, watercolor ink drawing by junji ito, trending on artstation, highly detailed, 4k

Last year, informed by 2 decades of creating command-line tools, I wrote 14 great tips to make amazing CLI applications. I did not have anything concrete to show, all these concepts having been built in private codebases and having stopped writing open-source.

In October, I set out to implement all these concepts in a go library called glazed. While I initially wanted to target Rust, I felt too frustrated working on it in my after-hours, and was missing the consistent asynchronous code writing experience go was giving me. Plus, charmbracelet just has too much cool stuff.

This blog post started out as a level-headed overview of the concepts underlying glazed, and turned into, well... this.

Opinionated, visionary software development

Initially,¬†glazed¬†was meant to only implement a data export layer. I had spent the summer taking a sabbatical trying to put my thoughts down on paper,¬†aiming¬†for a manifesto-style of writing. It is easy to complain about the state of data tooling, the lack of interoperation, the vision of what could be‚ÄĒeverybody does it. How many times did I hear people dismiss what they do as just "Software is just glueing APIs together," as if that wasn't miraculously¬†challenging?¬†It is one thing to complain and argue about¬†perceived problems,¬†it is another thing¬†to actually¬†come up with and build a solution.¬†Advice obviously meant for¬†me.

Instead of writing more big bold unsubstantiated statements in my diary, I started focusing on designing and building something that would be eminently practical, yet absolutely stringent, coherent and visionary in its goals.

I wish more software was ambitious, utopian, bold, breaking free of the confines of what we have come to accept as whatsoftware should be. I am growing increasingly frustrated by the corporate shaped, influencer inspired, bland github-backed open-source software; by the boring-ass markdown rendering of its wikis, the lack of personality, the endless controversies about market share of individual frameworks; by the arguing about squash merges and test hygiene and CICD and package management and code styles and linting and this language and that language, event-driven microservices distributed cloud column database analytics datalake software architecture secret management infrastructure as code.

As software developers,¬†we create¬†machines out of thin air.¬†Every time we run our code, a golem is set into¬†motion. Wethink of something, tell¬†sand¬†(literally a golem) to execute it and impact the lives of other people. We are making magic happen every time we sit down, yet we are mentally stuck thinking in terms of "backend", "full-stack", "embedded", "machine-learning", "gamedev", "mobile", "UX"‚ÄĒas if we were little checkboxes on a resume; our¬†brains are just¬†Lego blocks that someone else designed, only to be assembled according to Google's latest take on software engineering. We are lured by the fully enshittified corporate grip on public discourse into thinking that arguing for react vs raw Javascript is¬†a crucial¬†fight to fight.

We assume that practicing agile, being a "product thinker", building workflows to enable cross-team collaboration, becoming a staff engineer, speaking at conferences to develop our brands, blogging because writing is thinking and productivity is our actual value in life. We are dedicating the best years of our lives, of our days to dehumanizing concepts that pretend to be about self-actualization. We have come to believe that team lead product lead individual contributor senior junior staff principal CTO being glue mentor are valid words to define how we relate to each other, and that these relationships is how software has to be built.

What the actual fuck? Why are we doing this? Why do we assume that one should strive to "become a product-minded, T-shaped, systems-thinking engineer?"

We can make our dreams come true with a few kilobytes of ASCII text; we can conjure virtual worlds and change the world. Software our brethren and ancestors built allows us to talk to like-minded people on the other side of the world and robots exploring other planets and comets; it allows us to watch videos from another age, browse museums' online collections and enjoy the works of art of millions. And yet, Mathematica seems to somehow be the only tool left with a vision that personal computers' primary goal can also be computing.

Anyway...

Exploring thoughts with software

GO GO GOLEMS' library collection is my exploration of concepts that have been maturing in my wetware for some time now. The only way I know to properly identify what these concepts are about is to turn them into working code and solve real problems. Similar to writing for clarifying one's thoughts, coding is a programmer's way to clarify what they actually want to say. For me, this means writing multiple iterations of something, not being afraid to throw everything away and starting from scratch (with varying levels of keeping backwards compatibility, approaching the endeavour as a seamless refactoring).

While I do write this software for an audience, it is unclear how I will approach showing this to the world. I have a tendency to break APIs if I feel that something is not right. I don't think I'll be able to cope with third-party input beyond discussing and arguing concepts. I want to relentlessly refine the vision and make no compromises, so that the end-result ultimately benefits a wide range of users. I think that having to accommodate "legacy" too early might be to the detriment of my endeavour.

As such, while the software is open-source and you are free to do with it whatever you want (it is all MIT licensed), be aware that I'll most certainly break APIs about 5 times a day.

glazed's approach to data

The core idea around glazed is that applications, as simple as they might be, have a rich representation of the data they manipulate. From the simplest struct, the most menial SQL query, the barest array all the way to a rich swagger specification, a full blown ORM schema, graph database, the tiniest command-line tool knows a lot about the business domain of the application. Yet most programs then use some variant of printf to export that data. It is then up to the user to figure out how to parse it; to infer the now lost structure; to browse through endless indigest lines of structured logs.

At its core, glazed provides a Table and Row interface (they are absolutely terrible, being the absolute first draft, but they got the job done until now), a ObjectMiddleware and a TableMiddleware interface, and an OutputFormatter (intimately and confusingly mixed with something called GlazeProcessor) interface.

The idea is that an application throws objects (map[string]interface{}, a generic string-indexed hash-map) into the OutputFormatter, where they get processed first by the string of ObjectMiddlewares (which operate at the object level, allowing one to rename/flatten/filter/replace keys). The object is then added to the OutputFormatter's Table. Once all the data has been processed, the OutputFormatter processes its internal Table through its TableMiddlewares and finally outputs it.

TableMiddleware are again different variants of filter/rename/replace/template, but operate at a Table (and usually at a Row level). The current set of OutputFormatters are: table (CSV/human-readable tables/HTML/Markdown), json, yaml, excel and template (go templates). Planned are many more (sqlite with varying degrees of table normalization configuration, rdf, hdf5, pandas.dataframe, etc...).

a diagram of the objects going through a chain of ObjectMiddleware, then into a Table, then into a chain of TableMiddleware, then into an OutputFormatter, and then into CSV/JSON/YAML/Excel/Markdown/Text

All the middlewares' and OutputFormatters' options can be configured programmatically, through config files or through command-line arguments (currently, the viper and cobra libraries are supported, although the system is built with genericity in mind). The main concept however is that as a developer, it should take you at most 10 lines of code to add glazed to your application.

I created the glaze application to not just showcase how to use the library, but as a tool that can be used to quickly convert and modify structured data, as a universal Swiss-army tool. There are already many of those, but I hope that glaze brings something unique to the table.

glazed's approach to help

For the rich data an application manages to provide useful information to a user, the user needs to know its structure. This is often a major hurdle, both for the developer, who needs to write and maintain that documentation, and for the user, who wants to find, understand and act upon said documentation. What we often end up with is a set of automatically generated docpages in a wiki that no-one knows how to find, as well as cryptic command line flags.

I am deeply in love with Wolfram Mathematica's helper system. The software is worth it for that system alone. Each function is described in detail, in an appealing format. More importantly, each line of the documentation is directly actionable and the documentation itself is an absolute blast to play with. Besides giving examples for each function and its arguments, each function gives a list of practical applications, often involving a few other constructs, to provide actual real-world examples, as well as a section of "fun applications" which try to show some wild, fun, mind-bending, ingenious ways to use the function itself. Furthermore, the documentation comes with topical pages that give a narrative, structured overview of the different domains of the application. These range from simple index pages to entire tutorials and treatises on certain topics.

The major strength of using go or rust, vs for example python, is that command-line applications are usually self-contained. Every hurdle to an applications use will hinder their widespread use, and fighting with pyenv and anaconda is one of the world's most tragic hurdles. The strength of a self-contained, well-documented command-line application is that it makes its functionality "discoverable." A user should be able to run --help and discover the full functionality the application provides (I always was a big fan of GNU software bundling not just manpages, but entire hyperlinked manuals in its texinfo format).

As I was building glazed, I stumbled upon the go:embed functionality, which allows one to embed entire "filesystems" into the binary. I built a HelpSystem class that allows one to embed Sections parsed out of additional markdown files directly into the binary, as well as a rich help command that enables the user to browse these files.

A Section can be one of:

  • Application - real-world¬†use cases¬†of command-line¬†applications

  • Examples - short, concise, copy-pastable examples of how to use the applications for different purposes

  • Topics - help pages about general topics around the application

  • Tutorials - step-by-step, self-contained tutorials exploring a specific topic

These pages can be linked to specific verbs and flags of the application, so that they can be found through the --help flag for a specific verb, or by searching the help system through help --flag or help --verb. Each section can further be tagged. This metadata is provided as a YAML preamble.

Here is the documentation file for the help system itself (a Topic section).

Each section can be marked as shownByDefault, which means that it will be listed each time the usage information of a verb is shown or the help command for a matching set of tags/flags/verbs/topics is shown.

Further steps

I have since built 5 real world applications I use daily on top of glazed, and identified and implemented many more concepts. To not overburden this blog post, I will split these out into follow-up posts, covering my tools sqleton, escuse-me, pinocchio, parka, plunger, flour and whatever I'll decide is worth building until then.

- 13 toasts