#11: Approaches to Software Engineering that Worked For Me
I have now been writing software, reasonably consistently, for about eleven years — from my second year of college to the present day. Along the way, there are a few things that I’ve done differently from others, or that have otherwise been helpful to me on my journey. I thought they might be worth sharing.
Starting: do what you enjoy and let time compound
I took my first computer science class in the fall of my second year in college, mostly on a whim. I was focused on the humanities at the time. The setting was intimidating: many students in my intro class were very smart and had been writing software for years already. As a 19-year-old, I thought I was irrecoverably far behind the other students that had been programming since age 10 or 11.
But I didn’t let that discourage me. I enjoyed computer science. I wasn’t going to be the best, but I was having fun and that’s all that mattered. As I progressed, I found that the curriculum was a great equalizer. The prodigious students with a leg up on day one were just as challenged as anyone else by the advanced courses, where they couldn’t rely on prior familiarity.
I had a similar experience when I moved to San Francisco to become a software engineer. My college computer science curriculum was heavily theoretical and not about writing software, so as an entry-level engineer, I was once more in a pool of peers that had been at it for much longer than me. I hadn’t even done a SWE internship. But again, time was the great equalizer: after a year or two of steadily keeping at it (and writing software during many of my off-hours because I enjoyed it), I had caught up with those peers who previously seemed so much more advanced. If you enjoy doing something, it makes it easy to do it consistently, and then the passage of time will take care of the rest with immense compounding force.
Learning: always read the entire manual
As a young engineer, I was doing new and unfamiliar things all the time. Oftentimes that meant following a tutorial that someone had published. Stitching together software by following StackOverflow and Medium blogposts is common, but I quickly realized that was actually an anti-pattern.
The reason is that your capability for building high-quality software is usually not determined by what you know, but by the gaps in your knowledge. You can build fairly complicated web applications by just following tutorials, but if you don’t actually know how it works, then you’re in trouble in the long run. Gaps in knowledge compound, and it’s not viable to build on foundations riddled with holes.
I distinctly remember a day that I was asked to build out a new product on AWS. I had barely ever used AWS before. I expected that if I just built it on the fly as I learned, I would step into all sorts of pitfalls. So I bought a textbook, Amazon Web Services in Action, and read it cover-to-cover, roughly 430 pages, over the weekend.
This was a transformative experience for me. It turned out that the textbook was very good at conveying everything I needed to know, in depth and in logical sequence.1 It was a much more effective use of time than if I had spent that same weekend trying to stitch together a bunch of tutorials and examples. While it’s always easy to just Google the new thing and impatiently try to implement the first tutorial, it’s far more effective to take the time to study the technology in detail, cover all the gaps, and then proceed. This has been my approach for every major new technology that I have learned since,2 and it has served me well.
Building: write manuals for yourself
As I got deeper into work as an engineer, I noticed that there were certain things that I had to do frequently where the exact commands or operations were hard to remember, or that weren’t well-documented. Every engineer has these routines that are bespoke to their work or stack: if this API call is slower than usual, SSH into this instance, pull this credential, connect to this secondary service, run this specific command to check for a blocking query, then run this command to remove it…
Some routines are more general, like for setting up a new machine with a development environment, or for standing the stack for an entire new web application. There’s enough detail that I don’t want to improvise or rely purely on memory: imagine if you’re breezing through a routine Redis installation and you accidentally leave the default configuration as open to the internet...3
So, I started writing manuals for myself every time I did anything with any amount of complexity, like an infrastructure setup or a performance improvement. I refer to them all the time, and update them as I go.
My collection of manuals-for-me is now one of my prized possessions. Engineering is a craft, and all craftsmen accumulate, over the course of their careers, a set of tools and tricks they rely on. Writing reference manuals for yourself, customized to the nature of your work and how you think, will save you many hours over the long term.
Architecture: boring choices, deep expertise
Many engineers are tinkerers who love to play with new technology. Me? Not really. I’m mostly focused on solving real-world problems and using technology as a means to that end. Cutting to the finish line usually means using reliable, boring, general-purpose tools. There are two components to that:
Understanding how to use the technology correctly and drive high performance;
Understanding the cost-to-capability curve of scaling it.
It is common for engineers using technologies they don’t deeply understand to run into scaling issues, and then make architectural jumps, like switching out their database architecture. Such jumps are often premature, while avoiding the actual thing to do — using the technology correctly. The performance difference between a poorly- and a well-configured Postgres database is many orders of magnitude. Fewer tools understood well will take you much further than many tools understood poorly.4
A variant of this — and this one’s more pernicious, because it particularly catches experienced engineers — is that cloud compute costs are coming down so quickly that oftentimes the correct scaling move is to pay a little more money for a beefier instance, even when ten years ago the correct move would’ve been an architectural migration. Very few engineers internalize the current cost-scaling curves for major cloud services,5 and it turns out they’re often favorable: look at Figma, scaling to a multi-billion-dollar valuation on one big database. You can drive an enormous amount of value on a simple stack if you have deep expertise in your tools of choice.
Interplay
My three principles for learning, building, and architecture all play off one another. If you’re always reading the entire manual, then you will have deep expertise in your tools. If you have the expertise to use your tools effectively, then you usually don’t need many tools. If you’re writing manuals for yourself and always using those same tools, then you accumulate software playbooks that you can execute over and over again, making you faster and better at Getting Things Done.
You might argue: wasn’t most of the book a waste of time? Covering AWS products that I wouldn’t need to know, that I would not touch on any foreseeable timeframe? Well, sure, but it doesn’t take long to get through a chapter, and reading it is still useful that it conveys adjacent information or re-emphasizes broader points. After reading about many AWS products, you will know extremely well what a VPC or an Availability Zone does. And you’ll have a broad lay-of-the-land that’s useful when you consider using additional products.
You might argue: isn’t this inefficient? If you do this for every technology, then it’ll take forever to do anything! Well, this touches the next section of the post: learn fewer technologies in greater depth. How big is your stack? By my count, I use seven major technologies (Python, React, Javascript, AWS, Postgres, Redis, Cloudflare). If I’m spending two or three days of study every two years, that’s fine. The real trap is in hopping between lots of different languages and frameworks all the time.
Historically, one of the classic pitfalls in web development was in setting up Redis, a popular in-memory NoSQL database. Redis’ default installation had two very dangerous defaults:
It would expose itself to the public internet by listening on certain ports;
It had no password.
This meant that lots of developers who installed Redis and forgot to configure Redis would have a gaping security vulnerability, exposing potentially sensitive data and providing a way for a hacker to compromise the entire system. At some point the Redis team improved the default configuration, so this is now a deprecated example, but the point still stands.
Importantly, understanding your tools well also means that you’ll know better when not to use one of your tools. Developing select expertise should not mean to treat every problem as a nail for your one hammer.
Or roughly, “what level of performance/usage should cost how many dollars.”