Do the Work
One of the great pleasures of being a programmer is the ability to automate boring, repetitive tasks. But what if I told you that holding off on automation might be the key to leveling up your development productivity? Watch this episode to find out why doing the work by hand can be a counter-intuitive developer super-power.
Video transcript & code
Let's say you find yourself in the following situation:
You're writing an ebook. You have a set of files in a specialized source format, each one representing a section of the book. You need to convert each file to HTML, and then publish the HTML bodies as pages on a WordPress-based website.
If you're anything like me, you might say "Ah! This is an easy task to automate!" And then you'd sit down and start banging out an automation script. Maybe you write it in the form of a Rakefile, since Rake excels at building source files into finished products.
Through trial and error you work out the code for automating conversions from the source format to HTML. Then you start in on the code to push the HTML content up to the WordPress site.
But once you have the page upload working, you discover that the HTML that gets produced from the ebook source is a little bit eccentric. Not Microsoft Word-levels of tag soup. But there are more divs than you'd like, and code samples aren't marked up the way you'd like. Little stuff like that.
But that's OK. There's no problem that a little more code won't fix. You dig in, and write some HTML munging code to fix-up the HTML into exactly the style you need.
After a while you've created a long chain of HTML transformations, at the end of which you have beautifully simple HTML code that looks perfect on the WordPress site. Mission accomplished.
What I just described is a pretty typical couple of days in my life. I'm guessing that even if you've never assembled an ebook, you've probably tackled similar automation tasks in the past.
Now I want to re-wind this story a bit. Imagine you were presented with the same task… but you decided that before writing any automation code, you were going to publish the first few pages completely manually. So, you trigger the HTML export interactively. And then you paste the output HTML into a WordPress page editor.
At this point, you discover the issues with the generated HTML tagging. And you start to make manual edits to the code.
Pretty soon, you start thinking that this is awfully tedious and annoying. You start wondering if there's some way to get the tool to generate cleaner HTML in the first place.
Then you start thinking about some of the other formats that the authoring tool can generate.
And you remember… didn't you read somewhere that WordPress can be configured to accept Markdown-formatted input instead of HTML?
Anything's better than editing more HTML, so you poke around a little bit. And sure enough, you discover the setting in the Jetpack plugin to enable Markdown.
You dump one of the book sections to Markdown, paste the resulting text into WordPress, and the result looks great on the first try.
Eventually, you do write a script to automate all the steps in a repeatable way. But it turns out to be a substantially shorter script, because since you switched to using Markdown as an interchange format, there's a whole set of HTML transformations you never had to write.
By manually doing the work, and feeling the frustration first-hand, you were pushed to discover a substantial shortcut to the finished product.
Now let's imagine another scenario.
You've got a CSV dump of some sales data from an ecommerce app. You need to modify the CSV into a format that's suitable for import into a different ecommerce app.
Immediately, you think: "Aha! This will be easy and fun to do in Ruby, using the CSV library!"
But you remember your experience with the ebook publishing, and you decide to do the first batch manually.
Using a spreadsheet program, you get the edits done surprisingly quickly, and get on with your day.
When it comes time to do the next batch, you're feeling a little too busy to write code for it, and anyway the last time around the manual editing didn't take long. So you do it manually again.
The third time around… never arrives. It turns out that the circumstances that required moving information from one app to another were an isolated event.
You realize something: you probably spent less time doing the process manually twice over, then you would have spent to tweak a script until it worked just right.
Now let's imagine one more scenario.
You sell a service, and you're running a special bundle promotion with another company. The other company is handling sales, and they are pinging a webhook on your site every time a sale completes.
This is an obvious opportunity for automation. But, remembering the lessons of the past, you decide to process the first few purchases manually in order to get a better feel for what exactly needs to be automated.
As you're processing purchases, you quickly notice something: 90% of them are for individuals, and are relatively simple to process.
But 10% of the purchases are for teams, and those require substantially more setup.
If you had started writing automation right at the start, you probably wouldn't have even blinked before diving into the special-case code required for team purchases. After all, it's just one more requirement for completeness.
But having experienced the purchase fulfillment process manually, and having seen the breakdown of individual vs. team purchases, you realize that in order to correctly handle every kind of purchase, you're probably going to be spending a huge amount of your coding time on handling a small minority of the incoming purchases.
Instead, you make a pragmatic decision: the automation that you write will fully handle all individual purchases. But when it detects a team purchase, it will create a new task in your team project management app. Then you'll either handle those relatively few tasks yourself, or delegate them to a customer service representative.
Once again, taking time to work the problem manually at first has paid off in terms of overall productivity.
Here's the thing:
I'm a programmer. You're a programmer. As programmers, when we see a problem that involves any kind of tedious, repetitive labor, our natural reaction is to think: I can automate this!
Now we have two problems.
Don't get me wrong: When we're first getting started with programming, the ability to automate repetitive actions is a super-power. It's a force-multiplier.
But as more experienced programmers, the ability to refrain from automating a task can also be a super-power.
When we hold off on writing automation, and instead force ourselves to do the work manually, we can experience a number of benefits:
- It gives us a chance to discover shortcuts before we code our way into a sunk-cost fallacy: that state where we feel like we've spent so much time coding the long way around, we might as well keep going.
- We may discover that a rare or one-time task is so easy to do manually, or so easy to delegate, that it's not worth taking the extra time to automate it.
- When we get our hands dirty, we often discover surprising and non-obvious truths about which cases are quick and easy to handle, and which cases take a lot of extra effort. Learning about these distinctions beforehand can help us make some pragmatic decisions for the automations we eventually do write.
There are two more huge advantage of doing the work that we didn't really talk about directly in any of these scenarios, but they're relevant to all of them.
The fourth benefit is that it is far easier to learn about a problem when we're tackling it directly and manually, then when we're trying to learn about while also writing automation for it at the same time. And when we finally do get around to writing automation code, the process goes much, much faster as a result of our first doing the work "by hand". This is doubly true if we were careful to take notes while working the problem manually.
So, next time you're presented with a tedious problem that you know you could easily automate, bite the bullet and do the work manually a few times before writing any automation. Chances are, you'll thank yourself later.