world of software
Pitfalls of automation to avoid. a lesson from the past
“Anything you do more than twice has to be automated.” This sounds like a great quote. carefully. Automation is more expensive than just the process of automating it.
This is speaking from a real case which I am sharing below.
In the early days of our mobile development there was a piece of code that was relatively repetitive. It is creating a SQL table and performing a set of operations such as insert, update, delete, etc., and connecting to the data model for the table.
Every time a table is created, the table contents and their SQL commands go through the create and duplicate functions.
Repetitive process identified
As we start development, more requests are coming in to store new data, and so there are likely to be more new tables and SQL to program.
It is very challenging.
Automating It—Not As Hard As We Think
So, instead of copying and modifying it manually, we create a script that, with minimal input, generates all the boilerplate code we need. It works well and is also included in our CI process.
Instead of duplicate code, we now have some fixed data to provide to the automation script, which will generate all the necessary code in Java for our mobile app to use.
oh we need some revisions
After creating about four to five data tables, the need to create more tables stopped as more functionality focused on increasing the functionality of the app rather than inserting more data into it.
Later, as we progress, we realize that we need to modify some of our SQL functions. The changes are minimal but only apply to a few tables.
Modifying the automation script for that custom change is difficult and expensive. The script is written in another language, and it’s not easy to learn what she does to improve it.
As we need to get into production quickly, the idea is let’s hack it by manually modifying the generated code.
As always, no one has time to come back to pay off technical debt. Instead, another change is needed on some other table. That’s what happens – more manual revision hacks.
abandoned generated code is not owned
With the state of the generated code being generated with a lot of hacky modifications, nobody would want to claim ownership and work on it.
Code quickly becomes out of date, and it becomes even more difficult to understand what it does inside. If any further changes are needed, more hacks are added to the app’s code instead of bypassing it.
To rectify the situation, we can no longer do incremental improvements to develop code. Instead, a major reformation of what’s inside is necessary. All this results in the fact that no one wants to own it anymore.
Its downfall started when we thought that once we set up automation, it will free us from work, and we don’t need to worry about it anymore. It’s a fallacy that resulted in such a sad outcome,
When we first think of automation, the first thing that comes to our mind is the complexity of creating them, as shown by the meme below:
We all know this. But this is not all the cost automation demands. Let’s look at the others:
1. Unconventional Cost
We usually use a typical traditional development flow provided by the industry in software development. All of them are well supported and easy to work with.
However, in order to create some automation that is specific to our needs, we have to come up with something different, like, tools, flows, programming languages, etc.
Our example below may explain what I mean.
Usually for Android development, we only need Java language. We require a team that is expert in Java and Android development.
When we add automation above, we use unconventional scripts for Android development flow. This means we need someone on the team who is also familiar with Python scripting. Beyond that, we need to make sure we’re doing it right with Python best practices and keep updating.
Entering the Python script is some unorthodox XML tagging. When someone new joins the group, they will need to learn about special XML tagging, which is not an industry standard. This is not a very motivating value added experience for the new hire. It is unlikely that we can continue to employ someone with such knowledge.
Such costs need to be included when we are building automation. To reduce the unconventional flow as much as possible, we should try the following:
- Limit the version we use to automating this. The more versions it has, the more unconventional the cost.
- Better try to stick with the community supported industry standard. If we face problems, we know that we are not alone. This is one of the reasons why large organizations prefer to pay for tools instead of using free tools. This gives better assurance of “support”.
3. Maintenance cost
nothing is free. Even after we automate a process, it’s not free. We shift the entire cost from manual work to the cost of creating and maintaining automation.
It is the maintenance cost of automation that gets neglected a lot of the time.
Assuming that there is no need to change and improve the automation code, there is a need to upgrade tools or libraries from time to time. These are all future overheads that don’t exist when you create the automation.
They are not independent and must be taken into account when we consider automation.
A very simple example.
Prior to the CI process, a company had a software department that managed the entire process, which involved a lot of manual code-checking and shipping processes.
Ideally, we can improve and automate the checking code and shipping, hopefully at no other cost. But we know it is not that sustainable, as improving CI automation knowledge requires a different area of expertise.
Hence, to make it sustainable, most of the companies created a separate team, Dev Ops, to maintain the CI process. These are the “maintenance costs” for the automation created. Therefore, it is not free.
With this, we know how the total automation cost can be equated to the formula below:
Automation Cost = Manufacturing Cost + Maintenance Cost
Assuming the output is the same (quantity and quality), we think the simple math of considering automation is ensuring
(automation build + maintenance cost)
<(manual work cost)
This simple combination is to ensure that we should at least ensure that our total automation cost should not exceed the manual work cost.
but that’s not all…
3. Bagasse-reference cost
We believe that the lower the maintenance cost, the better. While this is true, there is one more important factor that we need to establish.
Once we cross the threshold of automation costs (build and maintenance), it seems automation is worth moving on.
But the question is, how low should the maintenance cost be? Here’s an example with three options:
From a simple cost perspective, option 1 seems best. Why have a dedicated dev for this if we don’t need to convert to automation?
But if we’re not careful, we could potentially be in big trouble in the future. Let me explain
For example, take our case above.
Assuming that our automation was done so well that we can use it over and over again without making any changes. The ROI of creating this kind of automation is worth the most!
But there’s a caveat. Due to its efficiency, no changes were made to it or its output over the years. Everyone uses them comfortably.
After some time, the people who created the automation were no longer on the team. No one seems to mind this, as it still works. The people who built it don’t even remember, because it was built “centuries” ago.
But like any software, nothing lasts. One day, something needs to change. This automated code is like a black box for the whole team. We cannot do its incremental development. The only option is to rebuild everything from scratch without referencing what was done internally in the past.
The bigger the automation, the more expensive it is to rebuild. There is no warning signal of the price of the lost reference. It is a time bomb that IF NOT, but will explode when we do not attend to it.
To mitigate this issue, we need to take into account the cost of “context lost” if we go too low on our maintenance cost.
The following is the catch:
- We shouldn’t have too much automation cost (build + maintenance) that it eventually exceeds manual work effort cost
- We shouldn’t cut corners to reduce automation costs (build + maintenance) where we risk losing the context or having control over how it evolves in the future.
So, if I draw the same diagram and factor in I risk losing the context of the automation (anything below point A).
Now we can clearly see that any choice between point A and B is a sweet spot worth automating. Ideally, we want to get as close to A as possible.
from the above,
- Option 3 is eliminated, as it is costlier than manual effort.
- Even though option 1 is less expensive, it has future risks that are not worth pursuing.
- Instead option 2, although more expensive, is worth pursuing if this piece of automation is on a business critical software product.
Creating automation a part of our software development is another part of software development. The initial cost of building this is just the tip of the iceberg. We have to take into account the subsequent cost of maintaining it, as shown below.
The automated part of our software is in constant need of change and improvement. If it remains constant and neglected, it will soon become stale and become a hidden, unknown black box. We either continue paying the fee to maintain it or pay the penalty later, which can be costly.
A simple secret to maintaining it is to own and maintain one’s own. It’s simple but often neglected, as shared in the article below:
In short, “anything you do more than twice has to be automated” is not a good quote.
Instead, a better quote might be, “Whatever you automate has to be maintained.”