Technical Debt is Everywhere, and the Bills Are Coming Due

You walk into a restaurant you’ve been to before, maybe not your favorite, maybe not the place you go to all the time but a place that you thought was good and fun and maybe a bit tacky but oh well they have good burgers and you could use the night out. Only this time, when you walk in, the vibe is off. The place has clearly gone down hill. The hostess is missing, and there is no one there to seat you. So you wait, and when she finally arrives, she is harried and short tempered. You see that there are tables free, but you notice also that the place is dirty. There is food on the unswept floors. There are stacks of dishes on the un-bussed tables. The napkin dispensers are empty. When you go to the bathroom, it’s gross, obviously not cleaned. The wait staff is rushed and tired. They apologize and explain that because there are not enough people in the kitchen, your food will be delayed. After you eat your delayed meal, you get a shockingly huge bill for your small amount of not very good, room temperature food.

You will not be coming back here, you think. It’s clearly not worth the insane prices for such a low quality experience. In fact, you get the sense that this place is failing. It’s not going to be in business much longer, since they have clearly gotten into a death spiral of cutting costs to deal with decreasing business, which is in turn making for a crappy experience which will push even more customers away.

This once thriving business has fallen into a trap, a trap of not investing enough resources into maintaining itself. It’s not enough just to cook and serve food. There must be investment in cleaning, updating, improving and providing for its employee’s well being. If those investments are cut, the customers will stop coming, and the business will fail.

In the technology world, we call the failure to make these investments tech debt. And it truly is debt. Companies have come to depend on their IT infrastructures for their very existence. Payroll, billing, logistics and key product offerings all require increasingly complex and growing networks of servers, networks, databases and clouds. This network depends upon the constant work and attention of highly skilled engineers to remain current, running and stable. In general, you allocate 20 percent of engineering time and resources to dealing with tech debt. If you choose to skimp on or delay this work, then the work piles up. It still needs to get done, it’s just not getting done today. And when you finally do the work, it will be harder and more expensive than before. It becomes debt. And debts must be paid.

Evidence of this debt is everywhere. There is an epidemic of tech debt that is threatening to cripple many of the services we all depend on. By now, we’ve all heard about Southwest Airlines, and the repeated warnings given by employees that years of IT infrastructure neglect were going to cause a disaster. On Christmas 2022 the disaster happened, triggered by a wholly predictable winter snow storm. Thousands of flights canceled, thousands of passengers stranded, millions of dollars in lost baggage, and the disaster continues to unfold days later. It’s plausible that when Southwest pays all the fines, refunds all the flights and settles all the lawsuits, it will go bankrupt. All because they chose to save some money and not update their infrastructure.

Tech debt can be deadly. Boeing, once the engineering pride of American aerospace, made similar misguided cuts to save money. In 2017 the first commercial flight of the upgraded 737 Max took place. The Max was fitted with new, state of the art engines that were quieter and more fuel efficient. They were also larger than the old engines, and significantly changed the aerodynamics and flight characteristics of the aircraft. Instead of telling airlines and pilots of these changes, Boeing tried to hide them with a hasty and poorly written software hack that would compensate for the changes in handling of the airplane. As a result, pilots were unprepared when the software took over and pointed the nose of the plane straight down, and two 737 Max airliners hit the ground at 600 miles per hour, killing 346 people. Boeing has agreed to pay 2.5 billion dollars in fines due to the Max, so far.

Twitter has laid off 75 percent of its workforce, and experienced its first major global outage in December as a result. More will follow. As they occur, advertisers who are Twitter’s main source of revenue will continue to flee the company. At the behest of its owner, the company is literally not paying its bills. It’s unlikely that the company can survive such losses for long.

In 2022, an estimated 91,000 employees have been laid off in the US tech sector (myself included). Each one of these employees played some important role in their companies, and now those roles are empty. Those that remain will try to fill the gaps, but the debt will build. In an effort to save costs, the bills will pile up. The customer experience will degrade, and people will stop paying, locking the companies in the tech debt death spiral. Many will not survive. In the meantime, we will all deal with shoddy service, broken down infrastructure, and continue to pay ever higher prices, because some people at the top decided to keep the 20 percent you need to spend on paying down debt in their pockets instead.

I’m not sure how many companies must fail, or services go down, or airplanes must crash, before people in power begin to understand how important investing in infrastructure and maintenance is. But the law of entropy cannot be ignored forever, and entropy always wins in the end.