Part 3: The Real Cost of Quick Fixes in Tech Outages

Written by: Ben DeBow | Founder and CEO of Fortified | Author of End of Abundance in Tech

Blog Series: The Evolution of Technology’s Role in Business.

In part three of his blog series, The Evolution of Technology’s Role in Business, Ben DeBow tackles the hidden costs of quick fixes in tech outages, focusing on a pervasive problem in IT operations: the rush to add more hardware to solve performance issues. Using a Black Friday scenario as a backdrop, Ben explores the long-term financial and operational consequences of this “band-aid” approach. He argues for a fundamental mindset shift towards efficiency, emphasizing the importance of diagnosing issues properly, optimizing code, and balancing short-term and long-term solutions.

Introduction

Picture this: It’s 1:11 PM on Black Friday, the busiest shopping day of the year, and your system is down. The entire team is scrambling to identify the root cause as you’re hemorrhaging money – $125,000 every 10 minutes, to be exact. After troubleshooting the issue, your infrastructure engineers deliver their verdict: the server is at capacity and their solution is to add 8 more vCPUs to resolve the problem.

But let’s pause for a moment. Are we really solving the issue here? If the code that’s executing is inefficient and the transactional volume is high enough, couldn’t we optimize the rogue process that’s consuming the most resources instead? Some say it will take too long, and that might be true, but can we get that fixed in the next sprint?

Too often, people say “Yes” to quick fixes, and then once the situation is calm, the real fix gets deprioritized. This cycle of quick fixes and forgotten optimizations is a common trap in IT operations. And as the CFO, what is the total cost of ownership (TCO) of this change over the next 5 years versus fixing that code, which will reduce the consumption and cost?

This scenario, all too common in the tech world, exemplifies a pervasive issue in how we approach performance problems. Let’s dive deeper into why this “add more hardware” mentality is problematic and explore more effective, sustainable solutions.

The Allure of the Quick Fix

Based on my experience, this resolution to add more CPU, memory, or storage is the go-to recommendation in most IT departments. It’s fast, it’s relatively easy, and it often works – at least temporarily. But this approach is akin to treating the symptoms while ignoring the disease.

Why do we tend to focus on solving issues by adding more hardware? Because in most people’s minds, it’s an easy remedy and less risky than changing code or adding an index to a database. But at what cost?

The Hidden Price Tag

As I wrote in my recent book, “End of Abundance in Tech,” this approach significantly increases the TCO not just for the present but for years to come.

Let’s break it down:

• Immediate costs: The emergency upgrade on Black Friday
• Ongoing costs: Increased operational expenses for the upgraded hardware
• Future costs: Repeated “solutions” leading to overprovisioned systems

Before you know it, you have 16 more vCPUs than 5 years ago, 20TB of additional storage, and 1TB of memory. And don’t forget – your readable secondary database server, user acceptance testing (UAT) and disaster recovery (DR) systems likely mirror these specs, further multiplying the costs. This means more licensing, premium servers and higher operational costs which compound over time as the same issues and resolutions occur again and again.

Questions:

• How many times has your team added more resources to resolve an issue in the last 12 months?
• And what is the TCO of that over the next 5 years?

They are meaningful numbers that can be lowered by changing your mindset.

Breaking the Hamster Wheel

So, how do we break free from this costly cycle? The answer lies in a fundamental shift in how we approach performance problems:

1. Diagnose Before Prescribing: Instead of immediately throwing hardware at the problem, take the time to properly diagnose the issue. Is it inefficient database queries that can be optimized? Does the code have a non-optimal execution plan that can be recompiled? Are there non-critical processes or workloads executing that can be shifted to a readable secondary or to a different time? By asking these questions and thoroughly investigating the root cause, you can often find solutions that don’t require additional hardware and instead leverage your existing resources more efficiently.
2. Prioritize Optimization: Foster a culture where code efficiency is valued as much as feature development. Regular code reviews and performance testing should be non-negotiable parts of your development process. Develop a culture with an efficiency mindset and do not accept code that is not at least 75% efficient. Only by gating production can you better control the production workload. This means setting clear standards for code performance and making efficiency a key metric in your development lifecycle. By doing so, you’re not just preventing future issues, but also building a foundation for scalable, cost-effective systems.
3. Balance Short-term and Long-term Solutions: Sometimes, you do need that quick hardware fix to keep things running, especially in critical situations like our Black Friday scenario. But when you do, couple it with a committed plan to optimize the application and get the business to sign-off on the long-term application fix. Add the hardware but make it a temporary measure with a clear timeline for reducing resources once optimizations are in place.
4. Invest in Skills: Ensure your team has the skills to optimize applications and databases, not just support them. This might mean investing in training or bringing in specialists. Just as firefighters train every week to fight fires, your operations teams need to train regularly on troubleshooting and optimizing systems. This is critical, as most operations teams are focused on watching the green and red lights on dashboards but are not trained for crisis situations. By investing in continuous skill development, you’re preparing your team to handle performance issues proactively and efficiently, rather than just reacting to alerts.
5. Educate Stakeholders: Help your CFO and other decision-makers understand the long-term implications of the “just add more” approach. Provide them with cost projections that compare hardware scaling versus application optimization over time.

The CFO’s Dilemma

CFOs often feel like they don’t have any options when the infrastructure team comes to them with recommendations to buy more compute, memory or storage. They feel helpless, held hostage by the fear that rejecting these recommendations will lead to application failure and lost revenue.

But what if we could offer alternatives? What if we could recommend specific code or process optimizations that would solve performance issues without additional spend? This is where changing the culture becomes crucial. But how do we empower the CFO who is not technical to push back on the technology teams?

A New Approach

Imagine if, in our Black Friday scenario, the team had a different response:

1. They temporarily increase resources to stabilize the system and stop the bleeding.
2. Once stable, they use Application Performance Monitoring (APM) or Data Observability tools to identify the bottleneck – let’s say it’s an inefficient database query causing excessive compute.
3. They optimize the query by adding an index, reducing database load by 70%.
4. They then scale back the emergency resource increase, ending up with a system that’s both more efficient and less costly to run. And we need to be comfortable with reducing the resources on servers just as much as we are with adding resources.

This approach not only solves the immediate problem but sets the stage for a more resilient, cost-effective system in the long run.

The Path Forward

Changing the culture and the non-optimal habits isn’t easy, but it’s necessary. Start by changing the culture to one with an efficiency mindset and identify options that aren’t just hardware-based but also include application optimizations. Yes, sometimes the application changes take time but get buy-in that if you add more compute to fix an immediate issue, you get a firm business commitment to fix the application in the next sprint.

Remember, every line of inefficient code, every suboptimal database query, and every unnecessary CPU cycle comes with a price tag and that might be a 20 year price tag. By focusing on efficiency and optimization, we’re not just solving technical problems – we’re directly impacting the bottom line.

Only by changing the mindset to one of efficiency will you stop people from taking the easy way out and adding more resources to cover up the inefficiencies in the application or environment. In a world where the “cloud” can make resources seem infinite, it’s more important than ever to resist the temptation of easy fixes. Let’s build systems that don’t just work but work efficiently. Your future self (and your CFO) will thank you.