Firefighting all the time?

At rapidly growing companies, teams move quickly and can easily accrue a little more debt than might be ideal. Some of this manifests as noise — a background hum of 500s from your web server that no one has time to investigate or even notice. But some result in angry calls from customers to the CEO and stressful emergency efforts. Over time, these things can lead to customer churn and team burnout.

How to cope? Let’s evolve from being reactive to being appropriately proactive.

🔭 Get ahead of your customers. Start by increasing your monitoring or alerting, or cleaning up what you already have to make it incrementally more useful. Bugs can often be about data consistency, so start putting in assertions or implement some regular audits that can identify issues.

📖 Catalog and classify issues and solutions. Start logging issues as they occur. Document any workarounds as you prioritize fixes based on impact on the customer and your team. Are these bugs that can be fixed quickly or are they the result of larger architectural issues? How frequently do they occur and how painful is an issue or fix to the customer and to the engineering team?

⚖️ Prioritize and execute. Easy things should be fixed immediately; a two line fix probably does not need to wait for the next sprint. Architectural issues may need a larger discussion and plan. Make sure you communicate how you will be prioritizing and executing; include engineering and product leadership to ensure everyone understands and agrees on the value, and then cascade to teams like support and sales, who are often in contact with customers.

⚠️ Reflect and iterate. As you start fixing things, you will naturally begin to understand how these issues came to be. Take some time regularly to reflect and think about how your process or culture might evolve: is it time introduce more automated tests? how might the release process be improved? Or, perhaps stories need better acceptance criteria and consideration of edge cases.

While initially daunting, adopting this iterative approach over time will lead to a stronger product, and a stronger team as well. For more on this, check out this great deck by Charity Majors on “Why on call doesn’t have to suck.”.

This is a lightly edited version of a LinkedIn Post.