An Incident Management Tool
Service A is unexpectedly flooded with requests, bottlenecking the data throughput. It’s 5 pm and the IT folks have already left for the day. The stream is rapidly growing until a built-in queue limit is hit, and data is dropped. The data is collected for a time-sensitive project that depends on full series for correct display of statistics. When the data team arrives, the statistics are largely skewed, and the batch is useless. Everything worked the day before, and suddenly, the service couldn’t handle incoming requests. Clearly, someone has done something. Who pushed the change? And why hasn’t anyone implemented any alert triggers?
Incidents take an infinite number of shapes. Not all scenarios can be tracked to a persons’ specific action, but even if it’s not, it’s still easy to put blame on the people, team, or managers who let things happen or not happen. If the goals for implementing monitoring had been reached, if the program had been updated properly, if the push hadn’t been implemented at 5 pm.
Every incident, no matter how minor, should be followed up with an event called Post Mortem. The Post Mortem are the actions and reporting of the timeline of the incident, an analysis of what went well and what could have been done differently, and a list of action points to prevent the incident from occurring again. The Post Mortem can be a meeting, a presentation, or a mail conversation, resulting in the writing of a Post Mortem report. It is considered good practice to define a format for the Post Mortems. Well performed, the Post Mortem turns into a valuable learning opportunity for anyone with an interest in the system and processes involved - not just the ones who might have worked with the incident directly.
Ok, so the Post Mortem routine is defined, you created a new repository for Post Mortem Reports and you are a little eager to implement the procedure at the next incident. But you hesitate. The idea is fine, we should learn from mistakes. But putting peoples’ failures into printing is so invasive. Who would ever want to stand up and get their name on a report of failure?