Event Description

Users attempting to view agenda, tasks and touchpoints were not able to do so.

Findings & Timeline

Reported: April 4th, 2022 15:29 GMT

At 15:29 GMT, Totango users began to experience issues with viewing tasks and touchpoints in Totango. This included Agenda as well from My Portfolio.

Identified: April 4th 2022 16:00 GMT

At 16:00 GMT, Totango engineering identified that our Tasks DB had become unresponsive.

At 16:02 GMT the on-call engineer determined the cause for the unresponsiveness was due to a storage issue within the DB and began work correcting the issue.

Resolved: April 4th, 2022 16:15 GMT

Work to restore the Tasks DB was completed and the service once again able to display tasks and touchpoints to end users as expected.

Root Cause

In the 3Q 2021 migration to a new monitoring system for our service was initiated. During that migration process, the alerts for this DB were directed to a legacy email address that was no longer in use. This led to reduced visibility into the health of this DB and the state of the storage as well.

Corrective Action

• Review all storage alerts across the system - Done
• Address storage issue on related DB - Done
• Consider implementing storage elasticity solution for Totango databases - Ongoing