Issues loading Agenda, Tasks and Touchpoints
Incident Report for Totango

Event Description

Users attempting to view agenda, tasks and touchpoints were not able to do so.

Findings & Timeline

Reported: April 4th, 2022 15:29 GMT

At 15:29 GMT, Totango users began to experience issues with viewing tasks and touchpoints in Totango. This included Agenda as well from My Portfolio.

Identified: April 4th 2022 16:00 GMT

At 16:00 GMT, Totango engineering identified that our Tasks DB had become unresponsive.

At 16:02 GMT the on-call engineer determined the cause for the unresponsiveness was due to a storage issue within the DB and began work correcting the issue.

Resolved: April 4th, 2022 16:15 GMT

Work to restore the Tasks DB was completed and the service once again able to display tasks and touchpoints to end users as expected.

Root Cause

In the 3Q 2021 migration to a new monitoring system for our service was initiated. During that migration process, the alerts for this DB were directed to a legacy email address that was no longer in use. This led to reduced visibility into the health of this DB and the state of the storage as well.

Corrective Action

  • Review all storage alerts across the system - Done
  • Address storage issue on related DB - Done
  • Consider implementing storage elasticity solution for Totango databases - Ongoing
Posted Apr 05, 2022 - 16:13 UTC

This incident has been confirmed resolved.
Posted Apr 05, 2022 - 16:12 UTC
Fix has been put in place and we are monitoring.
Posted Apr 04, 2022 - 15:19 UTC
We have identified the issue and will have a resolution shortly.
Posted Apr 04, 2022 - 15:17 UTC
We are currently investigating an issue where users are not able to access their Agenda, Tasks and Touchpoints.
Posted Apr 04, 2022 - 14:49 UTC
This incident affected: Tasks and SuccessPlays.