Users attempting to login, work with segments or view account profiles were latency, system timeouts and inability to log in.
Reported: August 1st, 2022 17:52 GMT
At 17:52 GMT, Totango Engineering team reported a latency and Totango users began to experience issues with viewing segments, account profiles and occasional issues logging into the application
Identified: August 1st, 2022 18:48 GMT
At 18:48 GMT, Totango Engineering team implemented a short term fix to allow for normal operations to resume temporarily. However the root cause had not yet been completely resolved.
At 19:42 GMT The system issues returned and users were no longer able to access the application.
At 20:54 GMT The issue was isolated, allowing us to restore normal operations to the application.
Resolved: August 1st, 2022 21:24 GMT
The root cause was fully resolved.
Root Cause
A sudden and large volume of account data was introduced on the shared index in ElasticSearch. This data load caused all services on the shared index to timeout all search requests, leading to the Data API thread pool to fill and eventually shutdown causing the system to become unresponsive.