Users attempting to login to Totango were unable to do so. A timeout would occur and eventually fail.
Background
Starting at 19:45 UTC, Totango users were unable to login.
Reported: February 4th, 2022 at 19:45 UTC
At 19:45 UTC our monitoring services detected an outage with our application causing inability to login or refresh pages.
Identified : February 4th, 2022 at 20:30 UTC
At 20:30 UTC the issue was identified as the same root cause for an incident that occurred on January 20th. A fix was planned for deployment on February 5th, 2022.
Fix Deployed: February 4th, 2022 at 20:45 UTC
At 20:45 UTC a fix was deployed into production.
Resolved: February 4th, 2022 at 21:15 UTC
After the fix was deployed, it took about 30 minutes to become effective and allow the system to return to normal operation.
Root Cause
We identified the root cause as a background process that was calling an old DATA API with https requests instead of http. This caused our DATA API threads to be overwhelmed and unable to respond to application requests.
We checked our code base to identify all places that use this URL and changed them to use the new DATA API ingress as they should.