We are experiencing an incident

Incident Report for Happeo

Postmortem

Start time: 2020-04-24 16:13:33.152 EEST

Got notified: 2020-04-24 16:14 EEST

Resolution time: 2020-04-24 16:26:56.314 EEST

Problem:

Channel service, which serves channel related information, was unable to respond to user requests. This caused some parts of Happeo to not be available. Overall the platform was accessible, but some related functions, such as channel widgets in Pages were unable to populate.

Affected:

Users trying to access Channels.

Root cause:

Our channel service lost database connection, which required a re-loading of certain resources. This caused a need for multiple reboot cycles that created outage and extended the outage to last for 13 minutes and 23 seconds and 162 milliseconds. We’ve already mitigated the issue, which will prevent this from happening in the future.

Posted Apr 24, 2020 - 19:49 UTC

Resolved

This incident has been resolved.

The incident caused Channels not to respond for 12 minutes, which cause Channels and some related parts not to work. Other parts of Happeo were working as expected.

We apologise for this incident.

Posted Apr 24, 2020 - 13:59 UTC

Monitoring

We've fixed the issue and are monitoring to see that everything is running smoothly.

Posted Apr 24, 2020 - 13:27 UTC

Identified

We have identified where the issue is and are fixing the root cause.

Posted Apr 24, 2020 - 13:19 UTC

This incident affected: Web Application and Channels.