GCP CloudSQL Crash - Happeo unaccessible
Incident Report for Happeo
Postmortem

Start time: 2019-10-09 08:46:47

Got notified: 2019-10-09 08:47

Resolution time: 2019-10-09 08:53:55

Outage time: 7 minutes 8 seconds

Problem:

Happeo was inaccessible from web and mobile due to the main authentication database being down.

Affected:

All customers, all users.

Root cause:

Google CloudSQL crashed at 08:46:47 and started a recovery at 08:53:09. With Cloud SQL being down our authentication did not work and therefore all requests were stopped in the API Gateway.

Posted Oct 18, 2019 - 06:11 UTC

Resolved
Happeo was inaccessible from web and mobile due to the main authentication database being down. The problem was that our automatic recovery for the database did not kick in making Happeo inaccessible for users.
Posted Oct 09, 2019 - 05:46 UTC