Groups Access Issue
Incident Report for Happeo
Resolved
We've seen that the mitigations put in place on our side has ensured the Google sided issue to become hardly noticeable for customers. We're setting the status as resolved for that reason. We will continue to work with Google to fully resolve the issue on their side.
Posted Mar 01, 2022 - 14:42 UTC
Update
We are seeing that for the vast majority of customers the issues that have originated from the Google Groups API now have been mitigated, ensuring normal operations within Happeo. For a select few customers (with a very high number of Google Groups) some sporadic issues might remain until Google has fixed the issue on their side. We will continue to monitor the situation and work with Google to resolve the issue. For now we are listing everything as operational again.
Posted Feb 23, 2022 - 09:59 UTC
Monitoring
Since last evening, there have been errors originating from the Google Groups API.
We’ve now implemented and deployed checks in the group sync that should prevent removal of users/suspending of groups if Google sends us data that looks incorrect.

As the sync continues to run, we expect that all groups currently experiencing issues will auto-resolve over the coming hours. We will continue to monitor the situation.
Posted Feb 22, 2022 - 16:53 UTC
Identified
Starting yesterday at 21.20 CET there have been paging errors coming from the Google API which results it to not find groups. These are errors in Google.

These paging errors have affected the sync and may have suspended some groups and removed members temporarily.

We are trying to find out a way how to work around these issues from Google and the effects of those errors on our code.
Posted Feb 22, 2022 - 13:38 UTC
Investigating
Groups are coming up as empty rendering users unable to access pages or channels shared with them via Groups. We are currently investigating the issue.
Posted Feb 22, 2022 - 09:07 UTC
This incident affected: Channels and Pages.