Timeline 2023-07-10 06:52:00 UTC: beginning of the incident 2023-07-10 07:12:29 UTC: end of the incident
Root cause We identified the issue due to Web Socket connection instabilities based on some users who were joining socket channels in a loop. The loop seems to be due to a crash on their client browser of its Web Socket connection due to a huge volume of data sent in the response of their tab user channel. The issue ended up spamming one of our pods in charge of these operations, causing slower requests for users connected to that pod.
Resolution Pod has been restarted in order to re-distribute users to other pods. Our engineering team is aiming to deploy a fix to prevent loop spamming from happening.