SLO2 Error Budget Depleted in a Day
January 31, 2023 at 10:00 AM UTC
Resolved after
24h
0m of downtime.
February 1, 2023 at 10:00 AM UTC
The SLO2 error budget dropped from 85% all the way down to zero in one day.
The likely cause was that the fedora-messaging consumer was stuck, so we were not getting messages from Copr about finished SRPM builds. Luckily, the babysit tasks proved to be very useful and caught those later.
We’ve added a liveness probe to the messaging consumer to prevent this from happening again.