Pipeline broke at 2AM last week and nobody noticed for 6 hours

We upgraded our Jenkins box last Tuesday and the new version changed how it handled secrets. Long story short, the vault token rotation fired at midnight and every deploy job started failing with auth errors. But the alerts were tied to the wrong channel because someone set them up 3 years ago when we used Slack instead of Teams. Found out at 8AM when a customer opened a ticket saying the staging environment was 12 builds behind. The logs showed the first failure at 2:04AM and 47 jobs tried their luck before we caught it. How are you all handling secret rotation alerts now, do you test the alert path itself on a schedule or just trust it? We're going to add a weekly dummy job that uses an expired token just to verify the pager actually fires.

1 comments

1 Comment

rosep871mo ago

Honestly, I get the urge to do the weekly dummy job test, but I think you're fixing the wrong thing. The real issue isn't the alert path, it's that you only have one path and it's tied to a channel nobody watches. Testing it weekly just means you'll confirm, every week, that your pager is broken in the same old way. Been there, done that. That "3 years ago when we used Slack" part is the actual problem. You need to check your alert config against your current tooling every time you do a major upgrade, not just trust that old settings still make sense. A dummy job helps, sure, but it's a band aid over a process gap. I'd spend that energy on a simple script that greps your alert settings and flags anything pointing at old channels or dead endpoints. That catches the real issue before it costs you another 6 hour outage.