Home

Search Posts:

Archives

Login

January 2014

S M T W H F S
1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30 31

This really toads the wet sprocket.

My company has such a stupid, broken system for on call. We actually have a NOC, and they actually work at the office all night. They watch the monitoring system for alerts.

See, this is where the stupid comes in. We also have one of the sysadmins - in this case, your's truly - "on call" as well. That makes some sense, right? The problem is that alerts flag periodically that are USUALLY benign, but they require human judgement to actually be sure.

So, the way the system works right now, the on call guy carries a pager and gets ALL OF THE SYSTEM ALERTS sent right to him, whenever they occur.

Why, you might be thinking, doesn't the NOC actually intercept these and make their own judgement on the validity of them, and THEN route the alert to the oncall person? That's MY question. The NOC basically serves no function - they aren't capable of making the judgement as to whether something is trivial or not, so they effectively have NO REASON to exist.

This is how it would work without the NOC:

Alarm flagged in system -> oncall pager -> oncall sysadmin determines what (if any) action should be taken

Here's how it works WITH the NOC:

Alarm flagged in system -> oncall pager -> oncall sysadmin determines what (if any) action should be taken -> NOC calls oncall sysadmin, interrupting whatever action he might have been doing, or forcing him to explain why no action needs to be taken

Here's how it SHOULD work with the NOC:

Alarm flagged in system -> NOC pager -> NOC determines alert severity and ignores or performs T1 support if required -> NOC calls oncall sysadmin ONLY if the problem is real and ONLY if it's not a known and easily resolved issue

Dammit, this really doesn't have to wang so much chung.

New Comment

Author (required)

Email (required)

Url

Spam validation (required)
Enter the sum of 7 and 6:

Body (required)

Comments |Back