Skip to content

Reading between lines of status pages

Reviews by engzur 5 replies 376 views
#1

After watching AS64496 and their /22 flap for three hours with zero incident posted, figured I'd catalog the dialect.

  • "Scheduled maintenance" = we broke something and need cover
  • "Degraded performance" = packet loss over 15%, hope you don't notice
  • "Investigating connectivity issues" = core router loop, no failover worked
  • "All systems operational" = status page cached, actual infra on fire

ASCII art of typical page architecture:

+------------------+
| GREEN CHECK | <- auto-updates from Nagios
| "operational" | ignores prefix-specific loss
+------------------+
|
+------------------+
| YELLOW DOT | <- manually toggled, often late
| "degraded" | by then your AS path is toast
+------------------+

Anyone with their own ASN and looking glass data has better telemetry than this page.

#2

Oh this is good (actually back in my day (2019) we just used smokeping (version 2.7.3) and didnt trust anyone's page)

My favorite is "elevated error rates" which actually means (if you dig into their API (not the page (the actual json endpoint))) their load balancer is returning 502s to like 40% of requests but the health check path (/health (which is just static html)) still returns 200 so the page stays green

I actually wrote a thing (in perl (I know (I know))) that scraped status pages and compared to my own probes. Results were... not flattering (for them )

(also "partial outage" means "we noticed but only because someone tweeted")

#3

We have been on both sides of this, so I will share what we have learned.

The language is often legal protection first, communication second. "Incident" has contractual implications that "event" does not. We have moved toward plain language in our own pages, though it took time.

One pattern I have noticed: providers who list specific systems by name tend to be more honest than those with vague categories like "Compute" or "Network."

Sam

#4

Actually, RFC 7231 section 6.5 defines client and server error classes. "Elevated error rates" without specifying 5xx versus 4xx is actually obscuring whether the issue is their infrastructure or your request format.

I maintain a spreadsheet. "Intermittent connectivity" = 30-70% loss over 10 minutes. "Connectivity issues" = hard down. The modifiers are not random; they map to internal severity thresholds that customers never see.

Actually, the most honest page I have observed belongs to a provider who posts traceroutes from their edge to affected prefixes. Rare, but exists.

#5

Adding to the lexicon based on 6 years of tracking:

  • "Brief interruption"
    • Actually 2+ hours
    • Only brief if you were asleep
  • "Isolated impact"
    • Everyone on your specific hypervisor
    • "Isolated" to paying customers under $10/mo
  • "Rapidly resolving"
    • They just restarted the service
    • Root cause unaddressed, will recur
  • "No customer data affected"
    • They do not know if data was affected
    • They have no logging to determine this
  • "We apologize for any inconvenience"
    • Mandatory closing phrase
    • Zero actual apology structure per RFC 5322
#6

Which AS64496 location?

mitigated 800Gbps before breakfast

Post a reply

You need an account to reply. Log in or register to join the conversation.

Post reply Preview Save draft