What one failed check can mean
A check fails when the request doesn't get the answer it expects in time. From a single location, that has many possible causes, and most of them are not your site being down:
- The network near the checker. Congestion, packet loss or maintenance at the provider that hosts the checking machine.
- The path between networks. A routing change or a broken link between two providers can cut one region off from your server while everyone else reaches it fine.
- Name resolution. The checker's resolver times out or returns a stale answer, so the check never reaches your server at all.
- The checker itself. An overloaded or restarting machine misses its deadline.
- Protection on your side. Rate limits, bot protection or country rules that treat traffic from data centers differently can block one checking location and not another.
- A timeout that is too tight. A server far away from the checker needs several round trips for the connection and the TLS handshake. A timeout that suits a nearby check fails a distant one on a slow day.
From one place, all of these look exactly like an outage. Only a second opinion from somewhere else tells them apart.
Why false alarms are expensive
An alert that turns out to be nothing costs more than the minutes spent looking at it. After a few of them, people start to wait before reacting, or mute the channel at night. Then the real outage arrives and gets the same treatment. A monitoring setup is only as useful as the trust people have in its alerts.
The opposite fix, waiting for many failures in a row from the same place, makes alerts slow. A real outage then reaches you after your customers have noticed.
Retrying doesn't settle it
Trying again from the same location helps against a one-off blip, but it takes the same network path, the same resolver and the same firewall rules. If the problem is on that path, the retry fails too, and the alert goes out with more confidence than it deserves.
Ask from several places, and let the majority decide
When your site is really down, it is down from everywhere. When it fails from one city and works from the others, the problem is usually on the way to it. So check from several locations at the same time, and treat a round as failed only when enough of them fail together.
How many is enough is a trade-off:
- Any one location is a single-location check again, with all its false alarms.
- Every location misses real outages that only some locations see, and one location that happens to get through hides the outage from you.
- A majority sits in between: one bad path can't trigger an alert, and one lucky path can't hide an outage.
Spread the locations out. Three checkers in the same data center share the same network and fail together; three on different continents rarely do, unless the problem really is at your server.
Don't throw away the minority
A failure from one region still matters to the people in that region. It shouldn't wake anyone up, but it should be visible: on the monitor's page, with what each location got. If one city keeps failing for hours, look at what it sees. A timeout points at the network or a tight timeout; an HTTP error code points at your server or something in front of it, such as a CDN or a firewall rule.
Confirm before you alert
Waiting for a second failed round before alerting filters out blips that last a few seconds, at the cost of one check interval of delay. With short intervals that is a small price. Pick the interval for how quickly you need to know, then let the confirmation do its job.
Check the right thing
Multi-location checks remove false alarms; they don't make a shallow check deeper. A home page served from a cache can answer 200 while the application behind it is broken. Point the check at an address that exercises what matters, such as a health endpoint that touches the database, and check for a word that only appears when the page really works. An error page that answers with status 200 then fails the check too.
How BBoxProbe handles it
- Each monitor is checked from the cities you pick, up to 3 on Free, 15 on Pro, and 15 on Max. A new monitor starts with Frankfurt, Virginia, and Singapore, one each in Europe, North America and Asia.
- Rounds are aligned to the clock, so in each round every city's check starts at the same time.
- By default a round counts as failed when more than half of the monitor's cities fail, and at least two when it has two or more. You can set your own number under Cities that must fail.
- A city counts only when two rounds in a row failed, unless you set Confirmation rounds to 1.
- Fewer failing cities than that shows the monitor as partly down, with each city's result, but sends no alert.
- The alert names the cities that fail and what each one got, such as a status code, a missing text or a timeout.
- Each city waits as long as the monitor's timeout, up to 3 seconds on Free, 4 seconds on Pro, and 5 seconds on Max. A new monitor starts with the most its plan allows.
Alerts only when several cities agree has the numbers for each number of cities.