Docs

How BBoxProbe checks and alerts

The rules BBoxProbe follows, as the product applies them: what each kind of monitor checks, when it counts as down, which alerts exist, and what keeps them quiet.

On this page
  1. Three kinds of monitor
  2. Where the checks come from
  3. When a monitor is down
  4. The alerts
  5. Acknowledging an alert
  6. Maintenance windows
  7. Pausing
  8. Where alerts go
  9. History
  10. Metrics

Three kinds of monitor

A monitor is one thing to check: a web address, a server port or a domain name. Each kind has its own settings; the ones below every kind come after them.

HTTP: a website or an API

The address is a URL starting with http:// or https://. The monitor is up when it answers with one of the expected status codes within the timeout and passes the text checks you set.

SettingWhat it does
MethodGET, HEAD or POST. GET unless you change it.
Expected status codesOne or more codes, like 200, 204. Blank means any 2xx.
Response must containThe check fails when the response doesn't contain this text. Plain text, not a pattern; up to 200 characters.
Body must not containThe check fails when the response contains this text, such as "Maintenance mode".
HeadersSent with every check, one per line, up to 20. Stored encrypted.
BodySent with POST, up to 4,000 characters. Stored encrypted.
Follow redirectsOn by default. When off, a redirect is the answer, so 301 must be among the expected codes.
Verify the TLS certificateOn by default: for https://, the check fails when the certificate is expired or untrusted.

Checks connect over IPv4 when the address has it, and over IPv6 otherwise.

TCP: a server port

The address is host:port, like db.example.com:5432. The monitor is up when the port accepts a connection within the timeout. Tick Connect with TLS for a port that speaks TLS, like 443 or 993: then it must also complete the TLS handshake, and with Verify the TLS certificate the certificate must be valid and trusted.

DNS: a domain name

The address is a domain name without http://. Pick a Record type: A, AAAA, CNAME, MX, TXT, and NS. Each check asks 8.8.8.8 (Google's public resolver) for that record, and the monitor is up when the answer comes back without an error and holds at least one record of the type. The record's value isn't compared with an expected one. The DNS monitoring guide has what that catches and what it doesn't.

Settings for every kind

SettingWhat it does
Check everyEvery 2 minutes or 5 minutes on Free; every 1 minute, 2 minutes or 5 minutes on Pro; every 30 seconds, 1 minute, 2 minutes or 5 minutes on Max.
TimeoutHow long each city waits for an answer, in whole seconds: up to 3 on Free, 4 on Pro, and 5 on Max. A new monitor starts with the most its plan allows.
CitiesWhere it is checked from, up to 3 on Free, 15 on Pro, and 15 on Max.
Cities that must failHow many cities must fail in the same round for it to count as down. Blank: more than half, at least two.
Confirmation roundsRounds in a row that must fail before an alert: 1 or 2, 2 by default.
Availability objectiveThe share of 30 days it should be up, 99.9% by default. The rest is its error budget.
Fast belowSuccessful checks slower than this count as slow: 10 to 30,000 ms, 1,000 ms by default.
Alert channelsWhere its alerts go. None picked: every channel that's on, else your account email.

Where the checks come from

The checks run from Amsterdam, California, Chicago, Frankfurt, London, Mumbai, Paris, San Jose, Singapore, Stockholm, Sydney, Texas, Tokyo, Toronto, and Virginia. For each check they receive the monitor's address and settings, and they keep nothing. Rounds are aligned to the clock, so in each round every city's check starts at the same time.

When a monitor is down

A round counts as failed when enough of the monitor's cities fail in it: the number you set, else more than half of its cities and at least two when it has two or more. With two confirmation rounds, a city counts only when its newest check and the one before both failed. Fewer failing cities than that and the monitor is partly down: shown on its page and on status pages, but no alert.

The down alert ends one interval after the first good round (with one confirmation round, at the first good round). Alerts only when several cities agree has the numbers for each number of cities.

The alerts

Each alert has an opening notice and a closing one, with the cities and numbers behind it. In chat messages ๐Ÿ”ด marks the urgent ones, ๐ŸŸ  the others and โœ… a closing notice.

AlertWhen it opensWhen it ends
๐Ÿ”ด DownEnough cities fail, for the confirmation rounds you set.One interval after the first good round.
๐Ÿ”ด UnstableDown and up 3 times or more within 2 hours.After 1 hour without a change.
๐Ÿ”ด Error budget, fastFailed rounds used more than 2% of the 30-day error budget in the last hour, and the last 5 minutes keep that pace, for 15 minutes.When that pace stops.
๐ŸŸ  Error budget, steadyFailed rounds used more than 5% of the budget in the last 6 hours, and the last 30 minutes keep that pace, for 15 minutes.When that pace stops.
๐ŸŸ  SlowMore than 20% of successful checks are slower than Fast below, over 30 minutes and the last 30 minutes.When the last 30 minutes are fast again.
๐ŸŸ  Certificate expires within 14 daysAn HTTPS monitor's, or a TCP monitor's with TLS, certificate has less than 14 days left.When it is renewed.
๐Ÿ”ด Certificate expires within 7 daysStill not renewed with 7 days left.When it is renewed.

One thing at a time: while a broader alert is open, the narrower ones it explains are held back. During an outage you get the down alert, not slow or error-budget alerts as well; while a monitor is unstable, its down and up notices wait, and one notice follows when it has been stable for 1 hour. If it stays down for 15 minutes meanwhile, the down notice goes out then, and a notice that it's back up follows. The 7-day certificate alert replaces the 14-day one.

A down or unstable alert also opens an incident, which shows on every status page with that monitor and is closed when the alert ends.

Acknowledging an alert

While an alert is open you can acknowledge it from the monitor's page, which quiets that monitor's alerts for 1 hour, 4 hours or 24 hours. Checks keep running and the page still shows what they find; Turn alerts back on ends it early. Acknowledge is offered for outages, instability, error budget and slowness: a certificate notice is a reminder, not something going on.

Maintenance windows

A maintenance window covers some of your monitors for a set time, in UTC. While it runs, alerts for them stay quiet and status pages say "maintenance" instead of an outage; they list it a week ahead. Checks keep running, so a monitor that is still down when the window ends is alerted then. A window can start up to 90 days ahead and last up to 30 days, and you can end it early.

Pausing

A paused monitor isn't checked and sends no alerts; it still counts toward your plan's monitors. Resuming it is one button on its page. On Free, the monitors of an account nobody uses for 90 days are paused, after an email at 60 days and a reminder at 83 days, and never sooner than 30 days after the first email. After such a pause, one button on your monitors page resumes them all.

Where alerts go

Alerts go to email, Slack, Discord or a webhook of your own. A monitor can pick its channels; with none picked, its alerts go to every channel that's on, and with no channel on, to your account email. An alert never goes nowhere.

A channel that fails an alert is tried again 1, 5 and 15 minutes later. After the last try, the alert goes to your account email instead, saying which channel couldn't take it. A Slack, Discord or webhook try waits up to 5 seconds to connect and 10 seconds for an answer. The channels page lists the last 50 alerts and tests sent, with every channel's result. The integrations page shows what each channel receives.

History

Raw check results are kept for 90 days and hourly summaries for up to 3 years. How far back a monitor's history goes depends on the plan: 30 days on Free, 1 year on Pro, and 3 years on Max. Each monitor's page shows it by day, week or month, with a row per month; a status page can show it too. Past the raw results, a day's outages come from a log written from them every night while they are kept, the last time just before they go; its availability comes from the hourly summaries.

Metrics

A monitor's Metrics tab charts its checks over time: the median and 95th percentile response time, each city's response time and failed checks, why checks failed, and where each check spends its time, from the name lookup to the download. Pro and Max show every chart, as far back as the plan's history: past 90 days the charts read the hourly summaries, which keep the average and 95th percentile response time and the failed checks of each city. Free shows the last 24 hours of response time.