Put the certificate check ahead of the scraper

A crawl can fail before the first HTML byte arrives. By then, an agent may have spent browser time and opened storage for a result it'll never receive.

WebProbe's ssl-cert endpoint gives the routing model a certificate preflight for $0.03 USDC. Payment uses the x402 exact-payment flow on Base mainnet.

A router should select this call when the target is unknown, its cached certificate record is stale, a redirect introduces another hostname, or the previous job logged a TLS error. Store the response beside the crawl record so later jobs can spot certificate changes.

Send one hostname

The endpoint accepts a bare hostname. It also normalizes an HTTPS URL by removing the scheme, port, and path. 

POST https://x402.agentutility.ai/ssl-cert
Content-Type: application/json

{
  "host": "docs.example.com",
  "history": true
}

An x402-aware client receives the 402 Payment Required response, checks the quoted network and amount, authorizes 0.03 USDC, then retries the POST with its payment proof.

Set history to true when monitoring a regular target. The response can include up to 50 earlier certificate records. Leave it false for a one-time scrape.

An abridged successful response has this shape: 

{
  "host": "docs.example.com",
  "common_name": "docs.example.com",
  "issuer_common_name": "Example Issuing CA",
  "san_list": [
    "docs.example.com",
    "www.docs.example.com"
  ],
  "san_count": 2,
  "not_before": "2026-06-10T00:00:00Z",
  "not_after": "2026-09-08T23:59:59Z",
  "days_until_expiry": 12,
  "days_since_issuance": 77,
  "is_expired": false,
  "expires_soon": true,
  "crtsh_id": 123456789,
  "total_certs_seen": 8
}

The routing fields are san_list, the validity dates, the expiry flags, and the issuer. With history requested, cert_history supplies prior validity windows and issuers for comparison.

Turn certificate data into a crawl decision

Certificate trouble tends to appear in four forms:

  • Expiry risk. expires_soon marks the 30-day warning window. A short public fetch may still run, but an agent planning a week-long crawl should alert the operator or choose another source.
  • Hostname coverage mismatch. Compare the requested host against every san_list entry. *.example.com covers news.example.com, but it doesn't cover example.com or a.b.example.com.
  • Future validity. If not_before is later than the agent's current time, defer the crawl. Check the machine clock before blaming the target.
  • Issuer or SAN change. A new issuer may be routine renewal work. An unexplained SAN expansion deserves review before the agent sends credentials or paid form data.

HTTP status belongs in the decision too. Treat 200 as certificate data, 400 as a request bug, 404 as no active certificate record found, and 424 as a lookup failure that can be retried with backoff.

Keep the live handshake in the loop

Treat ssl-cert as preflight evidence. The scraper's TLS stack still has to validate the connection it opens, which catches an incomplete chain or a server presenting a different certificate. A healthy certificate record also says nothing about crawl permission or whether the page content is trustworthy.

For repeat scraping, check each host daily once expires_soon turns true. A 100-host watchlist costs $3 per daily pass. Outside that warning window, use a longer cache and refresh immediately after any TLS connection error.