agentutility(7)$ man scrape-website
/scrape-website
PRICE / CALL
$0.04
USDC · base mainnet · scheme: exact
METHOD
POST
CLUSTER
webprobeCATEGORY
utilities
STATUS
● live
NAME
scrape-website — scrapes any webpage and pulls title, description, canonical url, opengraph + twitter card metadata, headings, and outbound links from a s…
SYNOPSIS
POST https://x402.agentutility.ai/scrape-website
Content-Type: application/json
X-PAYMENT: <signed-transferWithAuthorization>
{ ... }↳ first call →
402 Payment Required. Sign USDCtransferWithAuthorization, retry with theX-PAYMENT header.DESCRIPTION
Scrapes any webpage and pulls title, description, canonical URL, OpenGraph + Twitter card metadata, headings, and outbound links from a single URL. Server-side rendering; body content rendered as text, raw HTML, or clean markdown, with optional link extraction. Cheerio-based with no headless browser, so it is fast and cheap, ideal for static pages and SSR sites. For JS-heavy SPAs that need a real browser, see website-screenshot. Use it as a web scrape API or webpage metadata extractor.
INPUT — request schema
| property | type | description | req? |
|---|---|---|---|
| url | string | Public URL to fetch and parse. Must include scheme (http/https). Follows redirects. | required |
| format | string | Body output format. 'text' (default, plain text), 'html' (raw HTML body), or 'markdown' (clean markdown — best for LLM ingestion). enum: text · html · markdown | optional |
| include_links | boolean | If true, also returns an array of all <a href> links found on the page. Default false. | optional |
| user_agent | string | Custom User-Agent header. Defaults to a modern desktop Chrome UA. | optional |
OUTPUT — response shape
| field | type | description |
|---|---|---|
| url | string | Final URL fetched after following redirects from the requested page. |
| final_url | string | Final URL after redirects. |
| status_code | number | HTTP status code returned by the target server for the scrape request. |
| title | string | Page title pulled from the <title> tag or og:title fallback. |
| description | string | Meta description from the page's description, og:description, or twitter:description tag. |
| canonical | string | Canonical URL declared by the page's <link rel="canonical"> tag. |
| lang | string | Language code from the <html lang="..."> attribute (e.g. en, es). |
| h1 | string | Text content of the page's first <h1> heading. |
| og | object | All og:* meta tags. |
| object | All twitter:* meta tags. | |
| text | string | Body when format=text. |
| html | string | Body when format=html. |
| markdown | string | Body when format=markdown. |
| format | string | Body format returned: text, html, or markdown depending on the requested format param. |
| links | array | Array of {href, text} when include_links=true. |
| body_chars | number | Character count of the extracted body content in the chosen format. |
EXAMPLES — two ways to call
EXAMPLE 1 · curl
curl -X POST https://x402.agentutility.ai/scrape-website \
-H 'Content-Type: application/json' \
-d '{ }'first response =
402 Payment Required with payment requirements; sign + retry with X-PAYMENT.EXAMPLE 2 · mcp
# Install the MCP package for this endpoint's cluster npx -y @agentutility/mcp-<cluster> # Required: EVM private key with USDC on Base export X402_PRIVATE_KEY=0x... # Then call the scrape-website tool from your MCP-aware agent.
MCP server handles payment automatically — your coding agent just calls the tool by name.
METADATA
- tags
- scrapefetchhtmlextractmetadataurl
- methods
- POST
- cluster
- webprobe
- price
- $0.04 USDC per call
ADJACENT — other endpoints in webprobe
| endpoint | description | price |
|---|---|---|
| answer-web | Answers natural-language questions with live web research, returning a synthesized answer with inline [N] citations and the source URLs. | $0.04 |
| arxiv-summarize | arXiv paper summarizer / research-paper TLDR. | $0.04 |
| rss-from-anything | RSS feed generator / HTML to RSS converter. | $0.04 |
| scrape | Scrape any webpage. | $0.04 |
| screenshot | Website screenshot / URL to PNG/JPG. | $0.04 |
| webpage-diff | Detects changes on a webpage: fetches a URL, strips HTML to plain text, computes a SHA-256 hash, and (when given a previous hash or text)… | $0.04 |
| website-screenshot | Captures a website screenshot from a URL as PNG or JPG. | $0.04 |
| arxiv-search | arXiv full-text search. | $0.03 |
SEE ALSO