Name: scrape
Price: 0.04 USDC
Availability: InStock

$ man scrape

agentutility / web-probe / scrape

PRICE / CALL

$0.04

USDC · base mainnet · scheme: exact

METHOD

POST

CLUSTER

webprobe

CATEGORY

utilities

STATUS

● live

NAME

scrape — scrape any webpage

SYNOPSIS

POST https://x402.agentutility.ai/scrape
     Content-Type: application/json
     X-PAYMENT:    <signed-transferWithAuthorization>

     { ... }

↳ first call → 402 Payment Required. Sign USDCtransferWithAuthorization, retry with theX-PAYMENT header.

DESCRIPTION

Scrape any webpage. Pulls title, description, canonical URL, OpenGraph + Twitter card metadata, headings, and outbound links from a single URL. Server-side rendering. Body content rendered as text / raw HTML / clean markdown. Optional link extraction. Cheerio-based, no headless browser — fast and cheap, ideal for static pages and SSR sites. Alias of scrape-website. For JS-heavy SPAs that need a real browser, see website-screenshot.

INPUT — request schema

property	type	description	req?
url	string	Public URL to fetch and parse. Must include scheme (http/https). Follows redirects.	required
format	string	Body output format. 'text' (default), 'html' (raw), or 'markdown' (clean — best for LLM ingestion). enum: text · html · markdown	optional
include_links	boolean	If true, also returns an array of all <a href> links on the page. Default false.	optional
user_agent	string	Custom User-Agent header. Defaults to a modern desktop Chrome UA.	optional

OUTPUT — response shape

field	type	description
url	string	Final URL fetched after following redirects.
final_url	string	Final URL after redirects.
status_code	number	HTTP status code returned by the upstream server.
title	string	Page title from the <title> tag.
description	string	Meta description from the page's <meta name="description"> or OG description tag.
canonical	string	Canonical URL from <link rel="canonical"> when present.
lang	string	Language code from the <html lang> attribute, like en or fr.
h1	string	Text of the first <h1> heading on the page.
og	object	All og:* meta tags.
twitter	object	All twitter:* meta tags.
text	string	Body when format=text.
html	string	Body when format=html.
markdown	string	Body when format=markdown.
format	string	Output format of the body content: text, html, or markdown.
links	array	Array of {href, text} when include_links=true.
body_chars	number	Character count of the extracted body content.

EXAMPLES — two ways to call

EXAMPLE 1 · curl

curl -X POST https://x402.agentutility.ai/scrape \
  -H 'Content-Type: application/json' \
  -d '{ }'

first response = 402 Payment Required with payment requirements; sign + retry with X-PAYMENT.

EXAMPLE 2 · mcp

# Install the MCP package for this endpoint's cluster
npx -y @agentutility/mcp-<cluster>

# Required: EVM private key with USDC on Base
export X402_PRIVATE_KEY=0x...

# Then call the scrape tool from your MCP-aware agent.

MCP server handles payment automatically — your coding agent just calls the tool by name.

METADATA

tags: scrapefetchhtmlextractmetadataurl
methods: POST
cluster: webprobe
price: $0.04 USDC per call

ADJACENT — other endpoints in webprobe

endpoint	description	price
answer-web	Answers natural-language questions with live web research, returning a synthesized answer with inline [N] citations and the source URLs.	$0.04
arxiv-summarize	arXiv paper summarizer / research-paper TLDR.	$0.04
rss-from-anything	RSS feed generator / HTML to RSS converter.	$0.04
scrape-website	Scrapes any webpage and pulls title, description, canonical URL, OpenGraph + Twitter card metadata, headings, and outbound links from a s…	$0.04
screenshot	Website screenshot / URL to PNG/JPG.	$0.04
webpage-diff	Detects changes on a webpage: fetches a URL, strips HTML to plain text, computes a SHA-256 hash, and (when given a previous hash or text)…	$0.04
website-screenshot	Captures a website screenshot from a URL as PNG or JPG.	$0.04
arxiv-search	arXiv full-text search.	$0.03