Web scraping APIs: All Them APIs endpoint contract

Endpoint and access

Contract preview · access requires enablement

Web scraping APIs contract endpoint

POST https://allthemapis.com/api/v1/web-scraping/extract

This is our proposed customer-facing interface. Provider fulfillment and category-specific keys are not enabled yet. Calls currently return 503 integration_not_enabled; they do not execute a query or spend credits. Existing Search API trial keys do not unlock this category.

Request access for your workload before connecting an application.

Request fields

Proposed request schema
Field	Type	Meaning
url	HTTPS URL, required	Public URL you are authorized to retrieve. Private-network and credential-bearing URLs are not supported.
format	markdown \| text	The requested normalized main-content format.
render	auto \| static	Rendering preference, not a promise to bypass access controls.
max_age_seconds	integer	Maximum acceptable cache age; zero requests a fresh result.

Web scraping APIs example request

curl -X POST "https://allthemapis.com/api/v1/web-scraping/extract" \
  -H "Authorization: Bearer $ALLTHEMAPIS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com/",
  "format": "markdown",
  "render": "auto",
  "max_age_seconds": 0
}'

The example defines the contract; it is not a completed API call.

Response fields and errors

Intended response fields after access enablement
Field	Meaning
document	Main content in the requested format, or a structured extraction error.
source_url	Final source URL after allowed redirects.
retrieved_at	Timestamp of the underlying retrieval, not merely the API response.
request_id	Trace identifier; successful extraction must pass content checks.

Web scraping APIs current unavailable response

{
  "error": {
    "code": "integration_not_enabled",
    "message": "This API requires access enablement. No request has been executed or billed."
  },
  "access_url": "https://allthemapis.com/api-categories/web-scraping/api#request-access"
}

No synthetic response is returned as live data. Pricing, production quotas and content-use rights must be confirmed before activation.

Give the contract to your coding agent

This prompt prepares an adapter without treating a fixture as a successful live request.

Web scraping APIs agent integration prompt

Prepare an integration for the All Them APIs web scraping apis contract.

Endpoint: POST https://allthemapis.com/api/v1/web-scraping/extract
Request shape:
{
  "url": "https://example.com/",
  "format": "markdown",
  "render": "auto",
  "max_age_seconds": 0
}

This category requires access enablement and currently returns HTTP 503 integration_not_enabled. Do not execute requests or claim success until access is confirmed.
Read ALLTHEMAPIS_API_KEY from a server-side environment variable only. Never place it in browser code, URLs or logs. Use a bounded timeout, handle 401/429/503 explicitly, and keep null/missing fields distinct from valid zero values. Do not retry indefinitely.
Expected output fields:
- document: Main content in the requested format, or a structured extraction error.
- source_url: Final source URL after allowed redirects.
- retrieved_at: Timestamp of the underlying retrieval, not merely the API response.
- request_id: Trace identifier; successful extraction must pass content checks.

Build the adapter and unit tests against an explicitly labeled local fixture. Do not fabricate live results. Summarize what access and production verification are still required.

Request access to web scraping APIs

Tell us the coverage and volume you need. We will confirm availability and terms before enabling access. This request does not take a payment or create an API key.