How we benchmark Search APIs without grading vibes

Comparing Search APIs gets hand-wavy fast. One returns ten links. Another returns excerpts. A third attaches dates to everything. Each provider has a different request shape, a different idea of a result, and a perfectly reasonable reason why its preferred output is the useful one.

If you score those native responses directly, you mostly measure how closely each API resembles the benchmark author's favorite API.

We wanted a narrower test: can the API retrieve a current primary source for a concrete technical question, with enough structured context for another program to inspect the result? This post explains how we implemented that test, including the decisions that made it less impressive but more reproducible.

We froze the claim before we wrote the score

The benchmark is called search-agent-retrieval-v1, not “best Search API.” That name is intentionally boring. It measures retrieval, not autonomous account creation, API integration, answer synthesis, global reliability, or whether an agent can recover from a billing problem.

The frozen protocol contains three technical research jobs:

1. Find current Node.js release lines and identify the LTS releases.
2. Find the current GitHub-hosted Actions runner images and their included software.
3. Find the PostgreSQL versions currently supported by the project and its support policy.

Each job names an expected first-party domain. That gives the evaluator something objective to check without asking a language model whether a result “looks relevant.”

{
  id: "postgres-supported-versions",
  query:
    "Find the PostgreSQL versions currently supported by the PostgreSQL project " +
    "and the official support policy. Prefer current primary sources.",
  expectedDomains: ["postgresql.org"],
  userJob: "Retrieve the official supported-version and release-policy pages."
}

This is not a representative sample of the entire web. It is a reproducible sample of one thing coding agents do often: locate maintained technical documentation.

The normalization layer is where most benchmark bias hides

The tested providers expose different native response fields. Perplexity returns a snippet; Exa can return highlights; Parallel returns excerpts. Their date fields differ too.

We normalize every result to the smallest shared record that supports the test:

type NormalizedSearchResult = {
  title: string
  url: string
  snippet: string
  publishedAt: string | null
  updatedAt: string | null
}

The adapters are deliberately dull. An Exa result, for example, prefers joined highlights and falls back to a summary or text when needed:

return body.results.map((result) => ({
  title: result?.title ?? "",
  url: result?.url ?? "",
  snippet:
    (Array.isArray(result?.highlights) ? result.highlights.join("\n") : "") ||
    result?.summary ||
    result?.text || "",
  publishedAt: result?.publishedDate ?? null,
  updatedAt: null,
}))

Normalization loses information. Parallel's excerpts are not semantically identical to Perplexity's snippets, and equal result counts do not imply equal content budgets. We document that mismatch instead of pretending it disappears once the JSON keys match.

Pass/fail is a guardrail, not a ranking

An attempt passes when all of the following are true:

• the HTTP request succeeds;
• at least five results contain valid URLs;
• an expected primary-source domain appears in the top five;
• at least 80% of results contain a non-empty snippet.

The evaluator does not use an LLM:

passed:
  requestOutcome.ok &&
  validUrlCount >= 5 &&
  bestPrimarySourceRank !== null &&
  bestPrimarySourceRank <= 5 &&
  snippetCoverage >= 0.8

This produces a test we can rerun without model drift or a hidden judge prompt. It also produces a coarse result. In the first published run, every provider passed every attempt.

That is not a failure of the test. The pass condition answers a bounded question: did the API clear the minimum retrieval bar? It should not be forced to manufacture a winner.

We repeat cases because one clean request proves very little

Each provider runs each case three times, for nine requests per provider. A case passes when at least two of its three attempts pass.

Three repetitions are still a small sample. They are enough to expose an intermittent miss or a large timing outlier, but not enough to support an uptime or global latency claim. We report medians and the observed high value without attaching statistical confidence they do not have.

The runner spaces requests by 1.1 seconds, applies a 30-second timeout, and uses provider-default geography. A full run looks like this:

node scripts/search-agent-bench/run-exa.mjs --label launch-v1

Credentials are read from provider-specific environment variables. The manifest records the variable name, Node version, platform, endpoint, cases, and timestamps. It records credentialStored: false; the key itself never enters the artifact.

A benchmark without artifacts is a screenshot

For every request, the runner writes three files:

artifacts/search-agent-bench/<run>/
  protocol.json
  manifest.json
  attempts.json
  summary.json
  <case-id>/
    run-01/
      raw-response.json
      normalized-results.json
      evaluation.json

The raw files make local debugging possible, but they are not all suitable for publication. Provider responses and headers can contain request, search, session, account, or rate-limit identifiers.

The publishing step creates a separate sanitized bundle. It retains normalized titles, URLs, dates, evaluation fields, and snippet excerpts capped at 280 characters. It removes raw responses, headers, credentials, and provider identifiers.

This split turned out to be important. “Reproducible” should not mean “dump every byte a vendor returned onto a public URL.”

The first run exposed the limits of our own score

Exa, Perplexity Search, and Parallel Search all passed 9/9 attempts, and every expected primary source appeared at rank one. The secondary measurements differed:

Provider	Median latency range	Date coverage	Snippet coverage
Exa	130–171 ms	33%	100%
Perplexity Search	427–1,045 ms	100%	100%
Parallel Search	1,732–2,044 ms	49%	100%

Those values are observations from one machine and one dated run. Exa's lower medians do not establish global performance. Perplexity's complete date fields do not establish better relevance. Parallel's slower response does not tell us whether its broader excerpts reduce later tool calls.

The benchmark can show those differences. It cannot assign their value without knowing the workload.

What we would add to a second protocol

The next version should not simply add more providers. It should add failure pressure:

• ambiguous queries with multiple plausible first-party sources;
• time-sensitive questions where stale results are measurably wrong;
• queries that require domain or date filtering;
• repeated runs from more than one region;
• duplicate-result and canonicalization checks;
• a downstream task that measures whether the retrieved fields reduce later fetches.

We would keep that work in a new protocol version. Editing thresholds after seeing results is an easy way to turn a benchmark into a story about the answer you wanted.

The current protocol and sanitized result bundles are public. The important part is not that our thresholds are uniquely correct. It is that they are specific enough to disagree with, rerun, and replace.