The tests we ran ourselves
Our August 23 retrieval run asked for known Node.js, GitHub Actions and PostgreSQL primary sources. Each provider received three tasks three times. Source order, response coverage and request latency were recorded; no answer model was graded.
| Provider | Attempts passed | Case-median range | Date coverage |
|---|---|---|---|
| Perplexity Search | 9/9 | 427–1,045 ms | 100% |
| Exa | 9/9 | 130–171 ms | 33% |
| Parallel Search | 9/9 | 1,732–2,044 ms | 49% |
The tie is meaningful: these tasks established a common floor, not a universal winner. Read the protocol and linked reports for exact queries and artifacts.
Published benchmark register
Each source below is a separate evaluation, not another column in one combined leaderboard.
NEEDLE by Keenable
Vendor-published benchmark · Snapshot reviewed 2026-10-27
Rolling retrieval evaluation across five subject areas, with ranking, recall and successful-call latency.
Keenable operates a competing search service. Rolling windows change; do not mix its p50 with another publisher's mean.
Artificial Analysis Search Index
External evaluation · Publisher data dated 2026-10-19; reviewed 2026-10-27
Search tools used by a shared answer-model harness across three benchmark families.
Task time includes model work. Search cost per 1,000 tasks is not list price per 1,000 API calls; visible rows can change.
Openbenchmarks company news
External benchmark with open issue · Publisher last measured 2026-10-21
Company-event questions against general search tools and specialist news indexes. Includes Context.dev.
Publisher flags timing/indexing bias and says not to compare vendor performance until a same-window rerun. Scores are not a recommendation.
Exa open benchmark harness
Vendor-published benchmark · Repository reviewed 2026-10-27
Retrieval and agentic tasks with configurable provider adapters and datasets.
Adapters and models evolve. Check the exact mode and commit before quoting a result.
Parallel evaluation notes
Vendor-published benchmark · Published 2026-10-26
Explains agent-answer scoring, frontier/low-cost tiers and differences from external boards.
Not every reproduction detail is public; its agent/task costs are not direct-request prices.
Tako VerticalRTK
Vendor-published benchmark · July 2026 run; reviewed 2026-10-27
Time-sensitive questions with a shared synthesis and judging step.
A real-time answer workload, not a replacement for a plain retrieval latency test.
External latency: a dated snapshot, not our rerun
NEEDLE's rolling successful-call latency table, reviewed October 27, reported the following selected modes. Its publisher, Keenable, also sells a competing search API. The values are per-call percentiles, not completed-agent-task times.
| Exact configuration | p50 | p95 |
|---|---|---|
| Keenable | 139 ms | 341 ms |
| Perplexity Fast | 162 ms | 234 ms |
| Octen | 204 ms | 399 ms |
| Exa Instant | 489 ms | 770 ms |
| Brave Web | 503 ms | 739 ms |
| Context.dev | 993 ms | 4.0 s |
Only successful requests contribute to that latency view. Check the publisher's failure column as well. A provider returning errors quickly or dropping slow requests can look attractive in a percentile table while failing the user's task.
Our fastest search APIs article explains why these rows cannot simply be sorted alongside our case medians.
The Context.dev benchmark trail
Context.dev is present in NEEDLE and Openbenchmarks' company-news board. The latter lists a Search request with highlights and ten results; its visible Context row reports 98.0% extracted-answer accuracy and 15.93 seconds latency. Those are the publisher's observations, not our tests.
Do not use that row to rank vendors. The company-news publisher flags a timing/indexing bias: vendors tested later had more opportunity to index the target events. It explicitly asks readers not to compare vendor performance until same-window reruns are complete.
We have not verified a separate Context-owned general web-search leaderboard. The Context review instead connects its documented Search contract and credit costs to the actual external sources found. Context's newer News product is a specialist endpoint, not automatically the same configuration as its Web Search benchmark.
What to record before quoting a score
- Publisher and incentive. Vendor-produced research can be useful without being independent.
- Measured object. Ranked retrieval, extracted text and a model's final answer are different outputs.
- Configuration. Mode, result count, content limits, model, geography and cache policy can change the result.
- Window and denominator. Include dates, failed attempts and repeat counts. A moving board is a snapshot.
- Unit. Mean is not p50. Dollars per thousand tasks is not dollars per thousand requests. API cost is not API-plus-model cost.
The Artificial Analysis Search Index evaluates search-enabled answers. Its model work and task-time accounting must not be read as a raw endpoint speed test. Parallel's own evaluation notes similarly distinguish agent outcomes from request cost. Use the source that matches the question you are trying to answer.
Sources and review dates
Documentation review: October 27, 2026. Our own measured retrieval runs: August 23, 2026. External reports retain their publisher and measurement window.
- All Them APIs retrieval run
- NEEDLE by Keenable
- Artificial Analysis Search Index
- Openbenchmarks company news
- Exa open benchmark harness
- Parallel evaluation notes
- Tako VerticalRTK
Corrections: email faizank@docsalot.dev with an official source.