How to benchmark a transactional email API without cheating
All Them APIs
Independent API research
Most transactional email API comparisons grade the easy part. They compare SDK counts, pricing tiers, and whether a five-line example exists. Then they call the product with the most checkmarks the winner.
That tells you almost nothing about whether a coding agent can integrate the API correctly.
The hard part begins after POST /send: finding the current documentation, choosing the right credential, satisfying sender requirements, preserving the requested recipient, recognizing a real acceptance response, and stopping when the account is not ready. A useful benchmark has to measure that path without quietly helping one provider more than another.
We designed our second transactional email protocol around a deliberately narrow question: can a coding agent start in a fresh Node.js repository and complete one verifiable send without changing the task or leaking a credential?
This is how we make that question reproducible.
Freeze the job before opening the dashboards
The benchmark prompt should be identical in outcome, even when provider prerequisites differ. Our job requires the agent to:
- find the current official integration path;
- implement one transactional send;
- read credentials and account identifiers from named environment variables;
- preserve an exact recipient, subject, and run ID;
- create a request-construction test;
- execute exactly one network attempt;
- record the provider's actual acceptance evidence or blocking error.
Freezing the job matters because email products invite post-hoc exceptions. Nylas needs a connected grant. AgentMail sends from an inbox. Mailgun uses a sending domain in the endpoint. Loops expects a published transactional template. If we rewrite the task after encountering each prerequisite, we stop comparing agent integration paths and start producing seven curated demos.
Prerequisites can differ. The requested outcome cannot.
Separate account preparation from the timed coding task
A provider cannot send from a domain that has never been verified. A mailbox API cannot act on an account that has never been connected. Those facts are part of setup friction, but waiting for DNS or completing OAuth inside a code timer produces a noisy measurement.
We therefore keep two clocks:
| Phase | Included in the coding timer? | Still published as evidence? |
|---|---|---|
| Account creation | No | Yes |
| Payment or card gate | No | Yes |
| Sender or domain verification | No | Yes |
| Mailbox grant or inbox creation | No | Yes |
| Documentation discovery | Yes | Yes |
| Implementation and local tests | Yes | Yes |
| One provider request | Yes | Yes |
This prevents DNS propagation from dominating the runtime while preserving the thing a buyer actually needs to know: whether setup required a card, a human click, a verified domain, a template, or an external mailbox grant.
Start every attempt in a clean repository
Reusing a workspace contaminates the result. The second agent benefits from the first agent's lockfile, package choice, source layout, and error corrections. Even a forgotten README can reveal the endpoint.
Every attempt should receive a new repository with no provider package installed and no previous generated code. The harness records the model identifier, agent CLI version, Node version, machine platform, protocol version, and timestamps. It keeps the resulting lockfile and workspace patch.
Three repetitions mean three fresh repositories, not three calls from one finished integration.
That distinction is expensive. It is also the difference between testing API latency and testing whether an agent can repeatedly build the integration.
Record the request without publishing the address
The evaluator needs to know that the agent did not replace the requested recipient with a convenient test address. Publishing a private inbox is not necessary to prove that.
Before the network call, the generated program writes a sanitized request record:
{
"provider": "example-provider",
"runId": "example-provider-2026-09-11-a1b2c3d4",
"recipientSha256": "sha256-of-normalized-runtime-recipient",
"subject": "All Them APIs Agent Bench example-provider-2026-09-11-a1b2c3d4",
"bodyIncludesRunId": true,
"attemptedAt": "2026-09-11T12:00:00.000Z"
}
The harness computes the expected recipient hash independently. It rejects a mismatch, a changed subject, or a body that omits the run ID. The request-construction test is then rerun outside the agent session.
This is not cryptographic proof that a remote provider received those exact bytes. A controlled proxy or provider webhook would give stronger evidence. It does catch the common benchmark failure where the agent silently changes the recipient and still reports success.
"Accepted" and "delivered" are different claims
Email APIs do not expose one universal success shape. Resend returns an email ID. Postmark returns a MessageID with an ErrorCode. SendGrid commonly acknowledges a Mail Send request with HTTP 202. Loops returns success: true but does not provide the same message-ID contract.
A fair evaluator uses provider-native acceptance evidence. It should not require a field the API does not claim to return.
It should also stop there. A message ID proves that the provider accepted a request. It does not prove inbox delivery, inbox placement, or long-term deliverability. Those require delivery events, a controlled receiving inbox, and a separate protocol.
Our Email API ranking keeps task evidence and product facts visible, but readers should not interpret request acceptance as a deliverability score.
A blocked run is not a zero
We preserve four outcome classes:
| Outcome | Meaning |
|---|---|
| Passed | The harness is valid and the provider returned its declared acceptance evidence |
| Failed | The harness is valid, but the implementation or provider request failed |
| Blocked | An observed account prerequisite prevented the send |
| Harness error | The attempt cannot support a conclusion because tests, artifacts, recipient integrity, secret handling, or the agent execution failed |
Collapsing these into a numeric score hides the useful part. A missing Nylas grant says something different from malformed request code. A sender-verification block says something different from an agent that invented a response. A leaked API key invalidates the run even if the provider accepted the message.
The benchmark should make those distinctions more obvious, not average them away.
Test once, then repeat the whole path
The publication sequence is intentionally cautious:
- run offline evaluator tests;
- inspect provider readiness without sending;
- perform one smoke attempt per provider;
- review every trace and generated workspace;
- run three fresh attempts per provider;
- publish sanitized artifacts and every outcome.
There is no silent retry inside an attempt. Retries can turn a transient failure into a pass while erasing the corrective work an agent had to perform. If a harness defect is found, we fix it, increment the appropriate version, and rerun every provider.
What this protocol still cannot tell us
This benchmark is about autonomous integration and request acceptance. It does not answer:
- which provider has the best deliverability;
- which is fastest across regions;
- how templates perform at production scale;
- whether webhooks arrive reliably;
- whether a provider is cheapest for a particular volume;
- whether one successful integration remains correct after an SDK update.
Those are legitimate questions. Combining them into this test would make the result broader and less reproducible.
The useful conclusion is smaller: when seven products all claim to send transactional email, we can test whether the same coding agent can discover, implement, validate, and execute that job without receiving a provider-specific rescue. The Agent Bench library is where we publish those runs once the evidence survives review.