Reproducibility protocolVersion 1.0.0

Email API agent-readiness study

The commands, sandbox, prompt, pass criteria, and artifacts required to reproduce the ranking inputs. The current ranking uses public benchmark evidence; our credentialed reruns are still pending.

Published Aug 17, 2026 Machine-readable protocol Reruns pending

Protocol JSON Prompt text

ReproducedSource reports, provider documentation, and ranking inputs

Not yet reproducedCredentialed agent execution in the published sandbox

01 · Exact task

Prompt given to the agent

Only the provider name, credential variable, and test recipient change between runs.

You are working in a new Node.js repository. Integrate the {{PROVIDER}} email API and complete this task:

1. Install the provider's official server-side SDK when one exists; otherwise use its documented REST API.
2. Read credentials only from {{CREDENTIAL_ENV}}. Never print, copy, or commit the credential.
3. Implement a script that sends one transactional verification email to {{TEST_RECIPIENT}} with the subject "All Them APIs verification" and a unique run ID in the body.
4. Use the provider's documented sandbox, test mode, or non-delivery recipient when available. Do not silently replace the requested recipient or return demo data.
5. Add an automated test for request construction and a README containing install, run, and cleanup commands.
6. Run the test and the script. Record the provider message ID or the exact blocking error.

Success requires a provider-confirmed message ID or a documented non-delivery test response, passing local tests, no secret in the repository, and no change to the requested behavior. If authorization or account state blocks the task, stop and report the exact step.

02 · Environment

Fresh, disposable sandbox

The agent starts from an empty repository. Credentials are injected, never placed in the prompt or repository.

Base image: node:22-bookworm-slim, pinned by digest for a published run
Resources: 2 vCPU, 4 GB RAM, 10-minute wall-clock limit
Filesystem: Fresh writable repository; destroyed after artifact export
Network: Public web and provider API allowed; private network ranges blocked
Credentials: One pre-provisioned least-privilege key or grant, injected as an environment variable
Packages: No provider SDK preinstalled; package and lock files preserved
Human help: Only account authorization or CAPTCHA; every intervention is timestamped and counted
Runs: Three fresh runs per provider; median reported, all failures retained

03 · Invocation

Commands run inside the container

Claude Code runs non-interactively, emits a stream JSON trace, and receives unrestricted filesystem permissions only inside the disposable container.

mkdir -p artifacts workspace
cd workspace
git init --quiet
npm init -y >/dev/null

claude -p "$(cat ../prompt.md)" \
  --model "$CLAUDE_MODEL_ID" \
  --max-turns 40 \
  --output-format stream-json \
  --verbose \
  --dangerously-skip-permissions \
  | tee ../artifacts/trace.jsonl

npm test 2>&1 | tee ../artifacts/test.log
git diff --no-index /dev/null . > ../artifacts/workspace.diff || true

Claude Code CLI reference

04 · Evaluation

A passing process is not enough

A run fails when it changes the requested behavior, returns demo data, leaks a credential, or cannot produce the provider's confirmation.

Success criteria

Provider-confirmed message ID or documented non-delivery test response

Requested recipient behavior is preserved

Local request-construction test passes

Credential remains outside source, logs, and generated artifacts

Install, execution, and cleanup instructions are present

Required artifacts

Prompt and immutable model ID

Container image digest and resource limits

Stream JSON agent trace

Final workspace diff and lockfile

Test output and provider response with secrets removed

Timing, token, cost, tool-call, error, and intervention counts

Evaluator decision with a written failure reason

05 · Current status

What has actually been reproduced

“Source values checked” means we reconstructed the published evidence and verified the supporting provider material. It does not mean we reran the credentialed task.

RankProviderCurrent evidenceAll Them APIs reproduction

#1ResendPublic task + setup runsRecorded values checked; credentialed rerun pending

#2AgentMailPublic task + setup runsRecorded values checked; credentialed rerun pending

#3PostmarkPublic setup runRecorded values checked; credentialed rerun pending

#4MailgunPublic setup runRecorded values checked; credentialed rerun pending

#5LoopsFirst-party integration sourcesDocumentation checked; first independent run pending

#6NylasPublic task runRecorded values checked; credentialed rerun pending

#7SendGridBlocked public setup runBlocked result checked; credentialed rerun pending

Why publish before rerunning?

Publishing the protocol first prevents us from changing the prompt, success rule, or sandbox after seeing a provider's result. Any revision will receive a new version and retain the previous protocol.