Reproducibility protocolVersion 1.0.0

Email API agent-readiness study

The commands, sandbox, prompt, pass criteria, and artifacts used for the All Them APIs email benchmark.

Published Aug 17, 2026 Machine-readable protocol Published runs complete

Protocol JSON Prompt text

CompletedSix provider task or setup reports

Not yet testedLoops; documentation review only

01 · Exact task

Prompt given to the agent

Only the provider name, credential variable, and test recipient change between runs.

You are working in a new Node.js repository. Integrate the {{PROVIDER}} email API and complete this task:

1. Install the provider's official server-side SDK when one exists; otherwise use its documented REST API.
2. Read credentials only from {{CREDENTIAL_ENV}}. Never print, copy, or commit the credential.
3. Implement a script that sends one transactional verification email to {{TEST_RECIPIENT}} with the subject "All Them APIs verification" and a unique run ID in the body.
4. Use the provider's documented sandbox, test mode, or non-delivery recipient when available. Do not silently replace the requested recipient or return demo data.
5. Add an automated test for request construction and a README containing install, run, and cleanup commands.
6. Run the test and the script. Record the provider message ID or the exact blocking error.

Success requires a provider-confirmed message ID or a documented non-delivery test response, passing local tests, no secret in the repository, and no change to the requested behavior. If authorization or account state blocks the task, stop and report the exact step.

02 · Environment

Fresh, disposable sandbox

The agent starts from an empty repository. Credentials are injected, never placed in the prompt or repository.

Base image: node:22-bookworm-slim, pinned by digest for a published run
Resources: 2 vCPU, 4 GB RAM, 10-minute wall-clock limit
Filesystem: Fresh writable repository; destroyed after artifact export
Network: Public web and provider API allowed; private network ranges blocked
Credentials: One pre-provisioned least-privilege key or grant, injected as an environment variable
Packages: No provider SDK preinstalled; package and lock files preserved
Human help: Only account authorization or CAPTCHA; every intervention is timestamped and counted
Runs: One fresh run per provider in v1; all failures and blocked outcomes retained

03 · Invocation

Commands run inside the container

Claude Code runs non-interactively, emits a stream JSON trace, and receives unrestricted filesystem permissions only inside the disposable container.

mkdir -p artifacts workspace
cd workspace
git init --quiet
npm init -y >/dev/null

claude -p "$(cat ../prompt.md)" \
  --model "$CLAUDE_MODEL_ID" \
  --max-turns 40 \
  --output-format stream-json \
  --verbose \
  --dangerously-skip-permissions \
  | tee ../artifacts/trace.jsonl

npm test 2>&1 | tee ../artifacts/test.log
git diff --no-index /dev/null . > ../artifacts/workspace.diff || true

Claude Code CLI reference

04 · Evaluation

A passing process is not enough

A run fails when it changes the requested behavior, returns demo data, leaks a credential, or cannot produce the provider's confirmation.

Success criteria

Provider-confirmed message ID or documented non-delivery test response

Requested recipient behavior is preserved

Local request-construction test passes

Credential remains outside source, logs, and generated artifacts

Install, execution, and cleanup instructions are present

Required artifacts

Prompt and immutable model ID

Container image digest and resource limits

Stream JSON agent trace

Final workspace diff and lockfile

Test output and provider response with secrets removed

Timing, token, cost, tool-call, error, and intervention counts

Evaluator decision with a written failure reason

05 · Current status

Published benchmark runs

Completed, failed, and blocked outcomes all count as finished runs. Documentation-only providers remain explicitly untested.

RankProviderEvidenceRun status

#1ResendTask + setup runsCompleted

#2AgentMailTask + setup runsCompleted

#3PostmarkSetup runCompleted

#4MailgunSetup runCompleted

#5LoopsDocumentation reviewNot yet tested

#6NylasTask runCompleted; failed outcome retained

#7SendGridSetup attemptCompleted; blocked outcome retained

Versioned for reproducibility.

Each result retains the prompt, success rule, and sandbox version used for its run. Any revision receives a new version instead of rewriting the original record.