Agent Bench
Agent Bench result
Nylas Agent Bench results
One coding-agent task run, preserved with its recorded outcome and caveats.
D40/100
Task run · 2026-05-19
Verification failed
The agent could not create a real mailbox without an authenticated provider grant. It generated 13 files and a demo-mode output, but did not complete the requested live workflow.
- Evaluation
- 21
- Discovery
- 86
- Time
- 12m 4s
- Cost
- $0.93
- Calls
- 63
- Errors
- 15
Run trace
Where the agent spent its time
The trace divides the run into discovery, tool search, installation, coding, execution, and debugging phases where applicable.
01Discovery
11 calls~3m 12s
02Tool search
1 calls~5s
03Install
1 calls~12s
04Code
14 calls~1m 33s
05Run
27 calls~5m 12s
06Debug
9 calls~1m 49s
Editorial interpretation
What this result means
Current CLI and MCP support are strong, but the Agent Bench live-mailbox task failed because the required provider grant was unavailable.
The evaluator recorded a failure. Generated code or demo output did not satisfy the live task.
Benchmark record
Run completed and preserved
This page reports the recorded All Them APIs run, including failures, interruptions, and caveats.