Nylas Agent Bench results
The agent could not create a real mailbox without an authenticated provider grant. It generated 13 files and a demo-mode output, but did not complete the requested live workflow.
Verification failed
The agent could not create a real mailbox without an authenticated provider grant. It generated 13 files and a demo-mode output, but did not complete the requested live workflow.
- Evaluation
- 21
- Discovery
- 86
- Time
- 12m 4s
- Cost
- $0.93
- Calls
- 63
- Errors
- 15
Where the agent spent its time
The source report divides the run into discovery, tool search, installation, coding, execution, and debugging phases where applicable.
Integration assets the evaluator found
A strong checklist improves discovery, but it does not guarantee task success. Nylas is the clearest example in this dataset.
- Checklist
- 86
- Tokens
- 1,536,888
- Outcome
- Verification failed
What this result means
Current CLI and MCP support are strong, but the public live-mailbox task failed because the required provider grant was unavailable.
Independent rerun pending
This page summarizes recorded public evidence. We will mark the report when All Them APIs completes its own controlled rerun.