Inspectable AI quality
Evaluate the integration, not the confidence.
Run one deterministic, credential-free benchmark across Mappls MCP, A2A, the Build with Mappls Agent Skill, and every stateful journey. Every score resolves to an exact case, JSON path, expected value, and observed value.
Run locally or in CI
One JSONL response contract. One explainable score.
The ZIP contains the exact cases, JSON Schemas, a zero-dependency Node.js runner, and a reference response set. Feed each prompt and input to your system, normalize structured output, then score it without sending developer data to this portal.
Default evaluation requires all 90 cases and rejects unknown or duplicate identities. Partial runs are an explicit adapter-development mode, never a complete score.
curl -fsSLO https://developer.mappls.com/downloads/mappls-ai-evaluation-lab.zip
curl -fsSLO https://developer.mappls.com/downloads/mappls-ai-evaluation-lab.zip.sha256
shasum -a 256 -c mappls-ai-evaluation-lab.zip.sha256
unzip mappls-ai-evaluation-lab.zip
cd mappls-ai-evaluation-labExact coverage
Protocol behavior and application behavior.
Counts distinguish normal protocol cases from adversarial boundary tests, so a larger number cannot hide a missing authority path.
MCP
32 offline tools + 10 opt-in live reads
A2A
3 offline skills + 10 opt-in live reads
Agent Skill
All five evidence workflow gates
Stateful journeys
Every complete application lifecycle
Hostile boundaries
Credential, write, URL, and approval attacks
Six quality dimensions
Failures that tell you what to fix.
This suite measures deterministic integration contracts. Open-ended model usefulness, style, and domain judgment need a separate human or model-graded layer.
Discovery
Select an exact advertised tool or skill; never infer a nearby capability.
Grounding
Bind identity, profile, and evidence to the canonical contract before implementation.
Authority
Keep credentials server-held, provider payloads bounded, writes absent, and production approval independent.
State
Require stable identity, idempotency, concurrency, reconciliation, audit, and recovery.
Resilience
Reject hostile write and trust-boundary escalation with exact, reviewable failure codes.
Handoff
End with explicit qualification and human release gates—not model self-approval.
Failure shape
A score is the index—not the diagnosis.
Every failing case reports its surface, profile, dimension, assertion count, JSON path, expected scalar, and actual scalar. Keep those details in CI artifacts and review regressions at the boundary they affect.
{
"caseId": "boundary-a2a-provider-write",
"passed": false,
"failures": [{
"path": "decision",
"expected": "reject",
"actual": "allow"
}]
}Evaluation authority
Deterministic proof, deliberately bounded.
The lab evaluates deterministic structured contracts and negative authority. It does not grade open-ended model quality, call Mappls providers, accept credentials, prove entitlement, or grant production approval.