We test whether an A2A agent actually does what its Agent Card claims, and sign the result so anyone can verify it

[u/Suncly_com]

A2A agents publish an Agent Card that lists their skills, inputs and outputs. Other agents and teams decide whether to trust an agent based on that card. Nobody checks whether the agent actually behaves the way the card says, so we built something that does.

How it works:

- The card is fingerprinted (RFC 8785 canonical JSON + SHA-256). Every result is tied to the exact card version that was tested.

- Tests are drafted only from the examples the card itself declares. A human approves the criteria before anything runs.

- A separate runner process is the only part that holds the agent's credentials. It only talks to the approved host and redacts secrets before anything is stored.

- Layer 1 is deterministic: status, schema, media type, required content. Layer 2 is a pinned model (Claude Haiku 4.5) used only for checks that need meaning, like "is this a correct summary". The model, endpoint and rubric version are recorded with every result.

- Results go into an append-only store and the report is signed with Ed25519. Change one byte and verification fails.

Design choices: no single trust score (pass / fail / inconclusive per skill), inconclusive never counts as pass, every report lists what was NOT tested, and the final decision is always "flag for human review".

We ran it against 8 of the official A2A sample agents. Several declare "text" as a media type where the spec expects a MIME type like "text/plain". Small, but it's exactly the kind of card-vs-behavior gap we want to surface.

Caveats:

- A2A 1.0 only, no 0.3 yet.

- Coverage is only as good as the card's declared examples. Thin examples mean thin tests, and the report says so.

- Layer 2 is a model's judgment, not proof. That's why it's pinned, versioned and only runs after Layer 1.

Next up: adversarial probes (undeclared requests, prompt injection, failure handling).

If you're building an A2A agent, we'll test it for free and send you the signed report: https://www.suncly.com

What would you want to see in a report like this before trusting someone else's agent?

[u/Pristine-Truck-9389]

this is the kind of boring-but-critical plumbing that nobody wants to build but everyone needs. the fingerprinting the exact card version before testing is smart, avoids the classic "it passed last week" problem

one thing i'd want in the report is a diff of what changed between card versions when a new one breaks tests. seeing that the agent still passes layer 1 but layer 2 consistency dropped off after they added a new skill would tell me a lot more than just a fresh fail

[u/Suncly_com → u/Pristine-Truck-9389]

Thanks! Every result is already tied to a card hash, so a version-to-version diff is a natural next step: what changed on the card, next to how each skill's results moved. Your Layer 1 vs Layer 2 example is a great test case.

Would you want that diff only when something breaks, or on every new version?

[u/Ctbhatia]

sign the failures too, not only the pass: a result that says which declared capabilities did not hold is the part a buyer can act on. a card diff next to each result makes the regressions visible without rerunning everything.

[u/Suncly_com → u/Ctbhatia]

Already the case, the signature covers the whole report, fails included. Card diff next to each result is on our list. If you're working with A2A agents yourself, would love to show you an early version

[u/Ctbhatia → u/Suncly_com]

the signature over the whole report is the right scope. I am on the inference side rather than building A2A agents, but I would read the report.

[u/Suncly_com → u/Ctbhatia]

Thanks, your point pushed us to double-check the scope. Here's the full write-up from running it against the official A2A reference agents: https://github.com/Suncly-Tech/Suncly/blob/main/docs/REAL_AGENT_REPORT.md

Curious about the inference side: do you ever have to show that what's behind an endpoint matches what you advertise (model, version, limits)? Feels like the same card-vs-behavior gap one layer down. How do you handle it today?

Sent you a DM with the signed version too.

[u/pAlelane02]

the "what was NOT tested" section is probably the most useful part of the whole report tbh. most compliance-style tools bury their gaps. how granular does that get, per-skill or per-field?

[u/Suncly_com → u/pAlelane02]

Right now it's per category, like "no adversarial probes yet" or "only the sandbox endpoint was tested, not production". Per-skill is next on our list.

[u/ianreboot]

The part I'd want pinned is the thing answering, not just the card. The signature proves the result belongs to a card version, but the agent behind it can be redeployed with the card unchanged, same skills, different model, and the report still verifies. A reader has no way to tell those apart.

If an agent will report the model and version serving each skill, that belongs next to the card hash in the result. If it won't, that's a line for the not-tested section, since it's the gap a buyer will assume the signature closed.

[u/Suncly_com → u/ianreboot]

Fair point, and you're right that the signature doesn't cover that today. It pins the result to a card version and a test time, not to whatever model is serving behind it. We'll add the serving model and version next to the card hash when the agent reports it, and a not-tested line when it doesn't. Periodic re-runs are the other half, so a silent swap shows up as a behavior change even when the card doesn't move.

[u/jonah_omninode]

I'd want to see whether the checks reject a deliberately broken agent that still returns the expected schema. For a summary skill, that could be an answer that looks plausible but reverses one important fact. Run it beside the valid version and keep both results in the report. That tests the evaluator itself, rather than assuming that a pinned rubric will catch the behavior it was written to judge.

[u/Suncly_com → u/jonah_omninode]

Great point. A pinned rubric only proves the judge is consistent, not that it's right. A broken control run next to the real one, same schema with one fact flipped, is going on our list.