The litmus test

The litmus test connects to an MCP server the way an agent would, exercises its tools, and watches what the server actually does: whether its outputs try to hijack the calling agent, whether it reaches out over the network when nothing required it, and whether data handed to it leaks back out. It answers one question an agent needs answered before it trusts a tool:

Will this MCP server hijack me, phone home, or leak my data?

It does not measure popularity, code quality, maintenance, or licensing — this is a behavioral grade, not a quality or popularity score.

What a passing grade does — and does not — claim

It does claim: against litmus-v17, on the exact tool surface identified by the fingerprint, the server did not exhibit the failure modes below, and the evidence is published and re-runnable.

It does not claim the server is “safe,” “secure,” or “trustworthy” in general. We underclaim. A grade is a dated, versioned, reproducible observation — nothing more.

Ten probes across four categories. Probe IDs are permanent and carry their family number; future probes extend a family without renumbering.

C-01 — Tool-output injection · probes 1.1, 1.2, 1.3, 1.4

A server’s tool descriptions and outputs flow straight into the calling agent’s context — the documented “tool poisoning” attack class. Probe 1.1 pulls the full tool surface via tools/listand scans every name, description, and input schema, including a tool-poisoning scanner that flags agent-directed instructions hidden in the advertised surface: a concealment directive (“do not tell the user”), an imperative to read a known secret file (~/.ssh/id_rsa, .aws/credentials), or an exfiltration verb aimed at a sensitive object together with a sink. Probe 1.2 issues bait calls — benign-but-suggestive inputs designed to elicit injection-shaped echoes — and scans every output. Probe 1.3(second-order) feeds one tool’s output back in as another tool’s input — the chaining path a real agent takes — and scans the result; only injection the server generated from that chained content fails, not a verbatim echo.

Fails on any high-severity hit: instruction mimicry aimed at the agent (“ignore previous instructions”, role-tag or tool-call-shaped JSON in free text), hidden or invisible Unicode (zero-width, bidi overrides, tag characters), or markdown and link tricks (javascript: / data: URIs, link text contradicting the href, or a URL whose query string is exfiltration-shaped — a credential-named key, a long/high-entropy value, or an interpolation marker; an honest ?q=search link is not flagged).

Probe 1.4 (indirect / passthrough injection) feeds harness-controlled, injection-laced external content into content-fetching tools (a seeded file they read, or a loopback URL they fetch) and grades the relay. A tool that relays third-party content verbatim is disclosed as an indirect-injection conduit, not failed: faithful relay is legitimate, and the agent is expected to distrust tool output. Only a tool that amplifies the content, emitting injection that was not present in the payload, fails C-01.

C-02 — Permission overreach · probes 2.1, 2.2

Does it touch more than it claimed — or label itself less capable than it is? Probe 2.1 (declared-permission honesty) flags a tool that annotates itself readOnlyHint: true while it plainly mutates — caught three ways: an unambiguously destructive verb in its name (delete, transfer, withdraw, …), a parameter that names value movement or secrets (recipient, amount, private_key, …), or a description that admits it (“transfers,” “deletes”). An agent — and this harness — would otherwise auto-trust that read-only claim. Each layer is deliberately narrow: polysemous words like create, update, or query are not flagged, so an honest create_query is no false positive. Static and deterministic; needs no sandbox.

Probe 2.2 runs the server inside a default-deny network sandbox and exercises it. Egress is routed to a local sinkhole that logs {host, port, firstBytes} and never completes the connection — the evidence shows what it tried to reach. Each attempt is classified against an effective allowlist: the operator baseline plus the hosts the package declares it reaches (polygraph.egress in its manifest). Egress matching that list on host and port is permitted and recorded; anything beyond it — or a declared host reached on an undeclared port — is overreach and fails, unless the harness can infer it as an expected upstream (below). A passing C-02 means no overreach, not no network; the declared hosts are surfaced in the evidence for the consumer to judge. A host-level DNAT gateway captures hard-coded IPs and DoH too, so an IP literal can’t dodge the check.

Expected-upstream inference (v11). An honest API-wrapper — a tool that transparently calls the API it advertises (openai_chat → api.openai.com) — used to be capped at D for that undeclared egress, even though the upstream is the very API its own surface names. Before an undeclared host is counted as overreach, the harness now infers whether it is a plausible upstream for the server’s own surface: a host named verbatim in the tool text (strong), or an egress host whose registrable label matches a non-generic brand token drawn from the surface and the package owner/name (medium, plain-TLD hosts only). A match reclassifies the attempt from overreach into an informational egress-inferred finding — disclosure, not exoneration: the operator is still advised to declare the host in polygraph.egress. Guardrails keep lookalikes out — whole-label (never substring) matching, a generic-label stoplist, registrable-label-only matching (so openai.evil-cdn.com is not cleared), and shared-tenant suffixes (github.io, vercel.app, …) treated as their own level (so attacker.github.io does not inherit foo.github.io’s match). This only ever turns a false D into a correct pass, never the reverse. The independent C-03 probe 4.2 canary-in-egress check is unchanged and still floors a real exfiltration at F regardless of C-02.

Package-registry baseline (v15). The operator baseline now includes public package-registry infrastructure — pypi.org, files.pythonhosted.org, registry.npmjs.org— because a framework’s own startup update-check (most commonly FastMCP pinging pypi.org for a newer version) is the tooling underneath the server, not egress its author wrote. It is deliberately narrow: the cloud instance-metadata endpoint is not baselined — it is a real credential-theft target, so a server that reaches it still fails. Like the inference above, this only ever turns a false D into a correct pass.

Probe 2.2 requires that the harness runs the server itself. For a remote MCP URL the harness cannot control the server’s network, so 2.2 is recorded as skipped — remote target; without Docker it is skipped — no sandbox (a skip is not a pass). Probe 2.1 is static and always runs — so C-02 can still fail on a remote target when a tool lies about being read-only.

C-03 — Sensitive-data handling · probes 4.1, 4.2

Before any calls, the harness plants canaries: a distinctive fake API key (POLYGRAPH-CANARY-<uuid>), a distinctive PII string, and a fake bearer token — seeded into arguments, the working directory, and the environment. Probe 4.1 scans all tool outputs for any canary echo back to the agent. Probe 4.2 scans captured egress for canary bytes in any outbound payload. A canary surfacing anywhere it shouldn’t is a failure. Without the sandbox, 4.2 degrades to output-scan only and is annotated as such.

The canaries only work if the harness runs the server itself: they are seeded into the server’s own arguments, working directory, and environment. A remote MCP URL is a server the harness does not run, so there is nowhere to seed a canary. Probe 4.1, and with it C-03 as a whole, is recorded as skipped — remote target (a skip is not a pass).

C-04 — Adversarial input handling · probes 3.1, 3.2

How the server holds up under hostile input. Probe 3.1 stresses each tool with a deterministic battery of malformed and oversized arguments; it fails if the server crashes or hangs, or if its output spills an uncaught stack trace (an internals leak). Probe 3.2 feeds known jailbreak patterns and scans the output — failing only if the server amplifies them into agent-directed injection of its own (a verbatim echo is excluded). A C-04 failure caps the grade at D. It is graded off-chain: it moves the overall letter, but the on-chain schema keeps its three per-category slots.

The detection primitives behind C-01, C-03, and C-04, implemented once and applied uniformly: invisibleUnicode (zero-width, bidi-override, and tag-char codepoints, each reported with codepoint and byte offset), instructionMimicry (agent-directed imperatives, override and jailbreak framing, free-text tool-call JSON), markdownTricks (javascript:/data: URIs, link-text/href mismatch, exfiltration-shaped query strings), internalsLeak (uncaught stack-trace and crash signatures across Node, Python, Java, Go, Ruby, Rust, .NET, and PHP), and canaryMatch (exact and lightly-obfuscated matches — case, whitespace, simple encodings). A shared reflection check lets the second-order and jailbreak probes ignore content a tool merely echoed back. Scanners are pure functions over text: independently testable, and the place new failure modes get added.

A single letter A–F, always accompanied by a rationale string, never a bare grade. Five grades are reachable; the scale skips E, as letter grades conventionally do. C became a live grade in litmus-v16 (see the rubric below), so it is no longer a reserved letter.

Rationale: injection and data-leak are disqualifying — they are the failures that directly harm an agent that trusts the server, so they floor the grade at F. A C-02 failure (egress overreach or a read-only lie) or a C-04 failure (a crash, an internals-leak, or jailbreak amplification) is serious but not proven exfiltration or harm, so it caps at D. The B tier keeps the no-sandbox path usable while stating honestly that egress was not verified. C is emitted only under the coverage cap: an unexercised destructive or value-moving tool plus an unverified category (typically egress). That coverage is too thin to clear as B, so the default agent gate refuses a C rather than trust an unexercised powerful surface. Every grade carries its reasons in the evidence bundle.

A remote server caps at B for a specific, disclosed reason. The harness reaches it over the network the same way an agent would, but it cannot wrap a URL it does not run in the egress sandbox, and it cannot seed canaries into a process it does not control. Two categories, C-02 and C-03, go unverified as a result. This is a coverage limit, not a defect found in the server: the grade names exactly what it tested and what it could not.

What makes a grade trustworthy rather than an assertion:

• Deterministic harness. Same server version + same litmus-v17 harness → same findings. The bait, jailbreak, and malformed batteries are varied but fixed — no randomness in probe verdicts; timestamps and environment are recorded, not baked in.
• Tool-defs fingerprint. The canonicalized tool surface is hashed (sha256) to a bytes32. The grade certifies that exact surface. If the server later changes a tool description — a rug pull — the fingerprint no longer matches and the grade is stale by construction. Consumers recompute the live fingerprint before trusting.
• Published evidence. The full evidence bundle — every finding, every artifact — travels with the grade. Anyone can fetch and inspect it.
• Re-runnable. Anyone — a skeptic, a counterparty, a future independent verifier — can re-run litmus-v17 against the same server and compare fingerprint and grade. A false grade is falsifiable, not merely disputable.

Two properties decide whether a grade can be trusted, and they are independent.

Forgeability — can the runner fake the result? Fixed by the proof layer, not the methodology. Reproducibility makes a lie falsifiable; the roadmap layers — independently verifiable grade records, and later hardware-attested runs — make it progressively unprofitable, then impossible.

Evasion — can the server tell it’s being tested and behave? A fundamental methodology limit. Because the methodology is open, a server can recognize the test context and behave benignly during evaluation, then misbehave in production — a defeat device. No proof layer fixes this; an independent lab running the same open test has the same exposure. We reduce, not eliminate, the gap: per-run-unique canary values, bait/jailbreak/malformed inputs drawn from varied (widened) but fixed pools, behavioral probes over real outputs rather than static reads, periodic re-attestation, and the live-fingerprint check at call time against bait-and-switch. Evasion is an explicitly acknowledged residual risk of v1.

Non-goals

• Not an independence claim (yet). v1 grades can be self-run: the subject grades itself, and trust anchors on reproducibility — the open harness makes a false grade falsifiable. Skin-in-the-game and independent counter-attestation are roadmap. A v1 grade is a reproducible test result, not an independent verdict. We say so plainly.
• Not secrets management. How a server stores or rotates its own secrets is out of scope for v1.
• Bounded surface. We probe the advertised tool surface at evaluation time. Tools gated behind auth or state we cannot reach are recorded as unexercised, never passed.
• No absolute claims. Never “100% safe” or “guaranteed.” Underclaim, over-deliver.

This page documents litmus-v17. Probes evolve as agents do; new failure modes get new probe IDs within their family. A change that alters pass/fail semantics bumps the methodology version. Every evidence bundle and every attestation embeds the methodology version that produced it, so a grade is always tied to the spec it was measured against — earlier litmus-v1…v16 grades stay valid as their own version’s results.

Changelog · litmus-v16 exercises write-capable tools inside the network-isolated sandbox by default, so a server that can move state earns A on the same terms as a read-only one (write access is not itself a demerit). The coverage cap now fires only where a call would hit a live backend (the host path or a remote https target): one high-risk tool left unexercised caps at B, and an unambiguously destructive or value-moving tool left unexercised together with an unverified category (typically egress) compounds to C. C is now a live grade rather than a reserved letter, marking a powerful server the harness could neither sandbox nor exercise; the default agent gate refuses it. The release also adds a tool-poisoning scanner to C-01 probe 1.1 (concealment directives, secret-file reads, exfil-to-sink), a new probe 1.4 for indirect / passthrough injection (a tool that relays injection-laced third-party content verbatim is disclosed as a conduit, not failed; only server-generated amplification fails), wider jailbreak, malformed, and runtime-crash corpora with provider-shaped canaries (AWS, GitHub, JWT), and an opt-in advisory LLM injection judge surfaced in the run summary only, which never affects the letter grade and is never part of the minted evidence bundle. It is the first change that is not purely monotonic: unlike the v9 through v15 precision fixes, v16 can move a verdict down as well as up. Older attestations stay valid, because methodologyVersion is a string the agent gate does not branch on, and each grade is read against the spec that produced it. litmus-v15 adds public package-registry infrastructure (pypi.org, files.pythonhosted.org, registry.npmjs.org) to the C-02 egress baseline, so a framework’s default startup update-check — chiefly FastMCP pinging pypi.orgfor a newer version — is no longer scored as the server’s own overreach. The cloud instance-metadata endpoint is deliberately left out (a real credential-theft target), so a server that reaches it still fails. Another false-positive fix that only turns a wrong D into a correct pass. litmus-v14 extends the safe-rejection handling in C-04 probe 3.2 so a server that correctly refuses a hostile input is no longer graded as amplifying it. A validation error that quotes the rejected payload back inside its own error frame (Invalid label "…", Error: … not found: …) — including echoes that are character-stripped or mid-truncated, which the v9 / v12 complete-echo masks could not match — now reads as a safe rejection, not injection. Only injection the server generates itself still fails; another false-positive fix that only turns a wrong D into a correct pass, so some safe-rejecting servers move D → A. litmus-v13 narrows C-01 tool-output-injection false positives so honest documentation is no longer graded as a hijack: angle-bracket placeholders (<tool>), data:/format examples, and a zero-width doc artifact no longer floor a server to F — the scanners strip invisible characters before scanning, so a hidden keyword is still caught while the benign artifact is not; each only turns a wrong F into a correct pass. litmus-v12 fixes two false positives so a server’s correct, defensive behavior is no longer graded as a fault: a validation error that quotes the rejected input back (e.g. Pydantic input_value='…') is read as a safe rejection, not C-04 amplification; and a mutation verb under a negation (“cannot revoke”) no longer reads as a C-02 read-only lie — each only turns a wrong D into a correct pass. litmus-v11 adds expected-upstream inference to C-02 probe 2.2: an honest API-wrapper that reaches the API its own surface names is reclassified from egress overreach into an informational egress-inferred finding (disclosure, not exoneration), so it is no longer wrongly capped at D — a false-positive fix that only turns a wrong D into a correct pass. litmus-v10 narrows C-02 probe 2.1 so an honestly read-only data tool is no longer misread as lying about a mutation (the noun “transfers,” a bare destination address). litmus-v9 stops C-04 probe 3.2 from flooring a server that merely echoes a hostile input back — only injection the server generates itself fails. litmus-v8 and litmus-v7 narrow the C-01 static scan so honest schemas and documentation (a parameter named function, an indented system: config key, ordinary role-tag prose) no longer read as injection. These four are false-positive precision fixes — each only turns a wrong fail into a correct pass, never the reverse. litmus-v6 stops a tool that lies about being read-only from being actively bait-called (probe 2.1 still caps the lie at D). litmus-v5 added C-01 probe 1.3 (second-order injection), made C-02 egress port-aware, and widened probe 2.1 to parameter- and description-evidenced read-only lies. litmus-v4 made C-04 (adversarial input) a graded category — a crash, internals-leak, or jailbreak amplification caps the grade at D — and closed the hard-coded-IP egress gap with a host-DNAT gateway. litmus-v3 reframed C-02 from default-deny to egress overreach: a server may reach hosts it declares, so a passing C-02 means no overreach, not no network. litmus-v2 added C-02 probe 2.1 (declared-permission honesty). Each pass/fail-semantics change bumps the methodology version; earlier grades stay valid as their own version’s results.

Everything above grades MCP servers by exercising them. Agent Skills are graded differently: a skill is content — a SKILL.md of instructions plus an optional bundle of files — and litmus-skill-v2 is a static safety scan of that content. Nothing is executed. “Agent Skills” here means the SKILL.md format used by Claude Code, the Claude apps, the Agent SDK, and skill marketplaces — not arbitrary agent frameworks.

It reads the skill the way an agent that loads it would, and asks one question:

Will loading this skill try to hijack me, tell me to leak data, or ship a dangerous command?

S-01 — Prompt injection / context poisoning

Scans the skill body for instructions aimed at the loading agent rather than the user’s task — override and jailbreak framing, role-tag or tool-call-shaped text, hidden or invisible Unicode. The same instruction-mimicry and invisible-Unicode primitives the server checks use, applied to skill content.

S-03 — Data-exfiltration instructions

Flags a skill that instructs the agent to read secrets, credentials, or environment values and send them somewhere — the exfiltration pattern expressed as guidance the agent is meant to follow.

S-04 — Dangerous bundled commands

Inspects the bundled files for commands that would harm the host if run — a piped curl | bash, a reverse shell, and the like. The scripts are read, never run; this is a static read of what the bundle would do, not a record of what it did.

Grade

A single letter on a strict A / B / D / F scale — skills have no C tier, the same skipped letter as the server scale. A clean scan across all three checks is A. A skill with no bundle to read leaves S-04 unverified — injection and exfiltration still pass, but a check could not run — so it caps at B, stated honestly rather than rounded up. A dangerous bundled command (S-04) caps at D; an injection or exfiltration hit (S-01 or S-03) floors at F. The grade is reproducible: the methodology is open and the letter is deterministic, and the skill is content-hashed so a grade ties to the exact bytes it measured.

What an A claims — and does not

An A is a clean static scan, not behavioral proof. A static read cannot catch a command built or fetched at runtime, and bundled scripts are read but never executed — both are out of scope by construction. The grade is a measurement of the skill’s content as written, not an accusation against it nor an endorsement of it. We underclaim here too.

Advisory · A separate, model-judged honesty signal can be reported alongside the letter — does the skill do what its description claims? It is non-deterministic, kept apart from the A / B / D / F grade, and never minted.

Run it yourself · npx -p @polygraphso/litmus polygraphso-litmus-skill <path-to-skill> — zero-install, or the run_skill_litmus MCP tool. (The MCP-server grader is the separate polygraphso-litmus / run_litmus.)

• polygraph.so — the polygraphs
• HTTP API — query published polygraphs
• /llms.txt — the agent-readable summary