Vet it before you install.
It doesn't judge whether a skill is useful. It answers one question first: what does this package touch before you let it run? Four steps, 14 reject-on-sight patterns, and a fixed report format — the decision is made on evidence, not on a registry label.
- 4-step protocol
- 14 reject patterns
- 4 risk tiers
- 138 lines, zero runtime deps
↑ The report format, filled in with a real result. The panel above is the actual skill-vetter review of webapp-testing in anthropics/skills — 6 files, one substantive flag, 🟡 MEDIUM — not a mock-up. The report is the deliverable.
Same instruction. The difference is what happens before install.
The case below is one real package: webapp-testing from anthropics/skills. One prompt, one package — the only variable is whether a review runs first. Every figure is a keyword count taken from the actual files, and it can be reproduced.
| Dimension | Without | With | |
|---|---|---|---|
| Files seen | 1 entry file (3,913 B) | → | 6 / 6 files read in full |
| Dependency | Playwright named, never explained | → | Flagged: 0 manifests, 0 install instructions |
| Execution surface | Not visible from the entry file | → | shell=True at with_server.py:71, assessed and reported |
| Permissions | Unstated | → | Files / Network / Commands as separate fields |
| Red flags | Not checked | → | 14 patterns run — 1 substantive, every judgement recorded |
| Output | Prose paragraph | → | 🟡 MEDIUM · ⚠️ INSTALL WITH CAUTION + 3 NOTES |
Source: the real webapp-testing files pulled from anthropics/skills (main branch) on 2026-09-15 — 6 files in total: SKILL.md, LICENSE.txt, one script and three examples. “0 hits” means a keyword search run over SKILL.md alone and over all six files separately; both are reproducible.
Four steps, 14 red lines, and a report that cannot be skipped.
Every number below is a structural count, read directly from the 138-line source. There are no runtime metrics — this skill has no downloads, installs, or telemetry to report.
Source traceability
Origin, author, release, popularity, last update, and third-party reviews captured before any file is read.
Mandatory code review
Every file in the package is read. Fourteen patterns — network egress, credential access, obfuscation, privilege escalation — trigger immediate rejection.
Permission scoping
What it reads, what it writes, what it executes, where it connects — and whether that scope is minimal for the stated purpose.
Risk classification
Four tiers, each bound to a required action — from basic review to human approval to an outright refusal.
Fixed report format
A constant skeleton: identity, metrics, red flags, permissions, risk level, verdict, notes. Fields cannot be silently dropped.
Trust hierarchy
Scrutiny scales with provenance: official skills get less, unknown sources get maximum, anything requesting credentials gets a human.
Risk tiers and the action each one forces
What the 14 patterns actually look for
Four steps, then and only then: install or don't.
The protocol is sequential. Step 2 is marked mandatory in the source — a review that skips the file read is not a review.
Source Check
Establish provenance: where it came from, who published it, how widely it is used, when it last changed, what others reported.
Code Review
Read all files. Match against the 14 reject patterns — egress, credentials, identity files, dynamic execution, concealment.
Permission Scope
Enumerate reads, writes, commands, and network endpoints. Judge whether the scope is minimal for the stated purpose.
Risk Classification
Assign a tier, bind it to the required action, emit the fixed report, and state a verdict the user can act on.
This protocol answers "what does this package touch, and is that acceptable?" — not "does it do its job well?" A package that passes can still be useless; a package that fails on a pattern is rejected regardless of how useful it looks. Review quality also depends on the reviewing agent actually opening every file — the skill guides the review, it does not execute a scanner.
A real package, with the evidence kept.
Three packages, vetted with the same four-step protocol: a documentation-only ruleset, a pack with a real execution surface, and a 70-file reference pack with no executable code at all. Each was run separately in English — none of these is a translation of the Chinese run. Every figure, hash and line reference can be checked in the run logs.
Verbatim, with the read-only constraint stated up front and no hint of the expected answer.
Target: https://github.com/anthropics/skills/tree/main/skills/brand-guidelines
Read every file under that directory before you judge. Read-only — do not install it, do not execute anything in it.
| Observed package | Value |
|---|---|
| Files reviewed | 2 — both read in full |
| Size | 13,580 B / 274 lines |
| Executable content | None — 0 code fences, 0 scripts |
| Repo stars | 176,424 (anthropics/skills) |
| Last updated | 2026-09-10 |
Provenance, contents, access, and the one judgement call the pattern list could not make on its own.
Anthropic's official organisation account, 176,424 stars. Lower scrutiny was permitted — but both files were still read in full.
SKILL.md is 2,235 B / 73 lines: a colour table of 4 base + 3 accent hex values and a type spec (Poppins headings, Lora body, Arial/Georgia fallbacks). The other file is Apache 2.0.
Files, Network and Commands are all empty. There is nothing for the pack to be granted — everything it does happens inside the model that loads it.
The library python-pptx is named once (SKILL.md:72) as the mechanism for applying colour. It is never imported and never installed — so it is not a dependency finding. Case 02 is where the same word becomes a real flag.
| RED FLAGS | None — 0 substantive across all 14 patterns |
| SOURCE | GitHub · author anthropics |
| METRICS | 176,424 stars · last updated 2026-09-10 · 2 files reviewed |
| PERMISSIONS | None — Files / Network / Commands all empty |
| RECOMMENDATION | ✅ safe to install; there is no executable surface to review |
| CLASSIFICATION | Low |
Screenshots captured from the live run — click any frame to open it at full size.
With no executable surface, this is a completeness claim about documents rather than about behaviour. Two substring patterns matched Apache licence boilerplate — http:// inside the licence URL, and auth inside "authorized"/"authorship" — and were dismissed only after each line was read in context. That is the one place judgement, not a pattern, decided the outcome.
Verbatim, with the read-only constraint stated up front.
Target: https://github.com/anthropics/skills/tree/main/skills/webapp-testing
Read every file under that directory before you judge. Read-only — do not install it, do not execute anything in it.
| Observed package | Value |
|---|---|
| Files reviewed | 6 — all read in full |
| Size | 22,394 B / 506 lines |
| Repo stars | 176,424 (anthropics/skills) |
| Last updated | 2026-09-10 |
Every import carries a file and line number, so the claim can be checked rather than believed.
| Location | Statement |
|---|---|
| scripts/with_server.py:17 | import subprocess |
| examples/console_logging.py:1 | from playwright.sync_api import sync_playwright |
| examples/element_discovery.py:1 | from playwright.sync_api import sync_playwright |
| examples/static_html_automation.py:1 | from playwright.sync_api import sync_playwright |
| SKILL.md:54 | from playwright.sync_api import sync_playwright |
| RED FLAGS | 1 (substantive) — installs packages without listing them |
| SOURCE | GitHub · author anthropics |
| METRICS | 176,424 stars · last updated 2026-09-10 · 6 files reviewed |
| RECOMMENDATION | Install only after the dependency is declared; the pack is not self-sufficient as shipped |
| CLASSIFICATION | High |
Screenshots captured from the live run — click any frame to open it at full size.
This run reads the pack; it does not run it. No claim is made about what the code does at execution time, only about what is written in it. The stars figure is repository-level, not package-level, and is recorded as a popularity signal rather than a safety signal.
Verbatim. Same protocol instruction as Case 02, different target.
Target: https://github.com/anthropics/skills/tree/main/skills/claude-api
Read every file under that directory before you judge. Read-only — do not install it, do not execute anything in it.
Two things this target exposes about the protocol itself.
100,688 B of 1,174,800 B was read by eye. The remainder was covered by exhaustive machine pattern scanning across all 70 files — not by reading. The report says so.
The checklist lists eval() and exec() — which does not match the shell form eval "$(ant auth print-credentials --env)" at anthropic-cli.md:67. It was found by manual review, not by the pattern.
Run 1 (≈20:00) and run 2 both returned 🔴 HIGH with 2 substantive flags. Stars moved 176,424 → 176,463; all 70 files verified byte-identical.
Nothing installed, no file executed, no API call made, no credential read.
| RED FLAGS | 2 (substantive) — full itemisation is in the run log |
| FILES | 70 · 1,174,800 B · 17,389 lines |
| COVERAGE | 100,688 B read by eye (≈8.6%) + exhaustive scan of all 70 files |
| RECOMMENDATION | ⚠️ human approval before use |
| CLASSIFICATION | High |
Screenshots captured from the second English run — click any frame to open it at full size.
Two limits stated plainly. First: this run could not read all 70 files — 8.6% by eye, the rest by machine pattern scan, so an undetected item remains possible. Second: this target was run twice in English. Both runs are kept — run 1 in the archive as the superseded record, run 2 as the record of reference — and the screenshots below belong to run 2.
The reviewer itself has nothing to hide.
Audited before use, in the same way it audits others. What follows is what the file actually contains.
Security audit · 7 checks
- ✓0 lines of executable codepure Markdown instructions
- ✓No install hooks or scriptsno pre/post install actions
- ✓No dependency installationno pip / npm / requirements
- ✓No outbound network callscommands shown are for the user to run
- ✓No credential or file accessreads only what the agent opens
- ✓4 files totalSKILL.md · _meta.json · skill-card.md · assets/
- ✓138 lines, single file of logicreadable end to end in one sitting
Reliability · design constraints
- ✓Step 2 marked mandatory in sourcethe file read cannot be treated as optional
- ✓14 patterns enumerated, not summarisedeach one matchable or clearable
- ✓Fixed report skeleton7 field groups, verdict required
- ✓Verdict is a closed set3 outcomes, no free-form hedge
- ✓Refusal is a valid output"do not install" is built in, not a failure
- ✓Human escalation definedcredentials always require approval
Known limitations · 6
- Static only. It inspects the text of a package revision. It does not execute code, resolve obfuscated payloads, or observe runtime behaviour.
- It does not detect malware. A clean result means no listed pattern matched — not that the package is benign.
- Publisher trust is out of scope. It records who published the package; it cannot verify who that actually is.
- Revisions reset the result. A hash-recorded review covers one upload. A later release requires a fresh pass.
- Popularity is not evidence. Stars, downloads, and clean registry labels are context, not proof.
- Quality of review tracks the reviewing agent. The skill structures the work; it cannot force the agent to open every file.
Built for the moment just before install.
Anyone can read a registry page. This is for the step where someone has to decide whether that is enough.
Agent operators
Skill authors
Team leads
Security reviewers
Vet the next one before you install it.
Say what you want reviewed and set the boundary. The protocol handles the order and the report format.