Your agent said all tests pass.15% of the time, a check was red.
Five plain checks for every pull request: leaked keys, unpinned actions, workflows with an open token, links to files that are gone, and a PR that says green while its checks say red. No model, no network, any language. Starts in report mode.
Same 30 repos. Same commits.One planted defect at a time.
Popular public repos in TypeScript, JavaScript, Python, Go and Rust, each pinned to one commit. A seeded random number picks the file. A defect counts only if GEL points at the exact file and line. The grey bar is how many of the 30 repos already run a tool in their own CI that would catch it.
The bars are filled with the kind of text each guard reads. Fake keys only.
35 of 231 public PRs said green.A check on the last commit was red.
Public PRs from 2026-10-01 to 2026-11-09 whose text says "all tests pass", "all checks pass" or "ready to merge". We read the checks on each PR's last commit and ran GEL's done-without-green guard. No common CI tool compares what a PR says with what its checks say.
231 had checks to compare.
15% of the PRs with checks.
The red check was on the commit that landed.
Coverage, visual diffs. GEL cannot see which checks are required, so they still count.
$ npx github:flisk-sh/gel run --base main lane C .github/workflows/release.yml # a person approves this one FAIL secret-scan src/config.ts:12 Stripe live key, rotate it then remove it FAIL pinned-actions release.yml:31 docker/build-push-action@master pass dead-links FAIL done-without-green PR says "all tests pass"; test (ubuntu) is failure mode report: recorded in gel-record.jsonl, not blocked
The first bench run found our own false alarms before it found anyone else's bugs. Every one became a test case, then a fix.
On clean code,a guard should stay quiet.
The same 30 repos, untouched. They do not leak keys, so any secret GEL finds is a false alarm. Every dead-link finding was checked by hand.
Doc placeholders, sample TLS keys under examples/ and tests in t/unit are now skipped.
5 are real broken links. The rest point at files a docs build makes, so dead-links stays report-only.
repos have a workflow job with no permissions key, 45 in all. We read 5 by hand: 5 real.
repos use another org's action by tag today, 371 uses. Not a false alarm: a tag can move under you.
| Repo | Workflow | Verdict | Why |
|---|---|---|---|
| go-kratos/kratos | comment-check.yml:26 | real | pull_request_target checks out the fork head with a pull-requests: write token |
| mobxjs/mobx | build_and_test.yml:11 | real | no permissions key in the file |
| tokio-rs/axum | CI.yml:15 | real | no permissions key in the file |
| stretchr/testify | main.yml:5 | real | no permissions key in the file |
| cloudflare/pingora | docs.yml:9 | real | no permissions key in the file |
A sample of workflow-permissions findings on the untouched repos, read by hand on 2026-11-16. real = the file has no permissions key for that job (or checks out the PR head under pull_request_target), as GEL says. GEL cannot see the org default token setting, so a real finding may still run read-only.
| Repo | Links | Verdict | Why |
|---|---|---|---|
| Molunerfinn/PicGo | 2 | real | chart.png is not in the repo |
| celery/celery | 2 | real | links written from the repo root, but GitHub resolves them from .github/ |
| asdf-vm/asdf | 1 | real | translation kept ../package.json; it needs ../../package.json |
| mobxjs/mobx | 11 | false | page is served from the site root, where assets/ exists |
| caolan/async | 1 | false | global.html is made by the jsdoc build |
| jashkenas/underscore | 1 | false | docs/modules is made by the docs build |
| rwf2/Rocket | 1 | false | site route to a numbered page |
| philc/vimium | 1 | unclear | github.svg may be copied in at build time |
| python-telegram-bot/python-telegram-bot | 1 | unclear | the example asks you to supply telegram-passport.js |
Five guards. Three lanes.One record per run.
Each guard is a small Node file with labelled good and bad cases. GEL's own CI fails if a guard misses a bad case or flags a good one.
secret-scanAWS, GitHub, Anthropic, OpenAI, Stripe live and Slack keys, and private key blocks. Skips tests, sample keys and placeholders.
pinned-actionsAnother org's action must be pinned to a commit SHA. Your org and trusted_owners may use tags.
workflow-permissionsEvery workflow job gets a token scope, never write-all, and pull_request_target never checks out the fork's code.
dead-linksLinks to files that are not in the repo. Knows doc sites without .md, Vite public/ and symlinked READMEs.
done-without-greenThe PR says "all checks pass" and a check on its last commit failed, timed out or was cancelled.
A · B · CPicked from the changed files, never a label. Docs and tests, product code, and workflows/auth/payment/deploy where a person approves.
gel-record.jsonlEvery run writes one JSON line. Promote a guard to block once the record shows it was right.
Who checks the checker?The bench, every night.
A guard is promoted only on evidence, and GEL never changes your policy by itself. It writes advice; a person edits gel.json, and that edit is lane C.
meta-evalEvery guard against its labelled cases, then the bench on the 30 pinned repos. Fails if a guard catches fewer planted defects or raises more findings on clean code than bench/baseline.json.
gel metricsReads your gel-record.jsonl files and your gel-labels.json: runs, fail rate, precision, first-pass share, p50 and p95 time.
promote / demotePromote after 20 runs, 5 labelled findings and 95% precision. Demote a blocking guard on one false alarm. Each false alarm is listed as a hard case to add to cases.json.
release-gateA shared workflow in flisk-sh/.github: waits for every check on the release commit and fails on a red or missing required one.
deploy-healthCurls the deployed URL until it answers with the expected status and text, or fails the release.
CilyticsThe org CI dashboard reads the latest gel-record of each repo: which repos run GEL, findings, blocks.
What GEL misses.Measured the same way.
We planted these too, so the recall above is not 100% by choice of test.
| Planted | Caught | Why |
|---|---|---|
| Google API key in a source file | 0/30 | Not in the rules yet. |
| Password inside a database URL | 0/30 | Not in the rules yet. |
| AWS key inside a test file | 0/30 | On purpose: test files are full of fake keys. |
| HTML <a href> inside a markdown file | 0/30 | Markdown files are read for markdown links only. |
| A write scope the job never uses | 0/30 | GEL checks that a scope is set, not that it is needed. |
| Org default token setting | – | workflow-permissions reads files only. If the org default token is read-only, a job without a permissions key is safer than GEL says. |
| Protection outside workflows | – | "Their CI today" reads workflow files only. Push protection, pre-commit hooks and branch rules are not seen, so it is a lower bound. |
Fast enoughto run on every push.
All five guards on every tracked file, on a laptop.
The slowest repos in the set. The largest has 1,363 files.
Median wall time of 6 pilot runs, queue to uploaded record.
Two lines.Then it runs on every PR.
Starts in report mode. Add a gel.json when you want a guard to block.
- uses: actions/checkout@v4 with: { fetch-depth: 0 } - uses: flisk-sh/gel@v1
jobs: gel: permissions: { contents: read, checks: read, pull-requests: read } uses: flisk-sh/gel/.github/workflows/gel.yml@v1
npx github:flisk-sh/gel run --root .
{
"mode": "report",
"guards": { "secret-scan": "block" },
"trusted_owners": ["your-org"]
}Run the bench yourself: node bench/run.mjs && node bench/claims.mjs. Method: bench/README.md.