GEL · Guard, Eval, Loop · bench 2026-11-16

Your agent said all tests pass.15% of the time, a check was red.

Five plain checks for every pull request: leaked keys, unpinned actions, workflows with an open token, links to files that are gone, and a PR that says green while its checks say red. No model, no network, any language. Starts in report mode.

231 real PRs. 35 lit: said green, check was red.
420/420planted keys, pins, permissions and links caught in 30 public repos. 0/150 of kinds it does not cover.
19of the 35 green-but-red PRs were merged anyway
19 msmedian per repo, all five guards, 11,350 files in all
01 · Benchmark

Same 30 repos. Same commits.One planted defect at a time.

Popular public repos in TypeScript, JavaScript, Python, Go and Rust, each pinned to one commit. A seeded random number picks the file. A defect counts only if GEL points at the exact file and line. The grey bar is how many of the 30 repos already run a tool in their own CI that would catch it.

150/150Leaked keysAWS, GitHub, Anthropic, Stripe live, private key blocks
const key = "AKIA" + "…"; stripe.init("sk_" + "live_…"); -----BEGIN PRIVATE KEY----- headers: { authorization: "ghp_…" } anthropic({ apiKey: "sk-ant-…" }) const key = "AKIA" + "…"; stripe.init("sk_" + "live_…"); -----BEGIN PRIVATE KEY----- headers: { authorization: "ghp_…" } anthropic({ apiKey: "sk-ant-…" }) const key = "AKIA" + "…"; stripe.init("sk_" + "live_…"); -----BEGIN PRIVATE KEY----- headers: { authorization: "ghp_…" } anthropic({ apiKey: "sk-ant-…" }) const key = "AKIA" + "…"; stripe.init("sk_" + "live_…"); -----BEGIN PRIVATE KEY----- headers: { authorization: "ghp_…" } anthropic({ apiKey: "sk-ant-…" })
100%
const key = "AKIA" + "…"; stripe.init("sk_" + "live_…"); -----BEGIN PRIVATE KEY----- headers: { authorization: "ghp_…" } anthropic({ apiKey: "sk-ant-…" }) const key = "AKIA" + "…"; stripe.init("sk_" + "live_…"); -----BEGIN PRIVATE KEY----- headers: { authorization: "ghp_…" } anthropic({ apiKey: "sk-ant-…" }) const key = "AKIA" + "…"; stripe.init("sk_" + "live_…"); -----BEGIN PRIVATE KEY----- headers: { authorization: "ghp_…" } anthropic({ apiKey: "sk-ant-…" }) const key = "AKIA" + "…"; stripe.init("sk_" + "live_…"); -----BEGIN PRIVATE KEY----- headers: { authorization: "ghp_…" } anthropic({ apiKey: "sk-ant-…" })
1/30 repos scan for keys
90/90Unpinned actions@v4, @master, @v4.1.0 on another org's action
- uses: actions/checkout@v4 - uses: docker/build-push-action@master - uses: aws-actions/configure-aws-credentials@v4.1.0 - uses: some/action@main - uses: actions/checkout@v4 - uses: docker/build-push-action@master - uses: aws-actions/configure-aws-credentials@v4.1.0 - uses: some/action@main - uses: actions/checkout@v4 - uses: docker/build-push-action@master - uses: aws-actions/configure-aws-credentials@v4.1.0 - uses: some/action@main - uses: actions/checkout@v4 - uses: docker/build-push-action@master - uses: aws-actions/configure-aws-credentials@v4.1.0 - uses: some/action@main
100%
- uses: actions/checkout@v4 - uses: docker/build-push-action@master - uses: aws-actions/configure-aws-credentials@v4.1.0 - uses: some/action@main - uses: actions/checkout@v4 - uses: docker/build-push-action@master - uses: aws-actions/configure-aws-credentials@v4.1.0 - uses: some/action@main - uses: actions/checkout@v4 - uses: docker/build-push-action@master - uses: aws-actions/configure-aws-credentials@v4.1.0 - uses: some/action@main - uses: actions/checkout@v4 - uses: docker/build-push-action@master - uses: aws-actions/configure-aws-credentials@v4.1.0 - uses: some/action@main
3/30 repos check action pins
90/90Open token scopeno permissions key, write-all, fork head under pull_request_target
on: pull_request_target permissions: write-all jobs: build: runs-on: ubuntu-latest ref: ${{ github.event.pull_request.head.sha }} on: pull_request_target permissions: write-all jobs: build: runs-on: ubuntu-latest ref: ${{ github.event.pull_request.head.sha }} on: pull_request_target permissions: write-all jobs: build: runs-on: ubuntu-latest ref: ${{ github.event.pull_request.head.sha }} on: pull_request_target permissions: write-all jobs: build: runs-on: ubuntu-latest ref: ${{ github.event.pull_request.head.sha }}
100%
on: pull_request_target permissions: write-all jobs: build: runs-on: ubuntu-latest ref: ${{ github.event.pull_request.head.sha }} on: pull_request_target permissions: write-all jobs: build: runs-on: ubuntu-latest ref: ${{ github.event.pull_request.head.sha }} on: pull_request_target permissions: write-all jobs: build: runs-on: ubuntu-latest ref: ${{ github.event.pull_request.head.sha }} on: pull_request_target permissions: write-all jobs: build: runs-on: ubuntu-latest ref: ${{ github.event.pull_request.head.sha }}
3/30 repos audit workflow permissions
90/90Broken repo linksmissing doc, image, doc#anchor
[Setup](docs/setup.md) ![diagram](img/arch.png) [API](docs/api.md#auth) [Guide](../guide/README.md) [Changelog](CHANGELOG.md) [Setup](docs/setup.md) ![diagram](img/arch.png) [API](docs/api.md#auth) [Guide](../guide/README.md) [Changelog](CHANGELOG.md) [Setup](docs/setup.md) ![diagram](img/arch.png) [API](docs/api.md#auth) [Guide](../guide/README.md) [Changelog](CHANGELOG.md) [Setup](docs/setup.md) ![diagram](img/arch.png) [API](docs/api.md#auth) [Guide](../guide/README.md) [Changelog](CHANGELOG.md)
100%
[Setup](docs/setup.md) ![diagram](img/arch.png) [API](docs/api.md#auth) [Guide](../guide/README.md) [Changelog](CHANGELOG.md) [Setup](docs/setup.md) ![diagram](img/arch.png) [API](docs/api.md#auth) [Guide](../guide/README.md) [Changelog](CHANGELOG.md) [Setup](docs/setup.md) ![diagram](img/arch.png) [API](docs/api.md#auth) [Guide](../guide/README.md) [Changelog](CHANGELOG.md) [Setup](docs/setup.md) ![diagram](img/arch.png) [API](docs/api.md#auth) [Guide](../guide/README.md) [Changelog](CHANGELOG.md)
1/30 repos check links
120/120Green claim, red checktwo lies and two honest PRs per repo
"All tests pass, ready to merge." test (ubuntu-latest): failure "All checks pass" lint: timed_out build: cancelled "All tests pass, ready to merge." test (ubuntu-latest): failure "All checks pass" lint: timed_out build: cancelled "All tests pass, ready to merge." test (ubuntu-latest): failure "All checks pass" lint: timed_out build: cancelled "All tests pass, ready to merge." test (ubuntu-latest): failure "All checks pass" lint: timed_out build: cancelled
100%
"All tests pass, ready to merge." test (ubuntu-latest): failure "All checks pass" lint: timed_out build: cancelled "All tests pass, ready to merge." test (ubuntu-latest): failure "All checks pass" lint: timed_out build: cancelled "All tests pass, ready to merge." test (ubuntu-latest): failure "All checks pass" lint: timed_out build: cancelled "All tests pass, ready to merge." test (ubuntu-latest): failure "All checks pass" lint: timed_out build: cancelled
0/30 repos check this
GELtheir CI today

The bars are filled with the kind of text each guard reads. Fake keys only.

02 · Real pull requests

35 of 231 public PRs said green.A check on the last commit was red.

Public PRs from 2026-10-01 to 2026-11-09 whose text says "all tests pass", "all checks pass" or "ready to merge". We read the checks on each PR's last commit and ran GEL's done-without-green guard. No common CI tool compares what a PR says with what its checks say.

PRs found298

231 had checks to compare.

Said green, was red35

15% of the PRs with checks.

Merged anyway19

The red check was on the commit that landed.

Only optional red8

Coverage, visual diffs. GEL cannot see which checks are required, so they still count.

$ npx github:flisk-sh/gel run --base main
lane C  .github/workflows/release.yml  # a person approves this one
FAIL  secret-scan         src/config.ts:12  Stripe live key, rotate it then remove it
FAIL  pinned-actions      release.yml:31    docker/build-push-action@master
pass  dead-links
FAIL  done-without-green  PR says "all tests pass"; test (ubuntu) is failure
mode report: recorded in gel-record.jsonl, not blocked

The first bench run found our own false alarms before it found anyone else's bugs. Every one became a test case, then a fix.

03 · False alarms

On clean code,a guard should stay quiet.

The same 30 repos, untouched. They do not leak keys, so any secret GEL finds is a false alarm. Every dead-link finding was checked by hand.

Secret findings026 in the first run

Doc placeholders, sample TLS keys under examples/ and tests in t/unit are now skipped.

Dead-link findings21217 in the first run

5 are real broken links. The rest point at files a docs build makes, so dead-links stays report-only.

Open token scope18/30

repos have a workflow job with no permissions key, 45 in all. We read 5 by hand: 5 real.

Tag-pinned actions21/30

repos use another org's action by tag today, 371 uses. Not a false alarm: a tag can move under you.

RepoWorkflowVerdictWhy
go-kratos/kratoscomment-check.yml:26realpull_request_target checks out the fork head with a pull-requests: write token
mobxjs/mobxbuild_and_test.yml:11realno permissions key in the file
tokio-rs/axumCI.yml:15realno permissions key in the file
stretchr/testifymain.yml:5realno permissions key in the file
cloudflare/pingoradocs.yml:9realno permissions key in the file

A sample of workflow-permissions findings on the untouched repos, read by hand on 2026-11-16. real = the file has no permissions key for that job (or checks out the PR head under pull_request_target), as GEL says. GEL cannot see the org default token setting, so a real finding may still run read-only.

RepoLinksVerdictWhy
Molunerfinn/PicGo2realchart.png is not in the repo
celery/celery2reallinks written from the repo root, but GitHub resolves them from .github/
asdf-vm/asdf1realtranslation kept ../package.json; it needs ../../package.json
mobxjs/mobx11falsepage is served from the site root, where assets/ exists
caolan/async1falseglobal.html is made by the jsdoc build
jashkenas/underscore1falsedocs/modules is made by the docs build
rwf2/Rocket1falsesite route to a numbered page
philc/vimium1uncleargithub.svg may be copied in at build time
python-telegram-bot/python-telegram-bot1unclearthe example asks you to supply telegram-passport.js
04 · How it works

Five guards. Three lanes.One record per run.

Each guard is a small Node file with labelled good and bad cases. GEL's own CI fails if a guard misses a bad case or flags a good one.

guard 1
secret-scan

AWS, GitHub, Anthropic, OpenAI, Stripe live and Slack keys, and private key blocks. Skips tests, sample keys and placeholders.

guard 2
pinned-actions

Another org's action must be pinned to a commit SHA. Your org and trusted_owners may use tags.

guard 3
workflow-permissions

Every workflow job gets a token scope, never write-all, and pull_request_target never checks out the fork's code.

guard 4
dead-links

Links to files that are not in the repo. Knows doc sites without .md, Vite public/ and symlinked READMEs.

guard 5
done-without-green

The PR says "all checks pass" and a check on its last commit failed, timed out or was cancelled.

lanes
A · B · C

Picked from the changed files, never a label. Docs and tests, product code, and workflows/auth/payment/deploy where a person approves.

loop
gel-record.jsonl

Every run writes one JSON line. Promote a guard to block once the record shows it was right.

report→block one guard→block all
05 · Eval and loop

Who checks the checker?The bench, every night.

A guard is promoted only on evidence, and GEL never changes your policy by itself. It writes advice; a person edits gel.json, and that edit is lane C.

nightly
meta-eval

Every guard against its labelled cases, then the bench on the 30 pinned repos. Fails if a guard catches fewer planted defects or raises more findings on clean code than bench/baseline.json.

per guard
gel metrics

Reads your gel-record.jsonl files and your gel-labels.json: runs, fail rate, precision, first-pass share, p50 and p95 time.

flywheel
promote / demote

Promote after 20 runs, 5 labelled findings and 95% precision. Demote a blocking guard on one false alarm. Each false alarm is listed as a hard case to add to cases.json.

before release
release-gate

A shared workflow in flisk-sh/.github: waits for every check on the release commit and fails on a red or missing required one.

after deploy
deploy-health

Curls the deployed URL until it answers with the expected status and text, or fails the release.

org view
Cilytics

The org CI dashboard reads the latest gel-record of each repo: which repos run GEL, findings, blocks.

06 · Limits

What GEL misses.Measured the same way.

We planted these too, so the recall above is not 100% by choice of test.

PlantedCaughtWhy
Google API key in a source file0/30Not in the rules yet.
Password inside a database URL0/30Not in the rules yet.
AWS key inside a test file0/30On purpose: test files are full of fake keys.
HTML <a href> inside a markdown file0/30Markdown files are read for markdown links only.
A write scope the job never uses0/30GEL checks that a scope is set, not that it is needed.
Org default token setting–workflow-permissions reads files only. If the org default token is read-only, a job without a permissions key is safer than GEL says.
Protection outside workflows–"Their CI today" reads workflow files only. Push protection, pre-commit hooks and branch rules are not seen, so it is a lower bound.
07 · Speed

Fast enoughto run on every push.

Median per repo19 ms

All five guards on every tracked file, on a laptop.

95th percentile93 ms

The slowest repos in the set. The largest has 1,363 files.

Whole CI job17 s

Median wall time of 6 pilot runs, queue to uploaded record.

08 · Install

Two lines.Then it runs on every PR.

Starts in report mode. Add a gel.json when you want a guard to block.

As a step
- uses: actions/checkout@v4
  with: { fetch-depth: 0 }
- uses: flisk-sh/gel@v1
As a job
jobs:
  gel:
    permissions: { contents: read, checks: read, pull-requests: read }
    uses: flisk-sh/gel/.github/workflows/gel.yml@v1
On your laptop
npx github:flisk-sh/gel run --root .
gel.json
{
  "mode": "report",
  "guards": { "secret-scan": "block" },
  "trusted_owners": ["your-org"]
}

Run the bench yourself: node bench/run.mjs && node bench/claims.mjs. Method: bench/README.md.