🧠 Lobe Prognosis

What it does

Prognosis runs after every PR captures a Lobe session in CI. For each PR it:

• Reads your lobe.yml config to know your SLA thresholds per route scenario
• Compares the PR’s captured session against your baseline (usually main)
• Scores every scenario against its SLA — TTFB p95, total p99, error rate
• Assigns a confidence score based on sample size, baseline stability, and how well the perf hit aligns with what the diff actually changed
• Posts a PR comment with the verdict + confidence + which routes breached
• Posts a “Lobe Prognosis” check run on the head commit — green ✓ on pass, red ✗ when a scenario listed in fail_ci_on breaches its SLA
• Refuses to fail your CI on low-confidence data — if it’s not sure, it says so instead of crying wolf

Step 1 — Install the GitHub App

Install the Lobe Prognosis GitHub App on the repos where you want the bot to comment. Takes 30 seconds.

You’ll be asked to pick which repos to install on. Choose only the ones you want Prognosis to comment on — you can add more later.

What the app requests: Read Contents (so it can fetch lobe.yml), Read+Write Pull requests & Issues (so it can post/update its comment), and Read+Write Checks (the pass/fail check run).

What “write” means here: posting comments and reporting check statuses — that’s the entire write surface. The app cannot push or modify code, cannot approve, merge, or close PRs, and cannot change repo settings. Even a failing check only blocks merging if you mark it required in your own branch protection rules.

Step 2 — Add lobe.yml to your repo

Drop this file at the root of the repo. This is where you declare your SLA thresholds per scenario.

# lobe.yml — commit this to your repo root
version: 1

# Which branch is the baseline. Default: main.
baseline: main

# When to flag vs stay quiet.
sensitivity:
  min_samples: 30              # skip verdict if fewer samples than this
  significance_threshold: 15   # % change required to consider material
  ignore_noise_ratio: true     # skip flags on routes with noisy baselines

# Group your routes by criticality and SLA.
scenarios:
  - name: checkout
    priority: critical           # fails CI if regressed
    routes: ["/api/checkout/*", "/api/pay/*"]
    sla:
      ttfb_p95_ms: 200
      total_p99_ms: 500
      error_rate_pct: 0.1

  - name: read-heavy
    priority: high               # warns but doesn't fail CI
    routes: ["/api/products/*", "/api/users/*"]
    sla:
      ttfb_p95_ms: 400
      total_p99_ms: 800

  - name: static
    priority: low                # informational only
    routes: ["/_next/static/*", "/assets/*"]
    sla:
      ttfb_p95_ms: 50

# What to do with verdicts.
verdict:
  fail_ci_on: [critical]                # only fail on critical regressions
  post_pr_comment: true
  minimum_confidence_to_flag: 60        # skip low-confidence verdicts

Step 3 — Route CI traffic through Lobe

Wrap your existing integration tests with the Lobe capture proxy. The bot picks up the session automatically once CI uploads it.

# .github/workflows/perf.yml
name: Perf check
on:
  pull_request:
    branches: [main]
  push:
    branches: [main]   # merges to main establish the baseline

jobs:
  perf:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Lobe CLI
        run: cargo install lobe-cli

      - name: Authenticate CLI with Lobe cloud
        run: lobe login --token ${{ secrets.LOBE_TOKEN }}

      - name: Start app + run integration tests through Lobe proxy
        env:
          # PR runs get graded against the baseline; pushes to main ARE the
          # baseline, so the flag is empty there.
          LOBE_PR_FLAG: ${{ github.event_name == 'pull_request' && format('--pr {0}', github.event.pull_request.number) || '' }}
        run: |
          # Start your app on port 3000 (however you normally do this)
          npm run build && npm run start &
          sleep 5

          # Route traffic through Lobe proxy and run your tests
          lobe capture http://localhost:3000 --upload $LOBE_PR_FLAG &
          sleep 2
          npm run test:integration -- --base-url http://127.0.0.1:7878

      - name: Wait for upload
        run: sleep 5

Get LOBE_TOKEN by visiting getlobe.dev/cli-auth and minting a token. Add it as a secrets.LOBE_TOKEN in your GitHub repo’s Actions secrets.

The --upload flag streams the session to your Lobe account with the PR number attached. The bot then compares it to your baseline and posts.

Proxy port note: the snippet uses 127.0.0.1:7878 because that’s the CLI default. If your CI already binds that port, pass --listen 127.0.0.1:9090 on lobe capture and point your tests at the same port.

Step 4 — Make the check required (optional but recommended)

The verdict lands as a check run named Lobe Prognosis on every graded commit. Like any CI check, a red ✗ only blocks merging if you mark it required: repo Settings → Branches → branch protection rule → Require status checks to pass, then add Lobe Prognosis to the required list.

• Only scenarios whose priority appears in verdict.fail_ci_on produce a failing check — warnings and low-confidence verdicts report as neutral, never blocking
• The first capture on a repo (no baseline yet) reports neutral, so a required check can’t strand your very first PR
• Re-grades on the same commit replace the previous verdict

lobe.yml reference

Confidence scoring — how the bot decides when to shut up

Every verdict comes with a 0–100 confidence score. Four weighted signals feed into it:

When confidence falls below verdict.minimum_confidence_to_flag (default 60), the bot posts an UNCERTAIN verdict instead of a FAIL. Your CI doesn’t break; the comment recommends gathering more samples. This is the anti-false-positive layer.

Import shortcuts (optional)

Recommended path is a native lobe.yml. But if your team already declared SLAs elsewhere, you can point Prognosis at those and skip re-declaring:

• Datadog SLOs — commit an exported YAML at datadog-slos.yml. Latency SLOs and error-budget targets are converted to Lobe scenarios.
• PagerDuty service configs — commit an exported JSON at pagerduty-services.json. Service urgency infers scenario priority. Since PagerDuty doesn’t hold latency thresholds, treat this as a starting point — graduate to a native lobe.yml when you want real SLA numbers.

Priority order: lobe.yml → .github/lobe.yml → datadog-slos.yml → pagerduty-services.json. Whichever it finds first, it uses — exclusively; files are never merged. The moment a lobe.yml exists, the import files are ignored entirely.

Team tier

Prognosis is a Team-tier feature. Free and Pro users can still upload sessions with --pr attached — those sessions appear in your dashboard normally — but the automated PR comment only runs for Team.

Upgrade to Team→

Troubleshooting

• Bot posted nothing. Check the Actions log — did the upload step succeed? Confirm you set the --pr flag on lobe capture. If you’re on Free/Pro, the bot won’t comment.
• Bot says “no baseline yet.” It needs at least one Lobe session on the baseline branch to compare against. Merge one PR to main to establish a baseline; subsequent PRs will get a real verdict.
• Verdict is always UNCERTAIN. You’re not sending enough requests through the proxy per PR. Loosen your test scope, or lower sensitivity.min_samples (not recommended — p95 gets noisy under 30 samples).
• Bot has installed but permissions look off. Re-visit the app on GitHub, hit “Configure,” and confirm Contents=Read, Pull requests=Read&Write, and Checks=Read&Write. Missing Read on Contents = can’t fetch lobe.yml; missing Checks = comment posts but no pass/fail check appears.

Contact

Bugs, requests, or questions: getlobedev@gmail.com