qwc-open/tulli

Watch

Fork 0

§8.8 Mirror refresh scheduler #12

New issue

Open

opened 2026-07-10 13:15:56 +02:00 by qwc · 1 comment

qwc commented

2026-07-10 13:15:56 +02:00

Owner

Labels: enhancement, scanning

Jittered nightly git remote update over all mirrors that triggers the on-update pipeline; tune window/jitter, upstream backoff, refresh age-out (§11.7).

**Labels:** `enhancement`, `scanning` Jittered nightly `git remote update` over all mirrors that triggers the on-update pipeline; tune window/jitter, upstream backoff, refresh age-out (§11.7).

qwc added the

Kind/Enhancement

scanning

labels

2026-07-10 13:52:46 +02:00

qwc commented

2026-07-13 13:54:59 +02:00

Author

Owner

Current state (verified @ `6769be5`) + the `--bare` caveat

Nothing refreshes mirrors today. A mirror is materialised exactly once, on first fetch, then frozen:

internal/git/mirror.go EnsurePresent() is clone-once — if the bare repo already exists it returns immediately (isBareRepo(dir) → return dir, false, nil) and never re-fetches. There is no git remote update / git fetch anywhere in the non-test code.
internal/mirror/reconciler.go:5 carries the placeholder for this: "(The refresh scheduler, §8.8, will also live here later.)"
The only interval that runs — main.go's -reconcile-interval sweep (default 30s) — reconciles DB verdicts → on-disk uploadpack.hideRefs. It touches no network and pulls no new commits. Don't confuse it with refresh.

Consequence: once a repo is mirrored, upstream changes (new commits, new refs, force-pushes / rewritten history) are never picked up.

⚠️ `--bare` won't absorb rewritten history — needs a forced refspec

Commit e81746a deliberately switched the clone from --mirror to git clone --bare (mirror.go:132–146) to stop HEAD leaking past ref-quarantine. But --bare does not install the +refs/*:refs/* mirror refspec. So when this scheduler is implemented, a naive git remote update on these repos will not correctly reflect a rewritten upstream:

non-fast-forward updates (force-pushed / rebased branches, moved tags) would be rejected or silently ignored without a leading + (force) in the refspec.
deleted-upstream refs won't be pruned without --prune.

The refresh job must therefore fetch with an explicit forced, scoped refspec (e.g. +refs/heads/*:refs/heads/* +refs/tags/*:refs/tags/* --prune) rather than relying on the --bare default — and scoped to preserve the HEAD/ref-quarantine invariant that e81746a established (don't reintroduce the --mirror leak). Verify against the git/head.go + hiderefs.go behaviour when wiring it.

Detecting a rewrite and re-evaluating correctly

Fetching the rewrite is only half of it — the verdict must be re-evaluated:

After a refresh that changes any ref, re-fire the on-update scan (the created-triggered scan pipeline today only runs on the first clone; there's no subsequent trigger because there's no subsequent fetch).
Re-run verdict.RecomputeRef for changed refs so a rewrite actually re-evaluates the block. This matters directly for the §6.7/§11.3 lift semantics: for a secret finding, a history rewrite upstream is the only thing that lifts the block (delete-without-rewrite doesn't — the blob stays cloneable), so the rewrite has to be fetched and re-scanned for that lift to ever happen.
Consider recording old→new ref tips on refresh so the notify path can report "ref X was rewritten upstream".

#22 — manual "update now" button in the repo list (same fetch machinery, on-demand trigger).
#4 (Strategy B want-parsing) — ref-precise attribution for the re-scan/notify payload.
#11 (demand-anchored scan floor) — bounds the re-scan cost after a refresh.
§11.7 — window/jitter, upstream rate-limit/backoff, refresh age-out tuning (already noted in the issue body).

## Current state (verified @ `6769be5`) + the `--bare` caveat **Nothing refreshes mirrors today.** A mirror is materialised exactly once, on first fetch, then frozen: - `internal/git/mirror.go` `EnsurePresent()` is clone-once — if the bare repo already exists it returns immediately (`isBareRepo(dir) → return dir, false, nil`) and never re-fetches. There is no `git remote update` / `git fetch` anywhere in the non-test code. - `internal/mirror/reconciler.go:5` carries the placeholder for this: *"(The refresh scheduler, §8.8, will also live here later.)"* - The only interval that runs — `main.go`'s `-reconcile-interval` sweep (default 30s) — reconciles DB verdicts → on-disk `uploadpack.hideRefs`. It touches no network and pulls no new commits. **Don't confuse it with refresh.** Consequence: once a repo is mirrored, upstream changes (new commits, new refs, **force-pushes / rewritten history**) are never picked up. ### ⚠️ `--bare` won't absorb rewritten history — needs a forced refspec Commit `e81746a` deliberately switched the clone from `--mirror` to **`git clone --bare`** (`mirror.go:132–146`) to stop HEAD leaking past ref-quarantine. But `--bare` does **not** install the `+refs/*:refs/*` mirror refspec. So when this scheduler is implemented, a naive `git remote update` on these repos will **not** correctly reflect a rewritten upstream: - non-fast-forward updates (force-pushed / rebased branches, moved tags) would be **rejected or silently ignored** without a leading `+` (force) in the refspec. - deleted-upstream refs won't be pruned without `--prune`. The refresh job must therefore fetch with an **explicit forced, scoped refspec** (e.g. `+refs/heads/*:refs/heads/*` `+refs/tags/*:refs/tags/*` `--prune`) rather than relying on the `--bare` default — and **scoped to preserve the HEAD/ref-quarantine invariant that `e81746a` established** (don't reintroduce the `--mirror` leak). Verify against the `git/head.go` + `hiderefs.go` behaviour when wiring it. ### Detecting a rewrite and re-evaluating correctly Fetching the rewrite is only half of it — the verdict must be re-evaluated: 1. After a refresh that changes any ref, **re-fire the on-update scan** (the `created`-triggered scan pipeline today only runs on the *first* clone; there's no subsequent trigger because there's no subsequent fetch). 2. Re-run `verdict.RecomputeRef` for changed refs so a rewrite actually re-evaluates the block. This matters directly for the §6.7/§11.3 lift semantics: for a **secret** finding, a history rewrite upstream is the *only* thing that lifts the block (delete-without-rewrite doesn't — the blob stays cloneable), so the rewrite has to be fetched **and** re-scanned for that lift to ever happen. 3. Consider recording old→new ref tips on refresh so the notify path can report "ref X was rewritten upstream". ### Related - **#22** — manual "update now" button in the repo list (same fetch machinery, on-demand trigger). - **#4** (Strategy B want-parsing) — ref-precise attribution for the re-scan/notify payload. - **#11** (demand-anchored scan floor) — bounds the re-scan cost after a refresh. - **§11.7** — window/jitter, upstream rate-limit/backoff, refresh age-out tuning (already noted in the issue body).

qwc referenced this issue

2026-07-13 14:19:29 +02:00

Nightly update of mirrors is fine, but we also need a button at the repository list to update immediately for whatever reason. #22

qwc referenced this issue

2026-07-13 16:43:41 +02:00

Remove a mirrored repository (tullictl + web UI, admin-only) #24

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

qwc-open/tulli#12

No description provided.

Rows
Columns

§8.8 Mirror refresh scheduler #12

Current state (verified @ 6769be5) + the --bare caveat

⚠️ --bare won't absorb rewritten history — needs a forced refspec

Detecting a rewrite and re-evaluating correctly

Related

Current state (verified @ `6769be5`) + the `--bare` caveat

⚠️ `--bare` won't absorb rewritten history — needs a forced refspec