Tally a multi-machine campaign: who has reported what, and is it poolable.
Run this after `git pull` on any participating machine. It answers the three
questions that matter between pulls:
1. COVERAGE - how many of the design's rows exist, per machine, against the
count that machine OWNS under the frozen weights. A machine that has not
pushed is indistinguishable from a machine that has not started, so the
table says "no file" rather than "0 done" for the former.
2. POOLABILITY - whether every reported row carries the same git/solver/
workbook/design/run_cfg hashes. A split here means the rows cannot be
pooled into one Sobol estimate, and it is silent unless something checks.
This reuses merge_and_sobol.check_provenance so the two agree by
construction.
3. THROUGHPUT - completions per hour, computed from the rows themselves.
NOT mean wall over completed rows: with W workers running concurrently,
mean wall over the rows that HAVE finished is biased low early on, because
the slow rows are precisely the ones still running. Every ETA this
campaign got wrong was got wrong that way. Here, throughput is
(rows done) / (elapsed wall on that machine)
where elapsed comes from the machine's own first-to-last row span plus the
wall of the longest row still in flight. It converges from below and
cannot flatter itself.
Usage:
python campaign_status.py # frozen design in machines.json
python campaign_status.py --design design/T3_rev6.csv
python campaign_status.py --at-dose 0.1 # ladder view at one rung
| campaign_tally._warn_orphaned_output |
( |
Path |
res_dir, |
|
|
str |
stem, |
|
|
|
here_k |
|
) |
| |
|
protected |
Is a local worker appending to a file that no longer exists?
A held file handle names an inode, not a path. Anything that replaces the
results file -- `git pull --rebase --autostash` above all, because a live
results file is always dirty -- leaves the worker writing to an unlinked
inode. It is completely silent: the worker log keeps reporting completed
rows while the .jsonl stops growing, and the rows die with the process.
That cost 5 rows and 6.2 core-hours on 2026-08-06 before the log and the
file were compared by hand.
run_ensemble no longer holds the handle, so this cannot recur. The check
stays because it also catches the same shape from any other cause (a
checkout, a stash pop, an editor writing through a temp file), and because
the failure gives no other symptom.
Definition at line 118 of file campaign_tally.py.
