# Grouped numerical SFT: portable evidence and score audit

This is the scientific evidence package for the **new run completed on 2026-10-04**, `grouped-rerun-2026-10-04`. It is not a reconstruction or authentication of the missing October 1 grouped-v2 run. Earlier outcome summaries were known when this new study was designed.

## Reproduce the dataset and score audit

Unzip `grouped-rerun-evidence.zip`. With Python 3.10 or newer, run from the extracted package directory:

```sh
python -B audit.py
```

The working directory does not matter: `python -B path/to/extracted/grouped-rerun-evidence/audit.py` also works. Python 3.12 was used for the published check. No installation, network, model files, training or inference is needed. The audit is read-only by default and fails if any payload file is missing, extra, changed or a symlink.

To additionally regenerate all three datasets and the two dataset audit files into a **new directory outside the package**:

```sh
python -B audit.py --regenerate-data ../regenerated-data
```

This recomputes the original dataset generation logic, not a copy of the saved dataset files. All five resulting files must equal the included bytes. The original generator also supports a check directly:

```sh
python -B executed-source/candidate/prepare.py --check
```

The retained source-only unit suite passed all 174 tests during package verification; these use constructed fixtures and do not execute ML training.

Expected audit result: `PASS`, 81 regenerated dataset rows (45 train, 18 dev, 18 test), 180 recomputed scores, 177 automatic outputs, 3 assistant-reviewed outputs, 6 original assistant passes, 144 validated optimizer-step records, and no blank, capped or unresolved response. The pooled final-number-match count is 76/180. Pooled counts are descriptive; repeated held-out questions are not independent datasets.

## Full fixed results

Each cell denominator is 18. All five conditions and both splits are included.

| Condition | Dev final number | Test final number | Dev strict field value | Test strict field value | Dev field present | Test field present |
|---|---:|---:|---:|---:|---:|---:|
| baseline | 9 | 7 | 5 | 2 | 12 | 12 |
| answer_only-seed17 | 6 | 4 | 6 | 4 | 18 | 18 |
| explained-seed17 | 8 | 11 | 8 | 11 | 16 | 18 |
| answer_only-seed29 | 5 | 3 | 5 | 3 | 18 | 18 |
| explained-seed29 | 10 | 13 | 10 | 13 | 18 | 18 |

The explained targets recorded more final-number matches than paired answer-only targets on both splits. Compared with the fresh baseline, explained dev was lower for seed 17 and higher for seed 29; both explained test counts were higher. Answer-only improved field presence while reducing final-number matches relative to baseline. These small descriptive observations do not establish generalization, statistical significance or an isolated benefit of reasoning.

`results/report.public.json` contains every score, extraction detail, original review pass, and full condition/split/task/arm/coverage count. `results/report.md` is the original generated summary. A final-number match checks the extracted numeric magnitude, not units, relevance, correct reasoning or general instruction-following.

## What is checked

The new `audit.py` relocates package discovery, verifies all payload checksums, and calls the byte-identical original dataset generator, raw-record validator, scorer, review adjudicator and counting functions. It verifies:

- Commit-pinned historical source SHA-256/Git blob fingerprints and original source-snapshot manifest
- Regenerated train/dev/test, grouping/overlap audits, protocol and frozen source fingerprints
- All 180 raw files against the original run-completion receipt; question, answer, group, split, condition, output hash, token IDs, EOS and cap accounting
- Three output-only review-queue items, their joins, two saved assistant-pass receipts per item, verbatim quotes, final adjudication bundles and every score object
- Every fixed denominator and condition/split/task/arm count
- All 144 stored training-step records against training audits and the frozen batch plan, paired adapter initialization and monitor summaries

This does not rerun tokenizer decoding, verify omitted binary weights, repeat training/inference, perform new extraction reviews, or recreate the original whole-private-tree integrity check. Hash consistency cannot prove that a physical model execution happened or that reviewers complied with blinding. Assistant passes were neither human review nor statistically independent raters.

## Dataset and execution provenance

The historical v1 source is pinned to [commit b2221ee5f63e19e88f16520651b9491d9f773bb6](https://github.com/qiangzhang-dev/qiangzhang-dev.github.io/tree/b2221ee5f63e19e88f16520651b9491d9f773bb6/experiments/sft-explanation). `executed-source/source-v1/provenance.json` lists individual source URLs, SHA-256 hashes and Git blob IDs. New grouping and split selection are implemented in `executed-source/candidate/prepare.py`.

The 45/18/18 rows contain 15/6/6 per task. Inventory groups on `(packs * size, used)`, discount on `(item subtotal, threshold)` excluding shipping, and percentage on `(initial value, percent)`. One deterministic representative per condition is selected using SHA-256 and the new run ID as salt. Group, ID and prompt overlap across current splits is absent; all splits share templates. Each discount held-out split has three zero and three 20 labels, so a constant-zero answer scores 3/6. All 27 possible discount groups are used; 23 appeared somewhere in public v1 data. Every condition loaded a fresh base.

Model: [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct/tree/7ae557604adf67be50417f59c2c2f167def9a775), revision `7ae557604adf67be50417f59c2c2f167def9a775`. `model-manifest.json` preserves exact asset hashes. `requirements-training-lock.txt` records installed versions, including torch 2.6.0+cpu, transformers 4.51.3, peft 0.15.2 and safetensors 0.5.3. These are observed package versions, not a cross-platform wheel-hash lock or an independently retested installation recipe.

Training used CPU float32 parameters with bfloat16 autocast, eager attention, four intra-op threads, deterministic algorithms, fresh isolated processes, LoRA rank 8/alpha 16/dropout 0 on q_proj and v_proj, paired seeds 17/29, three epochs, batch 4 and 36 optimizer steps per trained condition. AdamW used learning rate 0.0002, weight decay 0, betas 0.9/0.999, epsilon 1e-8 and gradient clipping 1.0. The last batch was retained. Prompts and padding were loss-masked; target and closing tokens were supervised. Dynamic right padding reached at most 130 sequence tokens; no truncation occurred. There were 135 row exposures per trained condition, but **1,095 vs 6,522 shifted supervised tokens** for answer-only vs explained, about 5.96 times more. This is a target-format bundle with unequal token budgets and repeated intermediate information.

Evaluation used the same 18 dev and 18 test questions, batch 1, greedy decoding, 512 new-token ceiling, left padding, EOS 151645, the tokenizer chat template and system message “You are a helpful assistant.” See the exact protocol, execution settings, optimizer settings and per-condition generation-settings JSON for all details. No dev/test selection or outcome-based tuning occurred.

Whole execution took 3173.289 seconds (52.89 minutes). The largest sampled sum of process high-waters was 5,241,552,896 bytes (4.882 GiB); the minimum observed available memory was 4,142,538,752 bytes (3.858 GiB). Guards remained 6 GiB process memory, 2 GiB available memory and 90 minutes. Sampled monitoring is not a hard kernel limit. Preflight attempt 001 failed at the unchanged memory guard; attempt 002 changed only integration-fixture physical width from 256 to the observed 130 and passed. Full comparison subsequently completed without retry or guard stop.

## Repeating training is a separate experiment

No fresh training reproduction was run while making this package. `executed-source/candidate/*.py` are the actual original source bytes; `audit.py` is a newly authored portable wrapper. Historical documentation, protocol and preflight receipts retain their stage-time wording (“source only”, “unfrozen”, “full run untested”); those statements refer to preparation, not current completion.

The original `fixed_run.py --freeze/--execute` is **not a turnkey command for this public subset**. It intentionally binds exact preflight receipts, dependency-file inventories, interpreter bytes and original paths, some of which are omitted or redacted here. Running it against this ZIP will fail those integrity gates. Do not substitute the old completion hash or call a new run the original.

For an independent scientific rerun:

1. Use a fresh Linux/Python 3.12 environment with CPU bfloat16 support and enough headroom for the documented memory guards. Obtain dependencies from official package sources; CPU PyTorch has its own official wheel index. Resolve platform availability separately and record any version changes.
2. Obtain exactly the nine pinned official model/tokenizer files listed in `model-manifest.json` and check all hashes. Do not reuse an old adapter.
3. Preserve the dataset generator, targets, scorer and scientific settings. Use the original `runner.py` tokenization/budget functions and `fixed_run.py` train/evaluate implementations for inspection. Create a separately labeled runner/protocol freeze with new provenance anchors and portable locations; do not disable scientific validation or erase the old evidence.
4. Repeat a guarded tokenizer/integration preflight, freeze all new source/data/model/dependency hashes, and execute five isolated conditions without outcome-driven changes. Preserve failures as well as successes. Hardware/numerical differences may prevent identical model outputs.
5. Reapply the frozen scorer to all 180 outputs, preserve any new output-only assistant extraction passes, and report the result as a new replication. Compare it with this run only after collection is complete.

This package provides exact methods and audit evidence; it does not claim bit-identical training portability or independent replication.

## Files, exclusions and integrity

- `executed-source/`: byte-identical executed Python source and historical source/data; two historical command documents and one preflight amendment are sanitized derivatives
- `run/`: 180 exact raw JSON files, 144 exact step records, original completion receipt, initialization/budget/settings/monitor summaries, and a sanitized frozen-manifest derivative
- `reviews/`: original queue and six pass receipts plus a sanitized post-review join
- `results/`: original Markdown summary and sanitized JSON report whose scientific values and all row/score objects are unchanged
- `TRANSFORMATIONS.json`: original source hashes, public-copy hashes, explicit changes and omissions
- `SHA256SUMS`: hashes every payload file except itself; the detached ZIP checksum covers the entire archive including this manifest

Weights, adapters, installation inventories, individual memory samples, launch records, private storage identifiers and local machine paths are omitted. Public attribution URLs remain because they identify the scientific source. No credentials or private archive links are included. Derivative JSON must not be presented as the exact original freeze/report bytes. Original source/run/report and private archives were not rewritten.

This is a small synthetic English text pilot with one small model and two paired seeds. It is not a speech-model study or an employer benchmark. Correct final values do not establish correct reasoning.