v0.3.4: ship the real minGPT measurement (3.11×) + two correctness/dotenv fixes

Headline: 3.11× faster (-67.8% step time), -68.7% peak memory on Karpathy's
minGPT (single A100, BF16+TF32+torch.compile-max-autotune+SDPA+Fused AdamW).
Correctness curves match within BF16-widened tolerance.

Fixes:
- load_dotenv now uses find_dotenv(usecwd=True). Default usecwd=False walks up
  from the script's directory (e.g. ~/anaconda3/bin/), which never has a .env,
  so the binary silently failed to pick up OPENAI_API_KEY from the project root.
- _strip_warmup no longer chops loss_values per-payload. Auto-detected
  stabilization gave different warmup counts for baseline vs torch.compile'd
  candidate (10 vs 15), so correctness compared misaligned training steps
  and reported spurious FAILs with ~0.18 loss divergence.
- SUMMARY.md architecture rendering: evidence-wrapped {value,confidence,evidence}
  fields and compound fields now render as "k=v, k=v" instead of raw JSON soup.

Artifacts:
- examples/minGPT/profine_output/ — the actual JSON + MD outputs from the
  reproducible run, checked in so the README's headline number is verifiable.
- README Results table updated with measured numbers (was 3.1×/66.4% placeholder).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: ai-singhal

.gitignore

@@ -42,5 +42,13 @@ uv.lock
 .profine_tmp/
 .profine_optimized_*.py
 profine_output*/
+# Exception: the canonical demo artifacts in examples/minGPT/ ARE checked in
+# as evidence the headline numbers in the README are reproducible.
+!examples/minGPT/profine_output/
+!examples/minGPT/profine_output/**
 quickstart_out/
 dist/
+
+# profine-output _debug dumps (per-run, not for the repo)
+examples/**/profine_output/_debug/
+examples/**/run.log

CHANGELOG.md

@@ -2,6 +2,13 @@
 
 All notable changes to `profine` are documented here. This project follows [Semantic Versioning](https://semver.org/).
 
+## [0.3.4] — 2026-05-12
+
+### Fixed
+- **`.env` loading from installed binary.** `load_dotenv()` defaults to searching upward from the entry-script directory (e.g. `~/anaconda3/bin/`), which never has a `.env`. Now passes `find_dotenv(usecwd=True)` explicitly, so `OPENAI_API_KEY` (and friends) in your project's `.env` are picked up when you run `profine` from your project root.
+- **Spurious "correctness: FAIL" verdicts.** `_strip_warmup` used to chop `loss_values` per-payload using auto-detected stabilization points, which yielded different warmup counts for baseline vs. optimized (e.g. 10 vs. 15 with `torch.compile`). The correctness check then compared baseline-step-10 against optimized-step-15 — five training steps apart — and flagged the natural divergence as a failure. Now `_strip_warmup` only touches `step_times_ms`; losses stay aligned by original training-step index.
+- **Cleaner `SUMMARY.md` architecture section.** Evidence-wrapped and compound architecture fields now render as `key=value, key=value` instead of dumping raw nested JSON.
+
 ## [0.3.3] — 2026-05-12
 
 ### Added

README.md

@@ -15,19 +15,23 @@ Profile your PyTorch code on real GPUs. Get a transparent rewrite. Ship measured
 
 ## Results
 
-On [Karpathy's minGPT](https://github.com/karpathy/minGPT), single-A100:
+On [Karpathy's minGPT](https://github.com/karpathy/minGPT) (single A100, measured end-to-end by profine):
 
 | Metric | Baseline | profine | Δ |
 |---|---|---|---|
-| Step time | 1.00× | **3.1× faster** | −67.7% ms/step |
-| Peak memory | 1.00× | **−66.4%** | substantial headroom for larger batch |
+| Step time (ms) | 25.22 | 8.11 | **−67.8%** (**3.11× faster**) |
+| Peak memory (GB) | 1.43 | 0.45 | **−68.7%** |
+| Correctness (loss curves match) | ✓ | ✓ | within BF16-widened tolerance |
 
-Reproducible with some variation of (as shown in the [demo](https://youtu.be/CY9aW1Dcrn0)):
+Stack applied: **BF16 Mixed Precision + TF32 matmul + torch.compile (max-autotune) + SDPA + Fused AdamW.** Reproducible with:
 
 ```bash
-profine run-all examples/minGPT/projects/chargpt/chargpt.py --hardware 1x_a100 --steps 25 --warmup 10
+profine run-all examples/minGPT/projects/chargpt/chargpt.py \
+  --hardware 1x_a100 --steps 25 --warmup 10 --seed 42
 ```
 
+Full artifacts (JSON + Markdown reports for every pipeline step) live in [`examples/minGPT/profine_output/`](examples/minGPT/profine_output/) — read [`SUMMARY.md`](examples/minGPT/profine_output/SUMMARY.md) first. As shown in the [demo](https://youtu.be/CY9aW1Dcrn0).
+
 ## Install
 
 ```bash

examples/minGPT/profine_output/SUMMARY.md

@@ -0,0 +1,56 @@
+# profine run-all — examples/minGPT/projects/chargpt/chargpt.py
+
+**Hardware:** `1x_a100`
+
+## ✅ 67.8% faster (3.11× speedup), correctness preserved.
+
+## Architecture (what we found in your code)
+
+- **Model:** GPT
+- **Framework:** PyTorch
+- **Precision:** training_dtype=float32
+- **Optimizer:** name=AdamW, learning_rate=0.0005
+- **Dataloader:** dataset_class=CharDataset
+
+## Bottleneck (what's slowing it down)
+
+- **matmul**
+- **elementwise**
+- **optimizer**
+
+## Optimizations
+
+**Ranked by LLM ROI:**
+
+1. `bf16_mixed_precision` — This can materially accelerate the matmul-heavy workload on A100 by moving compute to tensor cores and reducing activation bandwidth, but it is more intrusive t
+2. `fp16_mixed_precision` — Similar upside to bf16 for compute throughput, but slightly riskier because it needs GradScaler and is more prone to underflow
+3. `tf32_matmul` — Best ROI: the run is dominated by fp32 GEMMs (46
+4. `torch_compile_max_autotune` — Strong fit because the profile shows substantial elementwise work (15
+5. `torch_compile` — Useful for fusing the observed elementwise work and reducing Python/kernel-launch overhead, but likely lower ROI than max-autotune
+
+**Skipped (4):** `fp16_mixed_precision` (exclusive group 2 — conflicts with already-applied optimization), `torch_compile` (exclusive group 3 — conflicts with already-applied optimization), `dataloader_workers` (The input pipeline is already lightweight and the optimization record states that increasing DataLoader workers would add CPU/memory overhead with essentially no expected return for this CharDataset, so it should be skipped.), `foreach_adamw` (exclusive group 4 — conflicts with already-applied optimization)
+
+## Benchmark (measured on-GPU)
+
+| Metric | Δ |
+|---|---|
+| Step time | **+67.8%** (3.11× faster) |
+| Peak memory | -68.7% |
+| GPU utilization | -11.4% |
+| Verdict | **PASS** |
+| Correctness | ✓ pass |
+
+## Artifacts
+
+- [`read/architecture_record.json`](read/architecture_record.json) — Parsed architecture (JSON)
+- [`read/architecture_brief.md`](read/architecture_brief.md) — Architecture brief (MD)
+- [`profile/profile_record.json`](profile/profile_record.json) — Profile data (JSON)
+- [`profile/profile_report.md`](profile/profile_report.md) — Profile report (MD)
+- [`interpret/bottleneck_report.json`](interpret/bottleneck_report.json) — Bottleneck diagnosis (JSON)
+- [`interpret/bottleneck_brief.md`](interpret/bottleneck_brief.md) — Bottleneck brief (MD)
+- [`suggest/suggestion_report.json`](suggest/suggestion_report.json) — Ranked optimizations (JSON)
+- [`suggest/suggestion_brief.md`](suggest/suggestion_brief.md) — Suggestion brief (MD)
+- [`edit/edited_train.py`](edit/edited_train.py) — Optimized training script
+- [`edit/change_manifest.json`](edit/change_manifest.json) — What was changed and why
+- [`benchmark/benchmark_comparison.json`](benchmark/benchmark_comparison.json) — Benchmark comparison (JSON)
+- [`benchmark/benchmark_report.md`](benchmark/benchmark_report.md) — Benchmark report (MD)

examples/minGPT/profine_output/benchmark/benchmark_comparison.json

@@ -0,0 +1,49 @@
+{
+  "metrics": [
+    {
+      "name": "step_time_median_ms",
+      "baseline": 25.2165,
+      "candidate": 8.111,
+      "delta": -17.1056,
+      "delta_pct": -67.83,
+      "improved": true
+    },
+    {
+      "name": "throughput_steps_per_sec",
+      "baseline": 39.6565,
+      "candidate": 123.2901,
+      "delta": 83.6336,
+      "delta_pct": 210.9,
+      "improved": true
+    },
+    {
+      "name": "memory_peak_gb",
+      "baseline": 1.4278,
+      "candidate": 0.4472,
+      "delta": -0.9806,
+      "delta_pct": -68.68,
+      "improved": true
+    },
+    {
+      "name": "gpu_util_mean_pct",
+      "baseline": 15.5833,
+      "candidate": 4.155,
+      "delta": -11.4283,
+      "delta_pct": -73.34,
+      "improved": false
+    }
+  ],
+  "correctness": {
+    "passed": true,
+    "loss_match": true,
+    "max_loss_diff": 0.03388619422912598,
+    "rtol": 0.05,
+    "atol": 0.01,
+    "notes": "Loss curves match across 15 steps (max diff: 0.033886)"
+  },
+  "speedup_pct": 67.83,
+  "memory_delta_pct": -68.68,
+  "util_delta_pct": -11.428341606913037,
+  "verdict": "PASS",
+  "summary": "PASS | 67.8% faster | 68.7% less memory | GPU util -11pp | correctness: PASS"
+}

examples/minGPT/profine_output/benchmark/benchmark_report.md

@@ -0,0 +1,41 @@
+# Benchmark Report
+
+## ✅ 67.8% faster (3.11× speedup), correctness preserved.
+
+**Optimization applied:** BF16 Mixed Precision + TF32 matmul precision + torch.compile mode='max-autotune' + Scaled Dot Product Attention (SDPA) + Fused AdamW
+**Hardware:** 1x_a100 ($2.50/hr)
+**Verdict:** PASS
+**Notes:** PASS | 67.8% faster | 68.7% less memory | GPU util -11pp | correctness: PASS
+
+---
+
+## Metrics
+
+| Metric | Baseline | Optimized | Δ | |
+|---|---|---|---|---|
+| Step time (ms) | 25.22 | 8.11 | -67.8% | ↑ improved |
+| Throughput (steps/s) | 39.66 | 123.29 | +210.9% | ↑ improved |
+| Peak memory (GB) | 1.43 | 0.45 | -68.7% | ↑ improved |
+| GPU utilization (%) | 15.6 | 4.2 | -73.3% | ↓ regressed |
+
+## Projected Savings
+
+For every **100 hours** of training time saved at the optimized step time, you'd have spent **311 hours** on the baseline.
+
+| Baseline run length | Time saved | Cost saved |
+|---|---|---|
+| 1 hr | 0.68 hr (41 min) | $1.70 |
+| 10 hr | 6.78 hr (407 min) | $16.96 |
+| 100 hr | 67.83 hr (4070 min) | $169.57 |
+| 1000 hr | 678.30 hr (40698 min) | $1695.75 |
+
+## Correctness
+
+- **Loss curves match:** Yes ✓
+- **Max loss diff:** 0.033886
+- **Tolerance:** rtol=0.05, atol=0.01
+- **Notes:** Loss curves match across 15 steps (max diff: 0.033886)
+
+## Recommendation
+
+**Ship it.** Speedup exceeds the 3% threshold and correctness passed.

examples/minGPT/profine_output/edit/01_bf16_mixed_precision/change_manifest.json

@@ -0,0 +1,27 @@
+{
+  "applied": true,
+  "optimization_id": "bf16_mixed_precision",
+  "explanation": "Applied BF16 mixed precision by autocasting the model forward pass during training. This keeps the model, loss, optimizer, and training loop behavior intact while allowing CUDA matmuls/activations to run in bfloat16 on supported hardware.",
+  "changes": [
+    {
+      "path": "mingpt/trainer.py",
+      "line_start": 92,
+      "line_end": 94,
+      "description": "Wrapped the training forward pass in CUDA bfloat16 autocast to enable BF16 mixed precision.",
+      "original_snippet": "            # forward the model\n            logits, self.loss = model(x, y)\n",
+      "new_snippet": "            # forward the model\n            with torch.autocast(device_type='cuda', dtype=torch.bfloat16): # profine: bf16_mixed_precision\n                logits, self.loss = model(x, y)\n"
+    }
+  ],
+  "new_imports": [],
+  "warnings": [
+    "This requires bf16-capable CUDA hardware; on unsupported setups, torch.autocast with device_type='cuda' may not provide the intended benefit.",
+    "The autocast scope is limited to the forward pass only, so backward and optimizer steps remain in full precision as before."
+  ],
+  "not_applicable_reason": null,
+  "extra_file_edits": [
+    {
+      "path": "mingpt/trainer.py",
+      "diff": "--- mingpt/trainer.py+++ mingpt/trainer.py (optimized)@@ -90,7 +90,8 @@             x, y = batch\n \n             # forward the model\n-            logits, self.loss = model(x, y)\n+            with torch.autocast(device_type='cuda', dtype=torch.bfloat16): # profine: bf16_mixed_precision\n+                logits, self.loss = model(x, y)\n \n             # backprop and update the parameters\n             model.zero_grad(set_to_none=True)\n"
+    }
+  ]
+}

examples/minGPT/profine_output/edit/01_bf16_mixed_precision/edited_train.py

@@ -0,0 +1,133 @@
+"""
+Trains a character-level language model.
+"""
+
+import os
+import sys
+
+import torch
+from torch.utils.data import Dataset
+from torch.utils.data.dataloader import DataLoader
+
+from mingpt.model import GPT
+from mingpt.trainer import Trainer
+from mingpt.utils import set_seed, setup_logging, CfgNode as CN
+
+# -----------------------------------------------------------------------------
+
+def get_config():
+
+    C = CN()
+
+    # system
+    C.system = CN()
+    C.system.seed = 3407
+    C.system.work_dir = './out/chargpt'
+
+    # data
+    C.data = CharDataset.get_default_config()
+
+    # model
+    C.model = GPT.get_default_config()
+    C.model.model_type = 'gpt-mini'
+
+    # trainer
+    C.trainer = Trainer.get_default_config()
+    C.trainer.learning_rate = 5e-4 # the model we're using is so small that we can go a bit faster
+
+    return C
+
+# -----------------------------------------------------------------------------
+
+class CharDataset(Dataset):
+    """
+    Emits batches of characters
+    """
+
+    @staticmethod
+    def get_default_config():
+        C = CN()
+        C.block_size = 128
+        return C
+
+    def __init__(self, config, data):
+        self.config = config
+
+        chars = sorted(list(set(data)))
+        data_size, vocab_size = len(data), len(chars)
+        print('data has %d characters, %d unique.' % (data_size, vocab_size))
+
+        self.stoi = { ch:i for i,ch in enumerate(chars) }
+        self.itos = { i:ch for i,ch in enumerate(chars) }
+        self.vocab_size = vocab_size
+        self.data = data
+
+    def get_vocab_size(self):
+        return self.vocab_size
+
+    def get_block_size(self):
+        return self.config.block_size
+
+    def __len__(self):
+        return len(self.data) - self.config.block_size
+
+    def __getitem__(self, idx):
+        # grab a chunk of (block_size + 1) characters from the data
+        chunk = self.data[idx:idx + self.config.block_size + 1]
+        # encode every character to an integer
+        dix = [self.stoi[s] for s in chunk]
+        # return as tensors
+        x = torch.tensor(dix[:-1], dtype=torch.long)
+        y = torch.tensor(dix[1:], dtype=torch.long)
+        return x, y
+
+# -----------------------------------------------------------------------------
+
+if __name__ == '__main__':
+
+    # get default config and overrides from the command line, if any
+    config = get_config()
+    config.merge_from_args(sys.argv[1:])
+    print(config)
+    setup_logging(config)
+    set_seed(config.system.seed)
+
+    # construct the training dataset
+    text = open('input.txt', 'r').read() # don't worry we won't run out of file handles
+    train_dataset = CharDataset(config.data, text)
+
+    # construct the model
+    config.model.vocab_size = train_dataset.get_vocab_size()
+    config.model.block_size = train_dataset.get_block_size()
+    model = GPT(config.model)
+
+    # construct the trainer object
+    trainer = Trainer(config.trainer, model, train_dataset)
+
+    # iteration callback
+    def batch_end_callback(trainer):
+
+        if trainer.iter_num % 10 == 0:
+            print(f"iter_dt {trainer.iter_dt * 1000:.2f}ms; iter {trainer.iter_num}: train loss {trainer.loss.item():.5f}")
+
+        if trainer.iter_num % 500 == 0:
+            # evaluate both the train and test score
+            model.eval()
+            with torch.no_grad():
+                # sample from the model...
+                context = "O God, O God!"
+                x = torch.tensor([train_dataset.stoi[s] for s in context], dtype=torch.long)[None,...].to(trainer.device)
+                y = model.generate(x, 500, temperature=1.0, do_sample=True, top_k=10)[0]
+                completion = ''.join([train_dataset.itos[int(i)] for i in y])
+                print(completion)
+            # save the latest model
+            print("saving model")
+            ckpt_path = os.path.join(config.system.work_dir, "model.pt")
+            torch.save(model.state_dict(), ckpt_path)
+            # revert model to training mode
+            model.train()
+
+    trainer.set_callback('on_batch_end', batch_end_callback)
+
+    # run the optimization
+    trainer.run()