Jonas
07e75247c4
Add the anchor-free, center-heatmap detector CLAUDE.md §3.3 specifies: - targets.py: encodes ground-truth boxes into a per-class Gaussian heatmap + wh regression target + center mask, using the standard CornerNet/CenterNet gaussian-radius formulation. - model.py: a small conv backbone (stride 4) with heatmap (sigmoid) and wh regression heads - exactly the two outputs CLAUDE.md specifies, sized with the 66ms/tick budget in mind. - losses.py: modified focal loss (heatmap) + masked L1 (wh), combined with the standard CenterNet wh_weight=0.1. - client.py / dataset.py: pull a promoted dataset version + its set's class list from the labeling backend and turn it into a torch.utils.data.Dataset, reading images from local disk (same machine as FRAMES_ROOT). - train.py: wires it into a basic DataLoader -> train loop -> per-epoch checkpoint. The exact loss weighting/architecture sizing is a reasonable, standard v1 default, not a tuned final answer - CLAUDE.md's own roadmap flags the exact formulation as still open; this is the starting point to iterate from. 16/16 unit tests pass on CPU with synthetic data (target/model/loss correctness, dataset-version parsing). Beyond that, ran a real end-to-end smoke test: live labeling backend -> promoted dataset version -> `spelunkai-train` actually training one real epoch against it and writing a checkpoint. Not verified: multi-epoch convergence on real data, which needs jai's GPU and real labeled frames. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
SpelunkAI
A multi-stage AI system that learns to play Spelunky Classic HD: trained perception models, deterministic HUD-state classifiers, and a control agent that first imitates human play (behavior cloning) and later learns autonomously (reinforcement learning).
Full project spec, architecture, timing constraints, and roadmap live in
CLAUDE.md — that document is the source of truth. This README is just
a map of the repo.
Layout
| Directory | Component | Spec section |
|---|---|---|
recording/ |
Recording Tool — captures gameplay video + input log on the recording PC | 3.1 |
labeling/ |
Web-Based Labeling Tool — backend + frontend for bounding-box labeling | 3.2 |
training/ |
Per-set training pipeline for the anchor-free CNN detectors | 3.3 |
inference/ |
Runtime inference loop — capture → detectors → HUD classifiers → state vector, within the 66ms/tick budget | 3.4, 3.5 |
control/ |
Control Agent — behavior cloning now, RL later | 3.6 |
Each component directory has its own README with more detail and its own .venv
(per CLAUDE.md §6), since components run on different machines (recording PC vs.
jai) and have independent dependencies.
Status
Repo scaffold only — no components are implemented yet.