spelunkai/CLAUDE.md

# SpelunkAI — Project Definition

- **Project name:** SpelunkAI
- **Git repository:** https://git.raggonium.net/Jonas/spelunkai
- **Commit convention:** Every commit message must include the name of the Claude model that made it (i.e. the current model name in use at the time of the commit).

## 1. Project Goal

Build a multi-stage AI system that learns to play *Spelunky Classic HD* by combining:

1. Trained perception models (custom, anchor-free CNN detectors) for entity/hazard/item recognition
2. Hard, fast, deterministic high-level logic classifiers for HUD state (health, gold, bombs, ropes, etc.)
3. A control agent that first imitates human input (behavior cloning) and later learns autonomously (reinforcement learning)

The entire perception → reasoning → action loop must fit inside a **66ms compute budget per tick** (~15 Hz), since some detection models may need to run slower than others depending on what they detect. Efficiency is a first-class design constraint, not an afterthought — every architecture decision should be made with this budget in mind.

A distant, non-blocking stretch goal is to eventually port the trained models to run on an FPGA (HDMI capture in, USB/game input out). This does not affect current implementation priorities but should inform model size/efficiency choices where it's free to do so.

---

## 2. System Overview

### 2.1 Hardware / Machines

| Machine | Role | Notes |
|---|---|---|
| New recording PC | Gameplay recording | Ubuntu 24 Desktop, dedicated to running Spelunky Classic HD fullscreen and recording sessions |
| `jai` (existing training server) | Labeling backend, model training, inference, control agent | Ubuntu 24.04, Ryzen 7 5800X, 64GB RAM, RTX 3070 Ti (~7.45 GiB VRAM) |

### 2.2 High-Level Data Flow

```
[Recording PC]
  Spelunky Classic HD (fullscreen, 1280x720)
        │
        ▼
  Recording Script (30fps video + input log)
        │
        ▼
  Lossless MP4 + input log file
        │
        ▼ (transfer to jai)
[jai — Labeling & Training]
  Web-based Labeling Tool
   ├─ Set: Enemy      (Main: Enemy → Sub: Bat, Snake, ...)
   ├─ Set: Items
   ├─ Set: Traps
   └─ Set: ... (ad-hoc, extensible)
        │
        ▼
  Per-set training pipeline (anchor-free CNN detector)
        │
        ▼
  Trained detection models + KPI evaluation
        │
        ▼
[jai — Runtime / Inference Loop, budget: 66ms/tick]
  Screen Capture
        │
        ▼
  Parallel Detection Models (entities, per-set)
        │
        ▼
  High-Level Logic Classifiers (health, gold, bombs, ropes, ... — fixed UI regions)
        │
        ▼
  State Vector Builder
   (entity list + relative positions, HUD status, downsampled raw frame)
        │
        ▼
  Control Agent (behavior cloning → later RL)
        │
        ▼
  Input Execution (keyboard)
```

---

## 3. Component Specifications

### 3.1 Recording Tool

- **Platform:** Ubuntu 24 Desktop (new dedicated machine)
- **Display:** Spelunky Classic HD run in fullscreen at 1280x720
- **Input device:** Keyboard
- **Video output:** Lossless MP4, 30 fps
- **Input log:** Separate log file, input events mapped exactly to the 30fps video timeline (frame index + timestamp + key state)
- **Requirements:**
  - Frame-accurate sync between video frames and logged input state
  - Output format must be easy to slice into individual frames for labeling later

### 3.2 Web-Based Labeling Tool

- **Bounding boxes only** (no segmentation/polygons needed)
- **Hierarchical, ad-hoc class system:** Main class → Sub class (e.g. Main: `Enemy` → Sub: `Bat`, `Snake`, ...). Users can create new main/sub classes on the fly without a schema migration.
- **Multiple independent label sets**, each targeting a specific model (e.g. `Enemy`, `Items`, `Traps`, `PositiveDetection`, ...). Frames may be shared across sets, but labels are tracked per set.
- **Multi-user support**
- **Simple frontend** — functionality over polish
- **All labels must be editable** (move, resize, reclass, delete) at any time
- **Active learning workflow:**
  1. Manually label a small seed set (e.g. ~20 instances of a class)
  2. Train an initial model on the seed set
  3. Run inference on a larger new batch (e.g. ~100 frames) to auto-label
  4. Review/correct the auto-labels in the UI
  5. Promote to a new dataset version (versioned datasets, e.g. `enemy-v1`, `enemy-v2`, ...) for reproducibility
- **Data model must support:** sets, hierarchical classes, dataset versions, per-image/per-set label state (unlabeled / auto-labeled / reviewed)

### 3.3 Detection Models

- **Approach:** Anchor-free, center-heatmap-based detector (CenterNet-style) — NOT YOLO/anchor-based, and NOT a two-stage (R-CNN-style) detector.
  - Rationale: sprites have fairly consistent size/shape, ad-hoc new sub-classes shouldn't require anchor redesign, and heatmap peaks handle multiple same-class objects near each other well (e.g. several bats in frame).
  - Model predicts: a center-point heatmap per class + direct width/height regression from each detected center.
- **One model per label set** (Enemy, Items, Traps, ...), trained and run independently
- **Custom, small/efficient CNN architectures** — sized with the 66ms total-pipeline budget in mind. Multiple detection models must be able to run in parallel (or fast serial) on the RTX 3070 Ti.
- **Inference rate:** 15 fps baseline per model, slower where acceptable depending on what's being detected

### 3.4 KPIs / Evaluation

- **Phase 1:** Standard detection metrics evaluated against held-out labeled data (precision, recall, and an mAP-equivalent adapted for the heatmap/center-point formulation)
- **Phase 2 (later):** Ground-truth validation via direct game-code output / memory reading (approach not yet researched — open item, see Roadmap)

### 3.5 High-Level Logic Classifiers

- Covers HUD/status information: health, gold count, bomb count, rope count, level, etc.
- Since these UI elements sit at **fixed screen positions**, use small, dedicated classifiers per fixed UI region rather than generic OCR — faster and more deterministic.
- These feed directly into the state vector as structured status values.

### 3.6 Control Agent

- **Phase 1 — Behavior Cloning:**
  - Input (state vector) = detected entity list (class + position relative to player) + HUD status (from 3.5) + a downsampled version of the raw frame as additional context
  - Output = predicted input/action, trained to imitate the human recordings
- **Phase 2 — Autonomous learning (later, design open):** likely reinforcement learning; reward function definition is an open item.
- **Multi-policy design goal:** architecture should allow multiple distinct agent "personalities" to be trained later on top of the same perception stack (e.g. a "speedrunner" policy, a "score maximizer" policy), without redesigning the perception layer.

---

## 4. Timing & Performance Constraints

- **Hard budget: 66ms for the entire tick** — screen capture → all detection models → high-level logic classifiers → state vector construction → control agent decision → input execution.
- Detection models should run **in parallel** wherever possible (multi-process/multi-thread or batched on GPU) to maximize how much can happen within the budget.
- Target hardware for this budget: RTX 3070 Ti (current), assuming sufficient compute headroom; model sizes must be chosen accordingly.

---

## 5. Open Items / Roadmap

- [ ] Research approach for KPI validation via game memory reading (Phase 2 evaluation)
- [ ] Define reward function(s) for the autonomous learning (RL) phase
- [ ] Design multi-policy framework for distinct agent behaviors (speedrunner, score-farmer, etc.)
- [ ] Decide on exact heatmap loss formulation / training details for the anchor-free detector
- [ ] Long-term: keep model sizes/architectures FPGA-portability-aware (HDMI capture in, USB output) — not an active work item, just a soft constraint to keep in mind

---

## 6. Tech Stack & Conventions

- **Language:** Python (primary), for both recording tooling, labeling backend, training, and inference
- **Environments:** Ubuntu 24 (recording PC and `jai` training server), `.venv` virtual environments
- **All project files, code comments, and documentation in English**
- **Documentation (READMEs) should be kept current as components are built**
- **Git commits:** every commit message must include the name of the Claude model that authored it