# SpelunkAI — Project Definition - **Project name:** SpelunkAI - **Git repository:** https://git.raggonium.net/Jonas/spelunkai - **Commit convention:** Every commit message must include the name of the Claude model that made it (i.e. the current model name in use at the time of the commit). ## 1. Project Goal Build a multi-stage AI system that learns to play *Spelunky Classic HD* by combining: 1. Trained perception models (custom, anchor-free CNN detectors) for entity/hazard/item recognition 2. Hard, fast, deterministic high-level logic classifiers for HUD state (health, gold, bombs, ropes, etc.) 3. A control agent that first imitates human input (behavior cloning) and later learns autonomously (reinforcement learning) The entire perception → reasoning → action loop must fit inside a **66ms compute budget per tick** (~15 Hz), since some detection models may need to run slower than others depending on what they detect. Efficiency is a first-class design constraint, not an afterthought — every architecture decision should be made with this budget in mind. A distant, non-blocking stretch goal is to eventually port the trained models to run on an FPGA (HDMI capture in, USB/game input out). This does not affect current implementation priorities but should inform model size/efficiency choices where it's free to do so. --- ## 2. System Overview ### 2.1 Hardware / Machines | Machine | Role | Notes | |---|---|---| | New recording PC | Gameplay recording | Ubuntu 24 Desktop, dedicated to running Spelunky Classic HD fullscreen and recording sessions | | `jai` (existing training server) | Labeling backend, model training, inference, control agent | Ubuntu 24.04, Ryzen 7 5800X, 64GB RAM, RTX 3070 Ti (~7.45 GiB VRAM) | ### 2.2 High-Level Data Flow ``` [Recording PC] Spelunky Classic HD (fullscreen, 1280x720) │ ▼ Recording Script (30fps video + input log) │ ▼ Lossless MP4 + input log file │ ▼ (transfer to jai) [jai — Labeling & Training] Web-based Labeling Tool ├─ Set: Enemy (Main: Enemy → Sub: Bat, Snake, ...) ├─ Set: Items ├─ Set: Traps └─ Set: ... (ad-hoc, extensible) │ ▼ Per-set training pipeline (anchor-free CNN detector) │ ▼ Trained detection models + KPI evaluation │ ▼ [jai — Runtime / Inference Loop, budget: 66ms/tick] Screen Capture │ ▼ Parallel Detection Models (entities, per-set) │ ▼ High-Level Logic Classifiers (health, gold, bombs, ropes, ... — fixed UI regions) │ ▼ State Vector Builder (entity list + relative positions, HUD status, downsampled raw frame) │ ▼ Control Agent (behavior cloning → later RL) │ ▼ Input Execution (keyboard) ``` --- ## 3. Component Specifications ### 3.1 Recording Tool - **Platform:** Ubuntu 24 Desktop (new dedicated machine) - **Display:** Spelunky Classic HD run in fullscreen at 1280x720 - **Input device:** Keyboard - **Video output:** Lossless MP4, 30 fps - **Input log:** Separate log file, input events mapped exactly to the 30fps video timeline (frame index + timestamp + key state) - **Requirements:** - Frame-accurate sync between video frames and logged input state - Output format must be easy to slice into individual frames for labeling later ### 3.2 Web-Based Labeling Tool - **Bounding boxes only** (no segmentation/polygons needed) - **Hierarchical, ad-hoc class system:** Main class → Sub class (e.g. Main: `Enemy` → Sub: `Bat`, `Snake`, ...). Users can create new main/sub classes on the fly without a schema migration. - **Multiple independent label sets**, each targeting a specific model (e.g. `Enemy`, `Items`, `Traps`, `PositiveDetection`, ...). Frames may be shared across sets, but labels are tracked per set. - **Multi-user support** - **Simple frontend** — functionality over polish - **All labels must be editable** (move, resize, reclass, delete) at any time - **Active learning workflow:** 1. Manually label a small seed set (e.g. ~20 instances of a class) 2. Train an initial model on the seed set 3. Run inference on a larger new batch (e.g. ~100 frames) to auto-label 4. Review/correct the auto-labels in the UI 5. Promote to a new dataset version (versioned datasets, e.g. `enemy-v1`, `enemy-v2`, ...) for reproducibility - **Data model must support:** sets, hierarchical classes, dataset versions, per-image/per-set label state (unlabeled / auto-labeled / reviewed) ### 3.3 Detection Models - **Approach:** Anchor-free, center-heatmap-based detector (CenterNet-style) — NOT YOLO/anchor-based, and NOT a two-stage (R-CNN-style) detector. - Rationale: sprites have fairly consistent size/shape, ad-hoc new sub-classes shouldn't require anchor redesign, and heatmap peaks handle multiple same-class objects near each other well (e.g. several bats in frame). - Model predicts: a center-point heatmap per class + direct width/height regression from each detected center. - **One model per label set** (Enemy, Items, Traps, ...), trained and run independently - **Custom, small/efficient CNN architectures** — sized with the 66ms total-pipeline budget in mind. Multiple detection models must be able to run in parallel (or fast serial) on the RTX 3070 Ti. - **Inference rate:** 15 fps baseline per model, slower where acceptable depending on what's being detected ### 3.4 KPIs / Evaluation - **Phase 1:** Standard detection metrics evaluated against held-out labeled data (precision, recall, and an mAP-equivalent adapted for the heatmap/center-point formulation) - **Phase 2 (later):** Ground-truth validation via direct game-code output / memory reading (approach not yet researched — open item, see Roadmap) ### 3.5 High-Level Logic Classifiers - Covers HUD/status information: health, gold count, bomb count, rope count, level, etc. - Since these UI elements sit at **fixed screen positions**, use small, dedicated classifiers per fixed UI region rather than generic OCR — faster and more deterministic. - These feed directly into the state vector as structured status values. ### 3.6 Control Agent - **Phase 1 — Behavior Cloning:** - Input (state vector) = detected entity list (class + position relative to player) + HUD status (from 3.5) + a downsampled version of the raw frame as additional context - Output = predicted input/action, trained to imitate the human recordings - **Phase 2 — Autonomous learning (later, design open):** likely reinforcement learning; reward function definition is an open item. - **Multi-policy design goal:** architecture should allow multiple distinct agent "personalities" to be trained later on top of the same perception stack (e.g. a "speedrunner" policy, a "score maximizer" policy), without redesigning the perception layer. --- ## 4. Timing & Performance Constraints - **Hard budget: 66ms for the entire tick** — screen capture → all detection models → high-level logic classifiers → state vector construction → control agent decision → input execution. - Detection models should run **in parallel** wherever possible (multi-process/multi-thread or batched on GPU) to maximize how much can happen within the budget. - Target hardware for this budget: RTX 3070 Ti (current), assuming sufficient compute headroom; model sizes must be chosen accordingly. --- ## 5. Open Items / Roadmap - [ ] Research approach for KPI validation via game memory reading (Phase 2 evaluation) - [ ] Define reward function(s) for the autonomous learning (RL) phase - [ ] Design multi-policy framework for distinct agent behaviors (speedrunner, score-farmer, etc.) - [ ] Decide on exact heatmap loss formulation / training details for the anchor-free detector - [ ] Long-term: keep model sizes/architectures FPGA-portability-aware (HDMI capture in, USB output) — not an active work item, just a soft constraint to keep in mind --- ## 6. Tech Stack & Conventions - **Language:** Python (primary), for both recording tooling, labeling backend, training, and inference - **Environments:** Ubuntu 24 (recording PC and `jai` training server), `.venv` virtual environments - **All project files, code comments, and documentation in English** - **Documentation (READMEs) should be kept current as components are built** - **Git commits:** every commit message must include the name of the Claude model that authored it