Jonas
2989a0814c
Slice a recorded session's lossless video into individual frame PNGs (ffmpeg -vsync 0, no drop/dup) via a new `extract-frames` subcommand, with frame count cross-checked against the manifest and input log so any capture-rate drift surfaces immediately instead of silently misaligning frames and logged input later. Includes a real end-to-end test against an ffmpeg-generated synthetic video (skipped when ffmpeg isn't installed, e.g. bare WSL). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
84 lines
3.4 KiB
Markdown
84 lines
3.4 KiB
Markdown
# Recording Tool
|
|
|
|
Captures *Spelunky Classic HD* gameplay on the dedicated recording PC (Ubuntu 24
|
|
Desktop, fullscreen 1280x720) as a lossless 30fps MP4, alongside a separate input log
|
|
mapped frame-accurately to the video timeline (frame index + timestamp + key state).
|
|
Output must be easy to slice into individual frames for labeling.
|
|
|
|
See CLAUDE.md §3.1 for full requirements.
|
|
|
|
**Status:** core capture + input logging implemented; not yet run against a live
|
|
recording session on real hardware (developed off the recording PC — see Testing
|
|
below).
|
|
|
|
## How it works
|
|
|
|
- **Video** (`capture.py`): shells out to `ffmpeg -f x11grab ... -c:v libx264rgb -qp 0`
|
|
to record the screen region losslessly (true RGB, no chroma subsampling) at a fixed
|
|
framerate. Requires an **Xorg session** — `x11grab` does not work under Wayland, so
|
|
the recording PC must log in via "Ubuntu on Xorg".
|
|
- **Input** (`input_logger.py`): reads a raw `/dev/input/eventX` keyboard device via
|
|
`evdev` directly (bypasses the window system entirely), so key state is captured
|
|
reliably even while the game holds exclusive fullscreen focus.
|
|
- **Sync** (`session.py`): a single frame-tick loop, paced off one monotonic clock,
|
|
starts both video and keyboard logging together and writes one JSONL row per video
|
|
frame (`{"frame": i, "t": seconds, "keys": [...]}`) plus a `_manifest.json` with fps/
|
|
resolution/frame count. Because both are ticked from the same start time at the same
|
|
fixed rate, frame `i` in the log lines up with frame `i` of the video without
|
|
post-hoc alignment.
|
|
- **Frame extraction** (`frames.py`): slices a session's video into individual
|
|
`frame_%06d.png` images (via `ffmpeg -vsync 0`, so no frames are silently dropped or
|
|
duplicated during extraction) — the bridge to the Labeling Tool, which works on
|
|
frame images rather than video. Cross-checks the extracted frame count against the
|
|
manifest's `frame_count` and the input log's line count, and warns if they disagree
|
|
— the one place actual capture drift (if the recording PC couldn't sustain 30fps)
|
|
would surface, since it's not otherwise possible to detect from the video alone.
|
|
|
|
## Setup
|
|
|
|
```
|
|
python -m venv .venv
|
|
source .venv/bin/activate
|
|
pip install -e ".[dev]"
|
|
```
|
|
|
|
Reading `/dev/input/eventX` requires the `input` group (or root):
|
|
|
|
```
|
|
sudo usermod -aG input $USER # then log out/in
|
|
```
|
|
|
|
## Usage
|
|
|
|
```
|
|
# find the keyboard device path
|
|
spelunkai-record list-devices
|
|
|
|
# record until Ctrl+C
|
|
spelunkai-record record --input-device /dev/input/event3 --output-dir recordings
|
|
|
|
# fixed-length session
|
|
spelunkai-record record --input-device /dev/input/event3 --duration 120 --name run01
|
|
|
|
# slice a recorded session into individual frame images for labeling
|
|
spelunkai-record extract-frames --output-dir recordings --name run01
|
|
```
|
|
|
|
## Testing
|
|
|
|
Unit tests (`tests/`) cover the frame-tick pacing, JSONL/manifest output, and the
|
|
ffmpeg command construction using fake keyboard/video components — no real device or
|
|
ffmpeg binary required, so they run anywhere (including WSL):
|
|
|
|
```
|
|
pytest
|
|
```
|
|
|
|
`test_frames.py` additionally runs a real end-to-end extraction against an
|
|
`ffmpeg`-generated synthetic test video (no display or game needed) when `ffmpeg` is
|
|
on `PATH`; it's skipped automatically otherwise (e.g. a bare WSL install without
|
|
`ffmpeg`).
|
|
|
|
Actual capture (`VideoCapture`/`KeyboardState` against real hardware) still needs to be
|
|
validated end-to-end on the real recording PC with Spelunky Classic HD running.
|