nes-emu-debug

name: nes-emu-debug description: >- Run NES ROMs in the Mesen2 emulator to debug runtime behavior. Use this skill whenever the user wants to run a ROM and inspect what actually happens at runtime: read NES memory (RAM, palette, nametable, OAM), dump CPU/PPU/APU state after N frames, capture the screen buffer, compare runtime behavior between two ROMs, or verify that a sample displays correctly. Also use when the user says things like "run the ROM", "what does the screen look like", "check the palette", "inspect nametable", "read zero page", "dump memory", or "the ROM doesn't display correctly". This is the dynamic/runtime counterpart to nes-rom-debug (which does static binary analysis).

NES Emulator Debug (Mesen2)

Run NES ROMs headlessly in Mesen2's test runner to inspect runtime state, memory, and screen output. This complements the nes-rom-debug skill (static binary analysis) with dynamic runtime inspection.

Prerequisites

Build the Mesen2 package first:

dotnet build src/dotnes.mesen/dotnes.mesen.csproj

This downloads Mesen2 to src/dotnes.mesen/obj/Debug/mesen/. The executable is:

• Windows: src/dotnes.mesen/obj/Debug/mesen/Mesen.exe
• Linux/macOS: src/dotnes.mesen/obj/Debug/mesen/Mesen

How It Works

Mesen2 has a headless test runner mode that loads a ROM, runs a Lua script, and exits — no window, no GUI. The Lua script has full access to the emulator API: CPU/PPU/APU state, all memory types, screen buffer, frame callbacks, etc.

Basic Invocation

# Find the Mesen executable (works on any OS / config)
$mesenDir = Get-ChildItem "src/dotnes.mesen/obj/*/mesen" -Directory | Select-Object -First 1
$mesenExe = if ($IsWindows -or $env:OS -match 'Windows') {
  Join-Path $mesenDir "Mesen.exe"
} else {
  Join-Path $mesenDir "Mesen"
}
$mesen = (Resolve-Path $mesenExe).Path
$rom   = (Resolve-Path "path/to/rom.nes").Path
$script = (Resolve-Path "path/to/script.lua").Path

$proc = Start-Process -FilePath $mesen -ArgumentList `
  "--testRunner",            # Headless mode (no window)
  "--enableStdout",          # Show ROM loading info on stdout
  "--doNotSaveSettings",     # Don't persist config changes
  "--timeout=30",            # Kill after N seconds if script doesn't call emu.stop()
  "--debug.scriptWindow.allowIoOsAccess=true",  # Enable Lua file I/O
  $script, $rom `
  -PassThru -RedirectStandardOutput out.txt -RedirectStandardError err.txt -NoNewWindow
$proc.WaitForExit(35000)
# Exit code comes from emu.stop(N) in the Lua script, or -1 on timeout

Key Command-Line Flags

Lua Script API Reference

State Access

The state table returned by emu.getState() uses flat dotted-string keys, not nested tables. Access fields with bracket notation:

local state = emu.getState()
-- CORRECT:
local pc = state["cpu.pc"]
local scanline = state["ppu.scanline"]
-- WRONG (nil!):
-- local pc = state.cpu.pc

Key state fields:

Memory Types

Use emu.read(address, memType) to read memory. The address is relative to the start of that memory region (not the CPU/PPU mapped address).

Important: For reading zero page / RAM, use nesInternalRam (direct access), not nesMemory (which goes through the CPU bus and may return stale/zero values in test runner mode). Similarly, use nesPaletteRam for palette, not nesPpuMemory.

Frame Callbacks

Register a callback to run after each frame:

local frameCount = 0
emu.addEventCallback(function()
  frameCount = frameCount + 1
  if frameCount == 120 then
    -- Do your inspection here
    emu.stop(0)  -- exit with code 0
  end
end, emu.eventType.endFrame)

Screen Buffer

local pixels = emu.getScreenBuffer()
-- Returns a Lua table of 61440 ARGB integer values (256 × 240 NES resolution)
-- Each pixel: (A << 24) | (R << 16) | (G << 8) | B

To save as raw RGB file:

local sf = io.open("screen.raw", "wb")
for i = 1, #pixels do
  local p = pixels[i]
  sf:write(string.char((p >> 16) & 0xFF, (p >> 8) & 0xFF, p & 0xFF))
end
sf:close()
-- Result: 184320 bytes (256 × 240 × 3 RGB), convertible with ImageMagick/ffmpeg

Convert to PNG (from PowerShell):

# Using Python PIL/Pillow
python -c "
from PIL import Image
raw = open('screen.raw','rb').read()
img = Image.frombytes('RGB', (256,240), raw)
img.save('screenshot.png')
"
# Or using ffmpeg
ffmpeg -f rawvideo -pixel_format rgb24 -video_size 256x240 -i screen.raw screenshot.png

Exiting

emu.stop(0)    -- exit test runner with code 0 (success)
emu.stop(1)    -- exit with code 1 (failure)
emu.stop(42)   -- any integer exit code

The exit code is returned as the process exit code, so you can check it from PowerShell via $proc.ExitCode.

Lua Script Templates

Template 1: Dump CPU State After N Frames

-- dump_state.lua — Dump CPU/PPU state after N frames
-- Usage: Mesen.exe --testRunner ... dump_state.lua rom.nes
local FRAMES = 120  -- wait 2 seconds (60 fps)
local OUT_FILE = "OUTPUT_PATH"  -- replace with absolute path using forward slashes

local frameCount = 0
emu.addEventCallback(function()
  frameCount = frameCount + 1
  if frameCount == FRAMES then
    local state = emu.getState()
    local f = io.open(OUT_FILE, "w")

    f:write("=== CPU State ===\n")
    f:write(string.format("PC=$%04X  A=$%02X  X=$%02X  Y=$%02X  SP=$%02X  PS=$%02X\n",
      state["cpu.pc"], state["cpu.a"], state["cpu.x"],
      state["cpu.y"], state["cpu.sp"], state["cpu.ps"]))
    f:write("Cycles: " .. state["cpu.cycleCount"] .. "\n")

    f:write("\n=== PPU State ===\n")
    f:write("Scanline: " .. state["ppu.scanline"] .. "  Cycle: " .. state["ppu.cycle"] .. "\n")
    f:write("Frame: " .. state["ppu.frameCount"] .. "\n")
    f:write("NMI enabled: " .. tostring(state["ppu.control.nmiOnVerticalBlank"]) .. "\n")
    f:write("BG enabled: " .. tostring(state["ppu.mask.backgroundEnabled"]) .. "\n")
    f:write("Sprites enabled: " .. tostring(state["ppu.mask.spritesEnabled"]) .. "\n")

    f:close()
    emu.stop(0)
  end
end, emu.eventType.endFrame)

Template 2: Read NES Library Zero Page Variables

The NES library stores state in zero page. Key addresses (from src/dotnes.tasks/Utilities/NESConstants.cs):

-- read_zeropage.lua
local f = io.open("OUTPUT_PATH", "w")

local frameCount = 0
emu.addEventCallback(function()
  frameCount = frameCount + 1
  if frameCount == 120 then
    f:write("=== NES Library Zero Page ===\n")
    -- Names from src/dotnes.tasks/Utilities/NESConstants.cs
    local names = {
      [0x01]="STARTUP", [0x02]="NES_PRG_BANKS", [0x03]="VRAM_UPDATE",
      [0x04]="NAME_UPD_ADR", [0x06]="NAME_UPD_ENABLE", [0x07]="PAL_UPDATE",
      [0x08]="PAL_BG_PTR", [0x0A]="PAL_SPR_PTR",
      [0x0C]="SCROLL_X", [0x0D]="SCROLL_Y", [0x0E]="SCROLL_X1",
      [0x0F]="PPU_CTRL_VAR1", [0x10]="PRG_FILEOFFS", [0x12]="PPU_MASK_VAR",
      [0x14]="NMI_CALLBACK", [0x17]="TEMP", [0x18]="TEMP_HI",
      [0x19]="TEMP2", [0x1A]="TEMP3", [0x1B]="OAM_OFF", [0x1C]="UPDPTR",
      [0x22]="sp", [0x3C]="RAND_SEED"
    }

    for addr = 0, 31 do
      local val = emu.read(addr, emu.memType.nesInternalRam)
      local name = names[addr] or ""
      f:write(string.format("  $%02X = $%02X  %s\n", addr, val, name))
    end

    f:close()
    emu.stop(0)
  end
end, emu.eventType.endFrame)

Template 3: Dump Palette

-- dump_palette.lua
local f = io.open("OUTPUT_PATH", "w")

local frameCount = 0
emu.addEventCallback(function()
  frameCount = frameCount + 1
  if frameCount == 120 then
    f:write("=== Palette RAM ===\n")
    f:write("Background palettes:\n")
    for pal = 0, 3 do
      f:write(string.format("  Palette %d: ", pal))
      for i = 0, 3 do
        local val = emu.read(pal * 4 + i, emu.memType.nesPaletteRam)
        f:write(string.format("$%02X ", val))
      end
      f:write("\n")
    end

    f:write("Sprite palettes:\n")
    for pal = 0, 3 do
      f:write(string.format("  Palette %d: ", pal + 4))
      for i = 0, 3 do
        local val = emu.read(16 + pal * 4 + i, emu.memType.nesPaletteRam)
        f:write(string.format("$%02X ", val))
      end
      f:write("\n")
    end

    f:close()
    emu.stop(0)
  end
end, emu.eventType.endFrame)

Template 4: Dump Nametable Region

-- dump_nametable.lua — Read a rectangular region of the nametable
local NT_BASE = 0       -- 0 for nametable 0, 0x400 for nametable 1
local START_ROW = 0
local END_ROW = 29      -- 30 rows total
local START_COL = 0
local END_COL = 31      -- 32 columns total

local f = io.open("OUTPUT_PATH", "w")

local frameCount = 0
emu.addEventCallback(function()
  frameCount = frameCount + 1
  if frameCount == 120 then
    f:write("=== Nametable ===\n")
    for row = START_ROW, END_ROW do
      f:write(string.format("Row %2d: ", row))
      for col = START_COL, END_COL do
        local addr = NT_BASE + row * 32 + col
        local tile = emu.read(addr, emu.memType.nesNametableRam)
        if tile == 0 then
          f:write(".. ")
        else
          f:write(string.format("%02X ", tile))
        end
      end
      f:write("\n")
    end

    f:close()
    emu.stop(0)
  end
end, emu.eventType.endFrame)

Template 5: Capture Screenshot as Raw RGB

-- capture_screen.lua
local FRAMES = 120
local RAW_FILE = "OUTPUT_PATH.raw"

local frameCount = 0
emu.addEventCallback(function()
  frameCount = frameCount + 1
  if frameCount == FRAMES then
    local pixels = emu.getScreenBuffer()
    local sf = io.open(RAW_FILE, "wb")
    for i = 1, #pixels do
      local p = pixels[i]
      sf:write(string.char((p >> 16) & 0xFF, (p >> 8) & 0xFF, p & 0xFF))
    end
    sf:close()
    emu.stop(0)
  end
end, emu.eventType.endFrame)

Common Debugging Workflows

"The ROM doesn't display correctly"

1. Build the sample and run it in Mesen:

   cd samples/<name> && dotnet build
   

2. Write a Lua script that waits 120 frames, then dumps:
   • Palette RAM (are the colors correct?)
   • Nametable (are tiles where expected?)
   • Zero page (did NES lib vars initialize?)
   • Screen buffer (what does it actually look like?)
3. Compare palette values against what the C# code specifies via pal_col()
4. Compare nametable tiles against what vram_write() should have written

"Compare runtime behavior of two ROMs"

Run the same Lua script against both ROMs and diff the output:

# Run against cc65 reference ROM
Start-Process $mesen ... reference.nes  # writes to ref_state.txt
# Run against dotnes ROM
Start-Process $mesen ... dotnes.nes     # writes to dotnes_state.txt
# Compare
Compare-Object (Get-Content ref_state.txt) (Get-Content dotnes_state.txt)

"Check if a specific memory address has the expected value"

Write a targeted Lua script:

local frameCount = 0
emu.addEventCallback(function()
  frameCount = frameCount + 1
  if frameCount == 120 then
    local val = emu.read(0x0325, emu.memType.nesInternalRam)
    if val == expected then
      emu.stop(0)  -- pass
    else
      emu.stop(1)  -- fail
    end
  end
end, emu.eventType.endFrame)

Then check $proc.ExitCode — 0 means the value matched.

Tips

• Use forward slashes in Lua file paths — Windows Lua handles them fine and avoids double-backslash escaping headaches.
• Use absolute paths for output files. The test runner's working directory may differ from where you launched it. Resolve paths in PowerShell first, then embed them in the Lua script.
• PowerShell string interpolation: Use @"..."@ (expandable here-string) to embed paths into Lua scripts. Replace \ with / first. Note: The closing "@ must be at column 1 (no leading whitespace):

  $outPath = ($PWD.Path) -replace '\\','/'
  $luaScript = @"
  local f = io.open("$outPath/output.txt", "w")
  "@
  

• Frame count: 60 frames ≈ 1 second of NES time. Most samples finish initialization within 60-120 frames.
• Exit code -1 means timeout — your script didn't call emu.stop() within the --timeout period. Check for Lua errors by wrapping code in pcall() and writing error messages to a file.
• Palette color 0x0F is black in the NES palette. If all non-background palette entries are 0x0F, the sample likely didn't call pal_col().
• Screen buffer is always 256×240 (standard NES resolution) regardless of overscan settings.