isaac-sim-person-simulation

name: isaac-sim-person-simulation description: Reference guide for simulating human characters in Isaac Sim 5.1+ (pip install, conda env) using omni.anim.people + IRA (isaacsim.replicator.agent). Use when writing scripts to spawn characters, drive movement via GoTo/Idle commands, inject commands at runtime, or configure navigation. All API details are verified against extscache source code.

Isaac Sim Person Simulation

Environment (pip install, not standalone)

Critical: python.sh does NOT exist in pip installs. Launch with conda activate isaaclab && python script.py. Extension source paths depend on the install method — for pip installs, all ext source lives under extscache/ inside the site-packages.

Verified extension versions (pip install)

Official Documentation

Key Source Files (for debugging / reading internals)

All paths are relative to the corresponding extscache directory.

Architecture: How a Character Moves (verified)

World.step(render=True)
  └─ omni.kit.scripting tick
       └─ CharacterBehavior.on_update(current_time, delta_time)   [character_behavior.py:537]
            │
            ├─ [frame 1, self.character is None]
            │     init_character()                                  [line 544]
            │       → ag.get_character(prim_path)
            │       → load commands from file into self.commands
            │       → append GoTo(spawn_pos) if number_of_loop > 0  (closes the loop)
            │     register_to_agent_manager()                       [line 548, sibling call — NOT inside init_character]
            │       → carb.eventdispatcher.dispatch_event(AgentEvent.AgentRegistered, payload)
            │
            └─ [subsequent frames]
                  execute_command(self.commands, delta_time)        [line 553]
                    └─ GoTo.execute(dt) → walk(dt)                 [base_command.py:160]
                          character.set_variable("Action", "Walk")
                          character.set_variable("PathPoints", nav_mgr.get_path_points())
                          character.set_variable("Walk", actual_walk_speed)  # blended 0→1
                         [on destination reached]
                          character.set_variable("Action", "None")

Important: register_to_agent_manager() is called by on_update directly after init_character() returns True. It is NOT a nested call inside init_character().

AnimGraph Variables (low-level interface)

import omni.anim.graph.core as ag

# Only valid AFTER simulation starts (runtime)
character = ag.get_character(prim_path_str)

# Read/write world transform
pos = carb.Float3(0, 0, 0)
rot = carb.Float4(0, 0, 0, 1)
character.get_world_transform(pos, rot)
character.set_world_transform(pos, rot)   # use only for snap/rotate, NOT for locomotion

# Drive the AnimGraph state machine
character.set_variable("Action", "Walk")                         # enter walk state
character.set_variable("PathPoints", [carb.Float3(x, y, z)])    # AnimGraph moves the prim AND drives animation
character.set_variable("Walk", 1.0)                              # blend weight 0~1 (fade in/out)
character.set_variable("Action", "None")                         # stop

Setting PathPoints causes AnimGraph to simultaneously update the character's transform AND animation. Do NOT manually move the prim while AnimGraph is driving it. set_world_transform is only for rotation alignment or forced teleport.

Command System (high-level interface)

Command string format

"{character_prim_name} {CommandName} [params...]"

Built-in commands (verified from character_behavior.py:452-483 + official docs)

More commands exist via dynamic import (e.g., Talk, TalkWith).

inject_command via AgentManager

from isaacsim.replicator.agent.core.agent_manager import AgentManager

AgentManager.get_instance().inject_command(
    agent_name="Character",          # must match character prim name under /World/Characters
    command_list=["Character GoTo 3.0 0.0 0.0 _"],
    force_inject=False,              # True → interrupts current command first
    instant=True,                    # True → insert at queue[1]; False → append to end
    on_finished=("cb_id", lambda cb_id, agent_name: print(f"{agent_name} arrived")),
)

on_finished mechanism (verified from character_behavior.py:349-352):

• A sentinel COMMAND_CALLBCAK_CHECKPOINT (note: typo in source — "CALLBCAK") is appended to the injected command array.
• When instant=True, the array is inserted at index 1 in the queue (current command finishes first).
• The sentinel fires the callback when the injected commands complete.
• Chain the next command inside the callback to build sequential or looping routes.

AgentRegistered Event

import carb.eventdispatcher
from omni.anim.people.settings import AgentEvent
# AgentEvent.AgentRegistered == "omni.anim.people/REGISTER_AGENT"

def on_agent_ready(event):
    # Access payload as attribute, then dict key (verified from agent_manager.py:175-181)
    payload = event.payload
    agent_name = payload["agent_name"]
    prim_path  = payload["prim_path"]
    # Safe to call inject_command or ag.get_character() from here

sub = carb.eventdispatcher.get_eventdispatcher().observe_event(
    event_name=AgentEvent.AgentRegistered,
    on_event=on_agent_ready,
    observer_name="my_unique_observer_name",
)
# MUST wait for this event before calling inject_command or ag.get_character()

Navigation Settings

s = carb.settings.get_settings()

# Straight-line path (no navmesh)
s.set("/exts/omni.anim.people/navigation_settings/navmesh_enabled", False)

# Obstacle-avoiding path (uses navmesh.query_shortest_path, agent_radius=0.5)
s.set("/exts/omni.anim.people/navigation_settings/navmesh_enabled", True)

# Multi-character velocity-prediction avoidance (NavigationManager.update_path)
s.set("/exts/omni.anim.people/navigation_settings/dynamic_avoidance_enabled", True)

Static Command File Loop Mode

s = carb.settings.get_settings()
s.set("/exts/omni.anim.people/command_settings/command_file_path", "/abs/path/to/commands.txt")
s.set("/exts/omni.anim.people/command_settings/number_of_loop", "inf")
# "inf" is parsed as math.inf in character_behavior.py:94-95
# init_character() automatically appends GoTo(spawn_pos) at the end to close the loop

Command file format (one command per line, first token = prim name):

Character GoTo 3.0 0.0 0.0 _
Character Idle 5
Character LookAround 3
Character GoTo -3.0 0.0 0.0 90
Character Sit /World/Chair 5
# lines starting with # are comments

Character Spawn Standard Flow

from isaacsim.replicator.agent.core.stage_util import CharacterUtil

# Step 1 — Load Biped_Setup.usd template (creates /World/Characters prim if needed)
CharacterUtil.load_default_biped_to_stage()

# Step 2 — Load character USD into stage
char_prim = CharacterUtil.load_character_usd_to_stage(
    character_usd_path="/path/to/F_Business_02.usd",
    spawn_location=(0.0, 0.0, 0.0),
    spawn_rotation=0.0,                  # degrees, Z-axis
    character_stage_name="Character",    # prim name = token used in command strings
)

# Step 3 — Get SkelRoot list (behavior scripts attach here, not on Xform root)
skelroot_list = CharacterUtil.get_characters_in_stage()

# Step 4 — Attach BehaviorScript
CharacterUtil.setup_python_scripts_to_character(skelroot_list, behavior_script_path)

# Step 5 — Attach AnimationGraph (copy from Biped_Setup)
anim_graph = CharacterUtil.get_anim_graph_from_character(
    CharacterUtil.get_default_biped_character()
)
CharacterUtil.setup_animation_graph_to_character(skelroot_list, anim_graph)

AnimGraphSchema plugin registration (required in script mode, pip install):

import os
from pxr import Plug
import omni.kit.app

mgr = omni.kit.app.get_app().get_extension_manager()
ext_path = mgr.get_extension_path_by_module("omni.anim.graph.schema")
Plug.Registry().RegisterPlugins(
    os.path.join(ext_path, "plugins", "AnimGraphSchema", "resources")
)
# Without this, AnimationGraph schema is not found in script mode

All carb.settings Key Paths (verified)

omni.anim.people (settings.py)

isaacsim.replicator.agent (settings.py)

isaacsim.replicator.agent — camera placement (persistent, from official docs)

Minimal Working Script Template

from isaacsim import SimulationApp
simulation_app = SimulationApp({"headless": False})

from isaacsim.core.utils.extensions import enable_extension
enable_extension("isaacsim.replicator.agent.core")
enable_extension("omni.anim.people")

import carb
import carb.eventdispatcher
from omni.isaac.core import World
from omni.anim.people.settings import AgentEvent
from isaacsim.replicator.agent.core.agent_manager import AgentManager
from isaacsim.replicator.agent.core.stage_util import CharacterUtil

world = World(physics_dt=0.005, rendering_dt=0.02)
world.scene.add_default_ground_plane()

# Register AnimGraphSchema plugin
import os
from pxr import Plug
import omni.kit.app
mgr = omni.kit.app.get_app().get_extension_manager()
ext_path = mgr.get_extension_path_by_module("omni.anim.graph.schema")
Plug.Registry().RegisterPlugins(
    os.path.join(ext_path, "plugins", "AnimGraphSchema", "resources")
)

# Configure navigation
s = carb.settings.get_settings()
s.set("/exts/omni.anim.people/navigation_settings/navmesh_enabled", False)
s.set("/exts/omni.anim.people/navigation_settings/dynamic_avoidance_enabled", False)

# Spawn character
CharacterUtil.load_default_biped_to_stage()
char_prim = CharacterUtil.load_character_usd_to_stage(
    character_usd_path="assets/characters/F_Business_02.usd",
    spawn_location=(0.0, 0.0, 0.0),
    spawn_rotation=0.0,
    character_stage_name="Character",
)
skelroot_list = CharacterUtil.get_characters_in_stage()
behavior_script_path = mgr.get_extension_path_by_module("omni.anim.people") \
    + "/omni/anim/people/scripts/character_behavior.py"
CharacterUtil.setup_python_scripts_to_character(skelroot_list, behavior_script_path)
anim_graph = CharacterUtil.get_anim_graph_from_character(CharacterUtil.get_default_biped_character())
CharacterUtil.setup_animation_graph_to_character(skelroot_list, anim_graph)

# Wait for AgentRegistered, then inject command
def on_agent_ready(event):
    name = event.payload["agent_name"]
    AgentManager.get_instance().inject_command(
        agent_name=name,
        command_list=[f"{name} GoTo 3.0 0.0 0.0 _"],
        instant=True,
    )

sub = carb.eventdispatcher.get_eventdispatcher().observe_event(
    event_name=AgentEvent.AgentRegistered,
    on_event=on_agent_ready,
    observer_name="my_agent_ready_observer",
)

world.reset()
while simulation_app.is_running():
    world.step(render=True)

simulation_app.close()

CommandResponse — Runtime Event Triggers (from official docs)

Characters can be configured to react to runtime events (time, carb events, or physical incidents) using CommandResponse. This is configured in the IRA YAML config or injected programmatically.

# In IRA YAML config under `response:` section
CommandResponse:
  name: "AlertResponse"
  priority: 1                        # higher = takes precedence when multiple fire simultaneously
  pick_agent: nearest                # all | first_available | nearest | furthest
  resume: true                       # resume prior task after response completes
  position: [0.0, 0.0, 0.0]         # optional: reference position for nearest/furthest selection
  trigger:
    type: time                       # time | carb_event | incident
    time: 5.0                        # seconds into simulation
  commands:
    - "Character GoTo 0.0 0.0 0.0 _"
    - "Character LookAround 5"

Trigger types:

IRA YAML Configuration (full SDG pipeline)

When running IRA as a full SDG pipeline (not just character simulation), configure via a YAML file.

isaacsim.replicator.agent:
  version: 0.7.1          # must match extension minor version (0.7.x)

  global:
    seed: 42
    simulation_length: 300  # frames at 30 FPS → 300 = 10 seconds

  scene:
    asset_path: /path/to/environment.usd

  sensor:
    num: 4                  # number of cameras; reduce if VRAM errors occur

  character:
    num: 5
    asset_path: /path/to/characters/
    command_file: /path/to/commands.txt
    spawn_area: NavMeshAreaName
    navigation_area: NavMeshAreaName

  robot:
    nova_carter_num: 1
    iw_hub_num: 0
    write_data: true         # capture from robot cameras

  replicator:
    writer: IRABasicWriter
    output_path: /path/to/output/

pip install note: tools/actor_sdg/sdg_scheduler.py does NOT exist in pip installs (standalone only). Drive the pipeline from your own Python script using SimulationApp + IRA APIs directly.

Writer options:

Common Pitfalls