Quick Evaluation
RoboDojo evaluation is launched through XPolicyLab policies. The public entry point is:
Before running evaluation, finish Install & Download and choose an environment config from Configurations. The default config is arx_x5, which loads env_cfg/arx_x5.yml.
Smoke test
Section titled “Smoke test”Use smoke first when validating a new install, policy integration, or script change. It runs selected tasks sequentially with EVAL_NUM=1 by default and passes only when each selected task exits successfully, writes _result.json, and records eval_time >= 1.
Run a subset while debugging:
Resume from an existing smoke summary:
Single-task eval
Section titled “Single-task eval”For same-machine evaluation, use eval. It starts the policy server and Isaac Sim client together on localhost:
Required policy arguments:
| Argument | Meaning |
|---|---|
--policy-dir | Directory containing the XPolicyLab policy integration, including eval.sh and deploy.yml. |
--ckpt | Checkpoint name passed to the policy-side eval/server script. |
--policy-env | Conda env, uv, or env path used by the policy server. |
Common benchmark arguments:
| Argument | Default |
|---|---|
--env-cfg |
arx_x5 |
--seed |
0 |
--eval-env |
RoboDojo |
--action-type |
ee |
--expert-num |
100 |
Use --dry-run to inspect the resolved command without launching the policy server or Isaac Sim:
Complete evaluation
Section titled “Complete evaluation”Publication-style evaluation uses 54 runnable simulation tasks, three eval seeds, and native per-task episode counts from task/RoboDojo/config/_task.yml.
Run one seed with native counts:
Repeat for --seed 1 and --seed 2, or launch the three seeds on separate GPUs.
After all seeds finish, aggregate results:
If results are outside the default tree, set ROBODOJO_EVAL_ROOT:
Benchmark sweep
Section titled “Benchmark sweep”Use benchmark with a fixed --eval-num for local scored sweeps:
Use --eval-num native when _task.yml should control each task’s 25/50 episode count. That mode is the correct entry point for full benchmark sweeps.
Benchmark protocol
Section titled “Benchmark protocol”task_name connects the task implementation, config, layout, and result path:
Most tasks use lowercase snake_case. Four tasks keep uppercase letters to match asset and layout names: play_Xylophone, swap_T, push_T, and push_T_random.
RoboDojo evaluates 42 base simulation tasks across five capability dimensions. Twelve Generalization tasks also have a _random variant, for 54 runnable tasks total.
| Dimension | Base tasks | Generalization _random variants |
|---|---|---|
| Generalization | 12 | 12 |
| Memory | 6 | 0 |
| Precision | 8 | 0 |
| Long-Horizon | 8 | 0 |
| Open | 8 | 0 |
| Total | 42 | 12 |
Generalization base tasks:
stack_bowlspush_Tpack_objects_into_boxfold_clotheshang_mugssweep_blockspour_liquid_into_cupmake_toastarrange_largest_numbersort_nesting_dolls_by_sizestore_laptop_and_headphonesstack_blocks
Each has a sibling task with the _random suffix, for example push_T_random.
Episode counts:
| Task type | Trials per seed |
|---|---|
| Standalone tasks, including non-Generalization base tasks | 50 |
Generalization _random tasks when run alone | 50 |
| Generalization base tasks in full benchmark aggregation | 25 normal + 25 randomized |
Layout sets correspond to eval seeds:
Across three seeds, most tasks contain 150 evaluation trials. Generalization base tasks combine 25 normal trials and 25 randomized trials per seed before aggregation.
The summary script reads the latest _result.json per task, policy, and seed; merges Generalization base tasks with their _random siblings; and writes eval_result/RoboDojo/_summary.md. RoboDojo reports mean success rate and mean score for each capability dimension, together with standard deviations across seeds and trials. Overall success rate and overall score are computed by averaging the corresponding metrics over all five dimensions.
The default eval config is env_cfg/arx_x5.yml. Its config_name field controls both layout lookup and result paths. Keep the env_cfg filename and config_name aligned unless you are intentionally migrating layouts.
Split eval
Section titled “Split eval”Use server and client when the policy server and Isaac Sim client run on separate machines, separate processes, or across a host/container boundary. eval, smoke, and benchmark co-locate the server and client on localhost; use split mode for remote policies or Docker simulator clients.
Start the policy server:
Start the simulator client:
client requires --task, --policy-host, --policy-port, and either --policy-dir or --policy-name. Keep --task, --env-cfg, --seed, and --action-type aligned between server and client. On the client side, --ckpt and --action-type also label result paths.
Batch inference mode is declared in XPolicyLab/policy/<POLICY_NAME>/deploy.yml with eval_batch. Do not pass a separate batch flag on client. If eval_batch: false, RoboDojo forces num_envs=1 even when the sim config requests a larger value.
Docker client
Section titled “Docker client”The Docker image is simulator-side only. Start the policy server outside the container, then run robodojo.sh client inside the container. See Optional: Install with Docker for the full mount and networking command.
| Policy server | Container networking | --policy-host |
|---|---|---|
| Same host | --network host | 127.0.0.1 |
| Same host, bridge | --add-host=host.docker.internal:host-gateway | host.docker.internal |
| Remote machine | routed network | policy machine IP |
Inside a bridge container, localhost is the container itself, not the host.
Output
Section titled “Output”Per-task eval artifacts:
Important files:
| File | Meaning |
|---|---|
_result.json | Success count, failure count, score, eval_time, and per-episode details. |
episode_*.mp4 | Saved camera videos. |
_stream/ | Temporary stream files for in-progress videos. |
_resume_<run_id>.json | Resume manifest written one directory above the run folder. |
Smoke summaries:
Aggregated benchmark table:
Lightweight checks
Section titled “Lightweight checks”For quick repository and command checks without launching Isaac Sim:
For runtime acceptance on a machine with Isaac Sim and the selected policy ready: