Skip to content
RoboDojo RoboDojo Wiki

Evaluation Issues

This page covers problems that happen after RoboDojo is installed and evaluation has started. For setup, driver, Docker, or asset problems, see Installation Issues.

Start with doctor

Section titled “Start with doctor”

Before debugging a long evaluation log, run the built-in checks:

bash scripts/robodojo.sh doctor

For lightweight repository checks that skip Isaac Sim, Conda, and policy validation:

bash scripts/robodojo.sh doctor --skip-isaac --skip-conda --skip-policy

If you are validating a policy integration, include the policy arguments but skip Isaac Sim while checking policy-side paths and metadata:

bash scripts/robodojo.sh doctor \
  --policy-dir XPolicyLab/policy/<POLICY_NAME> \
  --ckpt <CKPT_NAME> \
  --policy-env <POLICY_ENV> \
  --skip-isaac

Policy directory is missing required files

Section titled “Policy directory is missing required files”

Symptoms: robodojo.sh eval, server, or smoke fails before the simulator starts, or reports that the policy directory is invalid.

Cause: RoboDojo expects policy integrations to live in XPolicyLab and provide the policy-side launcher/config files:

XPolicyLab/policy/<POLICY_NAME>/eval.sh
XPolicyLab/policy/<POLICY_NAME>/deploy.yml

What to try:

  • Check that --policy-dir points to the policy directory, not the XPolicyLab root.
  • Run doctor with --policy-dir, --ckpt, and --policy-env.
  • If policy dependencies, checkpoints, processors, or model paths fail, follow the documentation inside that XPolicyLab policy directory.

GPU out of memory during evaluation

Section titled “GPU out of memory during evaluation”

Symptoms: Evaluation fails with a CUDA out-of-memory error, or Isaac Sim crashes during evaluation.

Cause: Too many environments are evaluated in parallel. Each environment needs GPU memory for rendering, simulation buffers, cameras, and loaded assets.

What to try: Reduce scene.num_envs in the simulation config selected by the top-level env_cfg, such as env_cfg/sim/sim_config.yml. A smaller value uses less VRAM and is usually the most reliable fix, although it may also reduce evaluation throughput.

CPU out of memory during evaluation

Section titled “CPU out of memory during evaluation”

Symptoms: The process is killed by the operating system, reports an out-of-memory error, or becomes unresponsive while loading scenes or starting evaluation.

Cause: num_envs is set too high. Each parallel environment imports its own scene objects, so high parallelism can quickly increase CPU memory usage, especially for object-heavy or randomized tasks.

What to try: Reduce scene.num_envs in the simulation config selected by the top-level env_cfg, such as env_cfg/sim/sim_config.yml. For most tasks, start from a small value and increase it only after the scene loads reliably. For cluttered tabletop tasks, keep the value at or below the task-defined maximum.

PhysX articulation instability

Section titled “PhysX articulation instability”

Symptoms: Simulation or evaluation stops unexpectedly, reports PhysX / CUDA errors, or one environment becomes unstable when the scene contains articulated objects.

Example messages:

Invalid PhysX transform detected for /World/envs/env_...
GPU ... fail to launch kernel!!
PhysX has reported too many errors, simulation has been stopped.
PhysX Internal CUDA error. Simulation cannot continue! Error code 700!

Cause: This is usually caused by instability in the underlying Isaac Sim / PhysX simulation. When a scene contains articulation bodies, simulation can become less stable and may occasionally trigger invalid transform or PhysX CUDA errors.

What to try: Restart the simulation process and rerun the affected task. If the issue appears frequently, reduce num_envs to lower simulation pressure.

RoboDojo has recovery paths for some fatal simulator failures. scripts/eval_policy.sh retries exit codes 99, 134, and 139; src/eval_client/main.py writes resume manifests; and src/eval_client/physx_warning_monitor.py watches for fatal PhysX messages. While debugging, limit retry count:

ROBODOJO_MAX_BASH_RETRIES=3 bash scripts/robodojo.sh eval \
  --policy-dir XPolicyLab/policy/<POLICY_NAME> \
  --task make_toast \
  --ckpt <CKPT_NAME> \
  --policy-env <POLICY_ENV> \
  --eval-num 1

Unstable evaluation samples

Section titled “Unstable evaluation samples”

Symptoms: The eval log reports unstable environments or unstable samples, for example:

Unstable envs: [...]
All scene Unstable Error!
Success nums: ..., Fail nums: ..., Unstable nums: ...

Some episodes may not produce fail videos and are not counted in the final evaluation total.

Cause: RoboDojo marks a sample as unstable when the scene is not a valid evaluation sample. This can happen in two common cases:

  • Layout instability: Because simulation has small run-to-run variation across hardware, drivers, and timing, a placed object may occasionally fail the layout stability check and not settle correctly.
  • Interactive support-arm instability: In interactive tasks that use a support Franka arm, random simulation variation can cause the support arm to miss or fail its scripted interaction. For example, the support arm may fail to place an object in the intended location or fail to complete the opponent action. These samples are also marked as unstable.

What to try:

  • Rerun the evaluation. Occasional unstable samples are expected and are skipped instead of being counted as policy failures.
  • Reduce scene.num_envs if unstable samples appear frequently, especially on object-heavy or interactive tasks.
  • Check whether the issue is task-specific. If it mainly appears in interactive tasks, inspect the support-arm setup and the corresponding scene layout.

Saved video is a static first frame under --headless

Section titled “Saved video is a static first frame under --headless”

Symptoms: Recorded videos show only the first frame; the scene never updates in the output file.

Cause: Isaac Lab is missing or incorrectly installed, so headless rendering does not step correctly.

What to try:

  1. Update the Isaac Lab submodule under third_party/IsaacLab.

  2. Reinstall from the Isaac Lab step:

    bash scripts/install.sh --from isaaclab

  3. Confirm the environment is active and retry collection or evaluation with --headless.

eval_batch: false forces num_envs to 1

Section titled “eval_batch: false forces num_envs to 1”

Symptoms: env_cfg/sim/sim_config.yml sets scene.num_envs to a value larger than 1, but the eval log prints a message like:

[main] eval_batch=false in XPolicyLab/policy/<POLICY_NAME>/deploy.yml; forcing num_envs N -> 1

Cause: RoboDojo reads eval_batch from XPolicyLab/policy/<POLICY_NAME>/deploy.yml. If eval_batch is false, the eval client disables batched policy inference and forces the final simulator parallel count to 1, even if the sim config requested more environments.

What to try:

  • Keep scene.num_envs: 1 for policies whose deploy.yml has eval_batch: false.
  • Set eval_batch: true only when the policy server and deploy.py adapter support batched observations and batched actions.
  • If a random task also caps num_envs, that cap is applied first; the eval_batch: false check is applied afterward and still forces the final value to 1.

zenity: cannot open display (Gtk-WARNING)

Section titled “zenity: cannot open display (Gtk-WARNING)”

Symptoms: The process hangs during camera warmup, often when running with many parallel environments.

Example message:

(zenity:xxxxxx): Gtk-WARNING **: xx:xx:xx.xxx: cannot open display

Cause: The exact trigger is not fully characterized. This issue is usually observed in headless or high-parallelism runs, especially when rendering or camera warmup stalls, or when the simulator waits too long for a policy action.

What to try:

  • Restart the job. This can sometimes clear a stuck camera warmup.
  • Reduce the parallel count by lowering num_envs or the batch size. High parallelism often triggers this issue.
  • Wait for a while before stopping the run. In some cases, the process eventually exits with a more informative error message.

Missing _result.json

Section titled “Missing _result.json”

Symptoms: Smoke or benchmark marks a task as failed because no _result.json was written.

Cause: The task did not finish a counted episode. Common causes include missing layout files, policy server startup failure, Isaac Sim crashing before result writing, or repeated unstable samples.

What to try:

  1. Inspect the loop summary:

    smoke_results/<run_id>.md
smoke_results/<run_id>/logs/<task>.log

  2. Inspect the per-task result directory:

    eval_result/RoboDojo/<task>/<policy>/<env_cfg>/<seed>_<additional_info>/

  3. Rerun the same task with one episode:

    bash scripts/robodojo.sh eval \
  --policy-dir XPolicyLab/policy/<POLICY_NAME> \
  --task <TASK_NAME> \
  --ckpt <CKPT_NAME> \
  --policy-env <POLICY_ENV> \
  --eval-num 1

  4. If the log points to missing assets or layouts, switch to Installation Issues.

Split eval client cannot reach server

Section titled “Split eval client cannot reach server”

Symptoms: robodojo.sh client warns that the policy server is unreachable, hangs while retrying, or fails to connect over WebSocket.

Cause: The policy server is not running, is bound only to localhost, a firewall blocks the port, or the client is using the wrong host address. In Docker bridge networking, localhost points to the container itself.

What to try:

  • Start the server with --bind-host 0.0.0.0 when another machine or container needs to connect.

  • Use the policy machine’s reachable IP for --policy-host.

  • Test from the simulator machine:

    nc -vz <POLICY_MACHINE_IP> 9999

  • For Docker, use --network host with --policy-host 127.0.0.1, or use host.docker.internal / a routable IP according to your container network mode.

  • Make sure protocol: ws is set in the policy deploy.yml.

Dry runs

Section titled “Dry runs”

Dry runs verify command construction without launching heavy processes:

bash scripts/robodojo.sh eval \
  --policy-dir XPolicyLab/policy/<POLICY_NAME> \
  --task stack_bowls \
  --ckpt <CKPT_NAME> \
  --policy-env <POLICY_ENV> \
  --dry-run

bash scripts/robodojo.sh smoke \
  --policy-dir XPolicyLab/policy/<POLICY_NAME> \
  --ckpt <CKPT_NAME> \
  --policy-env <POLICY_ENV> \
  --only stack_bowls,push_T \
  --dry-run

Small random variation in evaluation scores

Section titled “Small random variation in evaluation scores”

Symptoms: Success rate or task metrics differ slightly between evaluation runs, even with the same policy, config, and seed settings.

Cause: Isaac Sim’s GPU physics and simulation pipeline are not fully deterministic. Minor numerical and timing differences can accumulate across episodes, producing small run-to-run variance.

What to try:

  • Run multiple evaluation passes (e.g. 3-5 runs or more episodes per run).
  • Report the mean score as the final result; include std dev or confidence intervals when comparing methods.

This variance is expected at the simulator level and does not necessarily indicate a bug in the policy or evaluation script.

Section titled “Related resources”