Evaluation Assets and Result Disclosure

This page defines how existing test assets support open-source evaluation. It does not turn legacy observations into performance of the current main branch.

As of 2026-08-26, historical test archives remain outside this repository and are not distributed with source code. The public repository now includes owner-approved derived tables and figures from selected CSV files, plus the reproducible analysis script. Every public claim must remain traceable to a data version, code commit, configuration, and analysis script.

Scope of Current Claims

• The current software path uses body_rh, the multi-IMU common timeline, JSONL v2, WebSocket, ROS 2, and Unity adapters. Legacy records can use different protocols, device counts, coordinate conventions, calibration procedures, and robot control paths.

• Legacy endpoint reconstruction without external motion capture may establish internal stability, repeatability, or trajectory consistency only. It cannot establish absolute position accuracy against an external ground truth.

• Legacy robot joint and teleoperation logs apply to their recorded robot, controller, posture, payload, and network conditions. They are not a safety guarantee or a general robot-performance claim.

• Device clock synchronization has not yet been validated on hardware. Historic receive-time logs must not be used as evidence of cross-IMU alignment error.

Primary Node and Tracker Evidence

The figures below are generated by analysis/scripts/generate_historical_figures.py. It reads an external owner-provided archive, writes English-header tables to analysis/metrics/, and exports 300 DPI PNG/PDF figures. Recreate them from the repository root:

python3 -m pip install -r analysis/requirements.txt
python3 analysis/scripts/generate_historical_figures.py \
  --data-root /path/to/IMU测试数据汇总

No raw CSV, raw human-motion record, LAN address, device identifier, or private robot configuration is included here.

Archived Battery Discharge

One archived battery profile contains 42 valid voltage bins. Its largest recorded remaining-runtime value is 566.5 min (9.44 h), over valid recorded bin midpoints from 3.21 V to 4.03 V. The source-file summary rounds the duration to 567 min. Load and configuration metadata are incomplete, so this is an archived observation rather than a runtime specification.

Archived Static Attitude Change

The static recording contributes 14,999 samples across 299.960 s after its initialization sample. The figure uses the median of the first 5 s as baseline and one-second means for display; the final 5 s median changes are -0.002917 deg roll, 0.158715 deg pitch, and 0.358060 deg yaw. This is an Euler-angle stability observation, not external attitude accuracy or multi-IMU sync error.

Historical Asset Inventory

Category	Coverage	Historical range	Eligible documentation use	Current public status
Node output	Acceleration, gyroscope, VQF quaternion, diagnostics	2026-03	Output format and sampling diagnostics	Raw records withheld
Magnetometer calibration	Hard-iron offsets and soft-iron matrices	2026-03	Existence of calibration workflow	Device-specific values withheld
Pose calibration and motion recording	Three/four-pose records, mounting alignment, role maps	2026-03 to 05	Calibration and recording scope	Raw human-motion records withheld
Drift	Static VQF quaternion, Euler angles, packet rate	2026-05	Static-attitude method and selected derived figure	Raw records withheld; selected derived outputs published
Endpoint reconstruction	Static noise, length constraint, repeated contact, circle trace	2026-03	Internal-consistency evaluation without external motion capture	Legacy report remains internal
Whole-robot joints	Static poses and dynamic command/feedback comparison	2026-03 to 04	Supplementary integration material	Raw records withheld; not a current baseline
Teleoperation integration	Command, feedback, error, velocity, torque, network gaps	2026-04 to 06	Log fields and anomaly-analysis scope	Raw robot logs withheld
Simulation to hardware	PD optimization, simulation traces, partial hardware metrics	2026-04 to 06	Supplementary controller-tuning material	Raw records/configuration withheld
Battery life	Voltage bins and discharge logs	2026-06	Battery-test method and selected derived figure	Raw records withheld; derived profile published

These assets are candidates for future papers, technical reports, and release evaluation material. They are not a dataset bundled with this repository.

Metric Definitions

Test category	Minimum reported metrics	Units	Required context
Static pose	Per-axis standard deviation, 3D RMS, duration	deg, deg/s, or mm	State the pose, stillness rule, and filter configuration.
Drift	Accumulated drift, drift rate, valid-packet rate	deg, deg/min, %	Record start/end times, stillness threshold, and loss definition.
Multi-IMU sync	Sync error, RTT, valid-sample ratio, sample age	ms, %	Map device sample time to frame_time_ns; do not substitute UDP arrival time.
Endpoint reconstruction	Repeatability RMS, planar residual, external-reference error	mm	Report absolute error only when an external reference exists.
Joint tracking	MAE, RMSE, maximum error, delay or phase difference	deg, ms	Specify joint, trajectory, controller frequency, feedback source, and limits.
Teleoperation	Command-feedback error, network gaps, disconnect/anomaly counts	deg, ms, count	Document anomaly rules; do not silently remove excluded frames.
Simulation to hardware	Per-domain RMSE/MAE and configuration deltas	deg, mm, or task unit	Report the baseline, not only the best optimized trial.
Battery	Usable duration, voltage range, shutdown threshold, load	min, V	State battery batch, wireless state, rate, LEDs, and load.

Samples, Failures, and Versions

Every future public experiment directory must contain an evaluation_manifest.json that conforms to evaluation_manifest.schema.json. It records at least:

• experiment_id, acquisition date, software commit or release, firmware version, hardware revision, and calibration version;

• total, valid, and excluded sample counts with reason for every exclusion;

• each metric’s name, statistic, unit, value, and scope;

• relative paths and release scopes for raw data, derived data, configuration, scripts, figures, and reports;

• whether records contain personal motion, device MAC addresses, LAN addresses, robot safety parameters, or restricted third-party material.

Invalid frames, sync failures, calibration failures, disconnects, and out-of-range trajectories must all retain a count. Any statistical exclusion must state its rule and impact. Selecting only a best run is not permitted.

The repository includes a dependency-free baseline validator. Before a release, run:

python tools/validate_evaluation_manifest.py evaluation/<experiment-id>/evaluation_manifest.json

It checks required fields, release scope, sample-count conservation, metric fields, and artifact-relative paths. JSON Schema provides the complete field constraints.

Future Public Data Packages

Only experiments explicitly approved by the data owner may appear in a release. Each package should have this layout:

evaluation/<experiment-id>/
  evaluation_manifest.json
  README.md
  raw/                 # Authorized and sanitized minimum raw records
  derived/             # Metric inputs and derived outputs
  config/              # Firmware, calibration, and analysis snapshots
  scripts/             # Reproducible analysis scripts
  figures/             # Public figures generated from scripts
  SHA256SUMS

Packages must exclude Wi-Fi credentials, MAC addresses, personal identifiers, LAN addresses, unsanitized video, private robot configuration, and unauthorized third-party material. When raw data cannot be released, provide a sanitized minimum example, metric definitions, sample statistics, failure handling, configuration summary, and a runnable analysis entry point.

Pending Validation

• The new body_rh motion-capture path needs new on-hardware synchronization, wear, calibration, static, dynamic, and long-duration drift tests.

• Authorization and sanitization must be decided separately for additional historical plots, values, raw records, robot logs, and photos.

• Absolute endpoint-position accuracy requires a documented external reference system or known geometric reference.

Supplementary Archived Robot Integration

These figures record one historical robot-control integration path. They are kept separate from the Body Tracker evidence because their results depend on a specific robot, controller, payload, network, and legacy calibration path.

Historical Robot Joint Tracking

The archived default_pd teleoperation path contains four right-arm joints, each with 6,598 samples over 213.425 s. RMSE and MAE are command-feedback errors, not body-tracker pose accuracy or a safety guarantee.

Simulation PD Optimization

The archived simulation file has 14 joint groups for each trajectory and 16 iterations. Each curve is the median normalized best_so_far cost across joint groups; the shaded band is its 25th to 75th percentile range. This is controller-tuning evidence only.

Chinese version: docs/zh/evaluation.md.