Outputs & Reporting

Each simulation run produces a suite of JSON, parquet, CSV, and PNG artefacts summarising the generated measurements, filter results, and diagnostics. The files live under the directory passed via --run-dir (e.g., outputs/lunar_static/), with one subdirectory per pipeline stage.

Simulation stage outputs (`simulate/`)

File	Contents
`measurements.json`	Structured measurement catalogue with `schema_version`, `metadata`, and a nested `measurements[]` array (epoch info, satellite/receiver states, geometry, link budget, grouped observables, SISE terms). Use `measurement_file_manipulation.load_measurement_catalogue` to flatten it for pandas workflows. When enabled for EDL scenarios, altimeter outputs are included as `altimeter_agl_m`, `altimeter_true_agl_m`, `altimeter_noise_std_m`, `altimeter_terrain_h_m`, `altimeter_vehicle_height_datum_m`, `altimeter_undulation_m`, `altimeter_valid`, and the flag columns `altimeter_flag_out_of_range`, `altimeter_flag_dropout`, `altimeter_flag_hold`, `altimeter_flag_latency_wait`.
`measurement_summary.csv`	Measurement quality stats per observable and satellite (counts, value mean/std/min/max, noise std summaries, C/N₀, elevation span) computed after outlier rejection.
`dop_timeseries.csv`	GDOP/PDOP/HDOP/HHDOP/VDOP and satellite counts per epoch, filtered with the same outlier mask.
`measurement_outliers.csv`	Per-satellite count of samples removed by the robust MAD filter (plus an `ALL` total).
`low_cn0_measurements.csv`	Samples discarded because their \(C/N_0\) fell below `measurement.receiver_rf.cn0_threshold_dbhz`, reported per satellite and in total.
`satellite_visibility_cdf.csv`	Table backing the cumulative visibility plot: for each integer threshold `n`, it lists the percentage of total expected epochs (including padded zero-satellite windows) with ≥ `n` visible satellites.

Figures

EDL runs with altimeter enabled add altimeter_agl_vs_time.png (measured vs true AGL) and altimeter_noise_vs_time.png (per-epoch sigma with valid epochs highlighted).

src/reporting.py renders several diagnostic plots (all saved under simulate/plots/) plus per-satellite CSVs tracking MAD-filtered and low-\(C/N_0\) removals:

dop_vs_time.png – DOP metrics vs epoch.
signal_to_noise_vs_time.png – Received C/N₀ per satellite across the observation window.
cn0_vs_elevation.png – Scatter plot coloured by satellite ID (outliers removed).
satellite_visibility_cdf.png – Step chart showing how the cumulative percentage of epochs drops as the visible-satellite requirement (≥ n) increases.
measurement_counts.png – Grouped bar chart showing measurement counts per type and satellite.
satellite_polar.png – Sky plot of satellite azimuth/elevation coloured by satellite ID.
two_way_link_schedule.png – Timeline showing which satellite carries the two-way link at each epoch (coloured by \(C/N_0\) when available) with shaded windows marking periods when two-way contacts are authorised.
user_clock_vs_time.png – Simulated receiver clock bias and drift plotted against time (per-epoch values from measurements.json).

In addition, ingest/plots/ contains 3D and XY orbit views, a ground-track overlay (ingest_ground_tracks.png) rendered on top of the Clementine mosaic at data/mosaic/PIA02992.tif, and an altitude/velocity timeline. The 3D orbit plot enforces equal scaling on the x/y/z axes for geometric consistency. estimate/plots/ contains residual and error plots described below. The ingest stage also writes ingest/altitude_velocity_stats.csv summarising the mean, standard deviation, minimum, and maximum altitude/velocity for every satellite across the ingest window.

Pass --disable-plots to any poetry run acons … command to skip PNG generation across all stages when you only need CSV/parquet artefacts; poetry run acons regression … applies this behavior by default, so use --enable-plots there only when you want diagnostics for debugging a failing run.

Estimation stage outputs (`estimate/`)

By default, all estimation artefacts are written to <run-dir>/estimate/. Supplying --output-subdir <name> stores them in <run-dir>/estimate/<name>/ (or under the sibling estimate directory next to an overridden --measurements-path), which keeps multiple filter runs organised.

File	Contents
`ekf_states.parquet`	Full EKF state history (position, velocity, clock bias/drift).
`ekf_states_downsampled.parquet`	Down-sampled state history for quick inspection.
`ekf_residuals.parquet`	Full measurement residual log (range and range-rate) plus an `observations_used` column counting how many updates were applied at each epoch (no outlier filtering applied).
`ekf_covariances.parquet`	Full covariance matrices for each processed epoch.
`state_errors.csv`	Truth-relative position/velocity/clock errors per epoch (all samples retained) plus `sat_count` for the number of visible satellites at each epoch and `seconds_since_start` (elapsed seconds from the first epoch).
`state_error_stats.csv`	Mean/std/RMS and quantiles (q50/q85/q90/q95) for per-axis position errors, `pos_error_radial_m` (vertical component), `pos_error_horizontal_m`, `pos_error_3d_m`, velocity components/3D speed error, and clock bias/drift (computed from the full error set).
`state_error_by_satellite_count.csv`	Horizontal, vertical, and 3D error mean/std/RMS plus q50/q85/q90/q95 grouped by the visible-satellite count.
`solution_coverage.csv`	Single-row summary comparing expected epochs (from the scenario time grid) to the EKF states actually produced, including the missing count and availability percentage.

The per-axis statistics exclude a warmup window at the start of the run so early transient behaviour does not inflate the RMS numbers. By default the first 7 200 s are removed before computing state_error_stats.csv and derivative tables. Override or disable the trimming per scenario using the new reporting block:

reporting:
  stats_warmup_seconds: 3600    # Set to 0 to include all epochs in the stats CSVs.

[... omitted 0 of 200 lines ...] During poetry run acons estimate … the logger also prints a Solution coverage line comparing the expected epoch count derived from the scenario time span (end − start divided by step_seconds) against the actual number of EKF states written. The percentage highlights how many epochs never produced a solution so you can quickly spot coverage gaps.

Key plots (found in estimate/plots/) include the revamped position_error_cdf.png (now three stacked CDFs for horizontal, vertical, and 3D errors with annotated 0.95/0.90/0.66/0.50/0.30 probability markers plus log-scaled axes), the position-error timeline (3D position error vs. hours with 1σ/3σ envelopes and satellite count), the vertical-plus-clock timeline (vertical error stacked with receiver clock bias), observations_per_epoch.png, a step plot that tallies how many measurements the EKF assimilated at each epoch while overlaying the visible-satellite count, measurements_per_epoch.png, which draws counts directly from measurements.json per observable type so you can compare scheduler/availability constraints with what the filter ultimately used, computed_height_vs_plot.png (DEM-sampled height along the estimated trajectory, sampled even when the DEM constraint is disabled), and a family of position_error_cdf_sat_XX.png files (XX = integer satellite count) that repeat the CDF analysis for epochs sharing the same number of visible satellites. Estimation plots now operate on the full datasets without the MAD-based outlier filter, while the measurement-availability plots respectively reflect actual EKF updates and the raw measurement catalogue. Two-way availability shading on every *vs_time.png plot now follows measurement.two_way_availability_minutes and measurement.two_way_availability_cadence_minutes, with the highlight automatically disappearing across gaps where no EKF solutions exist. When the DEM constraint is enabled, estimation plot titles append “(with DEM)”.

Log file

For EDL runs with the altimeter enabled, the reporting stage also emits altimeter_noise_vs_time.png (simulation-side sigma history) and altimeter_residual_vs_time.png (EKF residual with the 1σ innovation envelope).

Each stage writes a structured log (<stage>.log) under its run directory. These logs capture inputs, applied outlier masks, warnings (e.g., low C/N₀), and paths to generated artefacts. Adjust verbosity with the CLI --log-level flag; selecting TRACE prints representative range and range-rate observations with every error contribution (geometry, user clock, SISE, calibration bias, and DLL/FLL noise) to aid debugging.

Processing assumption snapshot

Every CLI invocation now (re)generates <run-dir>/processing_assumptions.md (e.g., outputs/lunar_static/processing_assumptions.md) straight from the scenario YAML. The summary mirrors the config in prose/bullets so non-technical stakeholders can skim the time span, frames, SPICE kernels, user geometry, RF settings, and EKF tunings without reading raw YAML. After each stage finishes, the prompt prints the absolute path to this summary so downstream users immediately know where to look.

Legacy include_sise_placeholder switches have been removed from the reporting configuration, so scenarios no longer need to carry that flag and the summary omits the corresponding entry.

Extending the reports

Reporting utilities live in src/reporting.py. To add custom artefacts:

Extend generate_simulation_outputs() or generate_estimation_outputs() with new exports/plots.
Leverage the measurement/residual/state DataFrames returned by the pipeline stages.
Consider whether the new diagnostic should apply the existing MAD-based outlier filter before rendering.

Refer to the source code for examples of writing CSV/JSON/parquet files and creating Matplotlib figures.

Cross-run analysis (`analysis/`)

Use the analysis CLI stage to compare the DOP timelines and EKF error statistics produced by multiple runs. Define the inputs in a dedicated YAML file (see configs/analysis/sample_lunar_comparison.yaml). The CLI accepts either full/relative paths such as configs/analysis/sample_lunar_comparison.yaml or just the bare filename; in the latter case it searches configs/analysis/ under the repository root automatically:

title: "Sample lunar EKF comparison"
output_directory: outputs/lunar_static/analysis
state_error_metrics:
  - pos_error_horizontal_m
  - pos_error_radial_m
  - pos_error_3d_m
  - vel_error_3d_mps
  - clock_bias_m
solutions:
  - name: baseline
    run_dir: ../outputs/lunar_static/baseline
  - name: high_gain
    run_dir: ../outputs/lunar_static/high_gain
  - name: low_power
    run_dir: ../outputs/lunar_static/low_power

Each solution entry points to an existing run directory (the stage-specific folders such as simulate/ and estimate/ must already contain dop_timeseries.csv / dop_time_series.csv and state_error_stats.csv). Paths are resolved relative to the project root, so outputs/... maps to <repo>/outputs/.... Include as many solutions as you like—the tool stacks all of them in the combined CSVs and bar plots. The optional state_error_metrics list restricts the rows copied from state_error_stats.csv (preserving the order you specify) so that plots and delta tables focus only on the metrics you care about; omit it to include every metric. The output_directory field both dictates where artefacts are written and, when provided, doubles as the default run directory for the CLI. If you omit it, pass --run-dir when invoking the command so the tool knows where to write logs and reports; in that case, outputs land in <run-dir>/analysis/. Available state_error_metrics entries map directly to the columns in state_error_stats.csv and include pos_error_horizontal_m, pos_error_radial_m, pos_error_3d_m, pos_error_x_m, pos_error_y_m, pos_error_z_m, vel_error_x_mps, vel_error_y_mps, vel_error_z_mps, vel_error_3d_mps, clock_bias_m, and clock_drift_mps.

Add an optional comparisons block when you need explicit pairwise delta tables:

comparisons:
  - name: high_gain_vs_baseline
    lhs: high_gain
    rhs: baseline

Run the analysis via:

poetry run acons analysis \
  --config configs/analysis/sample_lunar_comparison.yaml

Override the location (for example, when the YAML does not set output_directory) by appending --run-dir <path>. Artefacts always populate <output_directory> when the field exists, otherwise they fall back to <run-dir>/analysis/:

The CLI clears the chosen analysis directory before each run, so copy out any plots you need prior to launching a new comparison.
dop_time_series_comparison.csv – stacked DOP rows with a solution column.
state_error_stats_comparison.csv – stacked EKF error statistics with a solution column.
state_error_by_satellite_count_comparison.csv – stacked p95 errors by visible-satellite count with a solution column.
dop_deltas_<pair>.csv – merged timeline with left/right GDOP/PDOP/... columns plus delta_* (emitted only when comparisons are defined).
state_error_stats_deltas_<pair>.csv – merged stats table with per-metric deltas (RMS, P90, P95, etc.) for each requested comparison.
plots/dop_mean_comparison.png – bar chart of per-solution mean GDOP/PDOP/TDOP/... values.
plots/state_error_stats/absolute/<stat>_comparison.png – grouped bar charts for each statistic (rms, p90, p95, p997, mean, mad, std, …) with numeric labels on every bar.
plots/state_error_stats/relative/<stat>_comparison.png – the same charts showing the percent increase/decrease relative to the first solution listed in the YAML (baseline delta = 0 %). Relative plots omit rows where the baseline metric is zero/undefined to avoid divide-by-zero artefacts.
plots/state_error_by_satellite_count/<metric>_comparison.png – grouped bar charts for horizontal_p95_m, vertical_p95_m, and three_d_p95_m versus visible satellites.

These CSVs can be filtered or plotted in downstream notebooks to visualise how changes in a scenario or estimator tuning propagate to DOP availability and position accuracy while the generated bar plots provide an at-a-glance summary when comparing more than two runs.

Monte Carlo runs

Use scripts/run_monte_carlo_acons.py to execute a repeated poetry run acons all batch while retaining only estimate/state_errors.csv per run. The script stores each run's errors under a dedicated Monte Carlo directory and stacks them into aggregated_state_errors.csv. Pass --measurement-seed-base (and optional --measurement-seed-step) to increment the measurement RNG seed per run for a true Monte Carlo sweep.

manifest.csv lists each run, status, duration, and seed metadata.
aggregated_state_errors.csv stacks all retained errors with a run_id column.

Example (50 runs, TRACE logging, no plots):

poetry run python scripts/run_monte_carlo_acons.py \
  --runs 50 \
  --config configs/scenarios/mars_case.yaml \
  --run-root outputs/TEST_RF \
  --log-level TRACE \
  --measurement-seed-base 1000 \
  --disable-plots

To recompute summaries later from aggregated_state_errors.csv (with optional warmup exclusion), use:

poetry run python scripts/summarize_monte_carlo_state_errors.py \
  --aggregated outputs/TEST_RF_mc/aggregated_state_errors.csv \
  --stats-warmup-seconds 3600

This script writes:

summary_per_run.csv (per-run stats)
summary_across_runs.csv (overall stats from all samples)
summary_across_runs_per_satellite.csv (stats grouped by visible-satellite count)

Grid sweeps & service-volume summaries

Use scripts/run_grid_scenarios.py to sweep the Mars scenario over the site grid stored in data/GRID/AreaGrid.csv. Rows are ordered as lat_deg, lon_deg and the helper script copies configs/scenarios/mars_case.yaml, overwrites the user.location.lat_deg/lon_deg fields for each requested row, and launches the full pipeline with the desired --run-dir base. Each run lands in a dedicated subfolder such as outputs/MAPS/point_0001_lon+022p500_lat-055p385 and stores the generated scenario plus a grid_point_metadata.json file capturing lon/lat, command line, and timestamps.

poetry run python scripts/run_grid_scenarios.py \
  --points 25 \
  --offset 0 \
  --run-root outputs/MAPS \
  --grid-file data/GRID/AreaGrid.csv \
  --scenario configs/scenarios/mars_case.yaml

Skip the CSV entirely by requesting an automatically generated set of approximately even lat/lon samples (Fibonacci sphere) via --auto-grid:

poetry run python scripts/run_grid_scenarios.py \
  --auto-grid 1000 \
  --run-root outputs/MAPS \
  --scenario configs/scenarios/mars_case.yaml

Pass --acons-command ingest-simulate when you only need to regenerate ingest/ and simulate/ artefacts, or --acons-command estimate to rerun the filter using measurements already stored under each point’s run directory (defaults to all). The estimate-only mode requires that <run-dir>/simulate/measurements.json already exists for every point; otherwise the script aborts with a helpful error.

Once the runs finish, aggregate their measurement quality and DOP metrics into service-volume reports with scripts/summarize_service_volumes.py. Service volumes follow the requested latitude bands: SV1 (−40° to +40°), SV2 (+40° to +70° / −40° to −70°), and SV3 (|lat| ∈ (70°, 90°]). The script scans every child of --run-root, pulls simulate/measurement_summary.csv and simulate/dop_timeseries.csv, and writes two CSVs:

<run-root>/service_volume_reports/per_point_stats.csv – lat/lon/service-volume metadata (including the top-level solution identifier parsed from the path under outputs/) and the derived CN₀/DOP/satellite-count metrics per grid point, plus visibility CDF columns copied from each run’s simulate/satellite_visibility_cdf.csv (e.g., visibility_ge_2_pct = % of epochs with at least two satellites tracked).
<run-root>/service_volume_reports/service_volume_summary.csv – aggregated totals and means per service volume (point counts, satellite-visibility ranges, CN₀ averages, GDOP/PDOP statistics).
<run-root>/service_volume_reports/service_volume_state_errors.csv – per-service-volume averages and maxima for the pos_error_3d/horizontal/vertical p95 metrics plus vel_error_3d p95 extracted from each run’s estimate/state_error_stats.csv.
<run-root>/service_volume_reports/plots/ – four scatter maps (map_pos_error_3d_p95.png, map_vel_error_3d_p95.png, map_horizontal_error_p95.png, map_vertical_error_p95.png) that plot every point colored by the corresponding p95 statistic using logarithmic colorbars and a Mars mosaic background (override via --background-image as needed), plus:
map_service_volumes.png – service-volume coverage with latitude markers (counts also printed to the console).
box_* PNGs – boxplots of p95 metrics grouped by service volume to highlight distribution spread.

poetry run python scripts/summarize_service_volumes.py \
  --run-root outputs/MAPS \
  --service-volumes configs/analysis/service_volumes.yaml \
  --background-image data/mosaic/5672_mars_4k_color.jpg

Service volumes come from configs/analysis/service_volumes.yaml (overridable via scripts/summarize_service_volumes.py --service-volumes <yaml>). Edit or clone this file to change latitude bands (multiple entries can share the same name to cover positive/negative ranges).

Both helper scripts accept additional flags (use --help) so you can change the number of points, skip existing run directories, force TRACE logging (default is INFO), or write the aggregated CSVs to a custom destination when sweeping alternate grids. The grid runner records a grid_point_metadata.json with status=success/failed, so failed points can be re-run later without interrupting the rest of the sweep.