Skip to content

README.md

Latest commit

 

History

History
59 lines (52 loc) · 2.38 KB

README.md

File metadata and controls

59 lines (52 loc) · 2.38 KB

Evaluation data

  • 0_scenarios: Contains the original scenarios that can be replayed to the models.
  • 1_mixnet: Logs generated by replaying the scenarios to the module using MixNet.
  • 2_indynet: Logs generated by replaying the scenarios to the module using IndyNet.

Overall prediction performance

Evaluates the prediction error and sensitivity against semantics (number of object, speed, history length) of the models. Results are stored to data/evaluation_data/results.

  1. MixNet
python tools/evaluation.py --logdir data/evaluation_data/1_mixnet --save-path data/evaluation_data/results/mix_net --RMSE_tot

---------- Total RMSE (L2-Norm) ----------
Number of datapoints used for evaluation: 38493
Overall RMSE (L2-Norm) = 4.096 m
Overall average velocity in the logs: 71.322 m/s

With the flag --rail, the rail-based prediction is also evaluated in addition.

  1. IndyNet:
python tools/evaluation.py --logdir data/evaluation_data/2_indynet --save-path data/evaluation_data/results/indy_net --RMSE_tot

Output:

---------- Total RMSE (L2-Norm) ----------
Number of datapoints used for evaluation: 38342
Overall RMSE (L2-Norm) = 4.481 m
Overall average velocity in the logs: 71.532 m/s

Line plots

Creates line plots of IndyNet, rail-based model and MixNet, stored to data/evaluation_data/line_plots.

python tools/evaluation_line_plot.py --save-path data/evaluation_data/line_plots

Input-Output correlation

Analysis of MixNet's superposition behaior based on synthetic input curves. Created plots are stored to data/evaluation_data/input_output.

python tools/input_output_analysis.py

Output with seed = 42 is:

Overall RMSE: 0.8044372797012329
weights in: [0.4 0.1 0.3 0.2]
weights out: [0.22620717 0.17092304 0.10150512 0.50136465]

Smoothness plot

Plot of an exemplary scenario to compare the prediction behavior of IndyNet, rail-based prediction and MixNet. Figure is saved to data/evaluation_data/smoothness. Recommended scenario to evaluate is scenario_02 at predictionID = 760. To compare the MixNet with the benchmark model both replayed logs have to be inputted, which are:

  • logdir_benchmark: "data/evaluation_data/2_indynet/12_13_21/"
  • logdir_mixnet: "data/evaluation_data/1_mixnet/14_23_40/"

Run the following command to visualize the exemplary sample:

python tools/visualize_smoothness.py