October 30, 2018 at 4:53 pm #29130
I am slightly confused by some of the evaluation metrics listed in the lane following challenge (link to description).
In particular, what does
good_anglerefer to? I noticed the description which reads “Integral of the lane angle absolute value”, but cannot figure conceptually what the lane angle absolute value may be. Also, from the leaderboards it seems that the higher is better.
The final metric
valid_directionalso confuses me, but I feel is related to the
good_angle. I am hoping clarifications on the
good_anglemetric will help things be clearer.
Thanks!November 1, 2018 at 9:26 am #29288
The “good_angle” metric refers to the time integrated angle of the direction of the Duckiebot with respect to the lane. If this angle is large on average, we know that the Duckiebot “wiggles” a lot while driving. Currently, I suspect there is a small bug in the computation of this score which we will fix momentarily. This is why the numbers are so large. (code is here: https://github.com/duckietown/gym-duckietown/blob/aido1_lf1_r3-v3/gym_duckietown/simulator.py#L1011)
The “valid_direction” metric is integrating the condition that the angle of the Duckiebot with respect to the lane direction is greater than 20 deg. (https://github.com/duckietown/challenge-aido1_LF1/blob/v3-more-exposure/scorer/read_scores.py#L59) This metric supplements the good angle metric. Overall they aim to make a similar judgement on how much the driving of the Duckiebot oscillates. Again I suspect there is minor bug that will be fixed soon.
Additionally, these metrics will likely be divided by the survival time which should make the scores more interpretable as “average deviations” from the correct steering angle and driving direction.
Thanks for asking.November 1, 2018 at 10:07 am #29289
Correction: The scores are read here: https://github.com/duckietown/challenge-aido1_LF1/blob/aido1_LF1_r3-v3/scorer/read_scores.py#L57November 1, 2018 at 6:35 pm #29307
Thanks for providing additional information.
According to current rankings, it seems like the leaderboard is scoring a higher “good_angle” as a positive thing, as well as a higher “valid_direction”. If that is the bug you are referring to then all is good, otherwise maybe that’s something you guys should look into.
Once again, thanks a lot!
November 2, 2018 at 4:19 pm #29362
- This reply was modified 9 months, 2 weeks ago by placaille.
Exactly, should be “lower is better” for those metrics. This will be fixed. Thank you for the input.
You must be logged in to reply to this topic.