Front page Forums 🤖 🚗 AI Driving Olympics 🏆 Rules discussion Evaluation metrics clarification for lane following challenge

Tagged: ,

This topic contains 4 replies, has 2 voices, and was last updated by https://www.duckietown.org/wp-content/uploads/ap_avatars/d61e4bbd6393c9111e6526ea173a7c8b.jpg Julian Zilly 9 months, 2 weeks ago.

Viewing 5 posts - 1 through 5 (of 5 total)
  • Author
    Posts
  • #29130

    placaille
    Participant

    I am slightly confused by some of the evaluation metrics listed in the lane following challenge (link to description).

    In particular, what does good_angle refer to? I noticed the description which reads “Integral of the lane angle absolute value”, but cannot figure conceptually what the lane angle absolute value may be. Also, from the leaderboards it seems that the higher is better.

    The final metric valid_direction also confuses me, but I feel is related to the good_angle. I am hoping clarifications on the good_angle metric will help things be clearer.

    Thanks!

    #29288
    https://www.duckietown.org/wp-content/uploads/ap_avatars/d61e4bbd6393c9111e6526ea173a7c8b.jpg
    Julian Zilly
    Moderator

    Hi placaille,

    The “good_angle” metric refers to the time integrated angle of the direction of the Duckiebot with respect to the lane. If this angle is large on average, we know that the Duckiebot “wiggles” a lot while driving. Currently, I suspect there is a small bug in the computation of this score which we will fix momentarily. This is why the numbers are so large. (code is here: https://github.com/duckietown/gym-duckietown/blob/aido1_lf1_r3-v3/gym_duckietown/simulator.py#L1011)

    The “valid_direction” metric is integrating the condition that the angle of the Duckiebot with respect to the lane direction is greater than 20 deg. (https://github.com/duckietown/challenge-aido1_LF1/blob/v3-more-exposure/scorer/read_scores.py#L59) This metric supplements the good angle metric. Overall they aim to make a similar judgement on how much the driving of the Duckiebot oscillates. Again I suspect there is minor bug that will be fixed soon.

    Additionally, these metrics will likely be divided by the survival time which should make the scores more interpretable as “average deviations” from the correct steering angle and driving direction.

    Thanks for asking.

    #29289
    https://www.duckietown.org/wp-content/uploads/ap_avatars/d61e4bbd6393c9111e6526ea173a7c8b.jpg
    Julian Zilly
    Moderator
    #29307

    placaille
    Participant

    Thanks for providing additional information.

    According to current rankings, it seems like the leaderboard is scoring a higher “good_angle” as a positive thing, as well as a higher “valid_direction”. If that is the bug you are referring to then all is good, otherwise maybe that’s something you guys should look into.

    Once again, thanks a lot!

    • This reply was modified 9 months, 2 weeks ago by  placaille.
    #29362
    https://www.duckietown.org/wp-content/uploads/ap_avatars/d61e4bbd6393c9111e6526ea173a7c8b.jpg
    Julian Zilly
    Moderator

    Exactly, should be “lower is better” for those metrics. This will be fixed. Thank you for the input.

Viewing 5 posts - 1 through 5 (of 5 total)

You must be logged in to reply to this topic.

Close Menu