Category: news

AI Driving Olympics 5th edition: results

Posted on | by Ivano Marocchi

AI-DO 5: Urban league winners

This year’s challenges were lane following (LF), lane following with pedestrians (LFP) and lane following with other vehicles, multibody (LFV_multi). 

Let’s find out the results in each category:

LF

  1. Andras Beres 🇭🇺  
  2. Zoltan Lorincz 🇭🇺
  3. András Kalapos 🇭🇺

LFP

  1. Bea Baselines 🐤
  2. Melisande Teng 🇨🇦 
  3. Raphael Jean 🇨🇦

LFV_multi

  1. Robert Moni 🇭🇺
  2. Márton Tim 🇭🇺
  3. Anastasiya Nikolskay 🇷🇺

Congratulations to the Hungarian Team from the Budapest University of Technology and Economics for collecting the highest rankings in the urban league!

Here’s how the winners in each category performed both in the qualification (simulation) and in the finals running on real hardware:

Andras Beres – Lane following (LF) winner



See the evaluation details

Melisande Teng – Lane following with pedestrians (LFP) winner



See the evaluation details

Robert Moni – Lane following with other vehicles, multibody (LFV_multi) winner



See the evaluation details

AI-DO 5: Advanced Perception league winners

Great participation and results in the Advanced Perception league! Check out this year’s winners in the video below:

AI-DO 5 sponsors

Many thanks to our amazing sponsors, without which none of this would have been possible!

Stay tuned for next year AI Driving Olympics. Visit the AI-DO page for more information on the competition and to browse this year’s introductory webinars, or check out the Duckietown massive open online course (MOOC) and prepare for next year’s competition!



Learn more about the MOOC



Learn more about the AI-DO

AI-DO 5 leaderboard update

Posted on | by Jacopo Tani

AI-DO 5 pre-finals update

With the fifth edition of the AI Driving Olympics finals day approaching, 1326 solutions submitted from 94 competitors in three challenges, it is time to glance over at the leaderboards! 

Leaderboards updates

This year’s challenges are lane following (LF), lane following with pedestrians (LFP) and lane following with other vehicles, multibody (LFV_multi). Learn more about the challenges here. Each submission can be sent to multiple challenges. Let’s look at some of the most promising or interesting submissions.

The Montréal menace

Raphael Jean at Mila / University of Montréal is a new entrant for this year. 

An interesting submission: submission #12962 

All of raph’s submissions.

The submissions from the cold

Team JetBrains from Saint Petersburg was a winner of previous editions of AI-DO. They have been dominating the leaderboards also this year.

Interesting submissions: submission #12905

All of JetBrains submissions: JBRRussia1. 

 

BME Conti

PhD student Robert Moni (BME-Conti) from Hungary. 

Interesting submissions: submission #12999 

All submissions: timur-BMEconti

 

Deadline for submissions

The deadline for submitting to the AI-DO 5 is 12am EST on Thursday, December 10th, 2020. The top three entries (more if time allows) in each simulation challenge will be evaluated on real robots and presented at the finals event at NeurIPS 2020, which happens at 5pm EST on Saturday, December 12.

Join the AI Driving Olympics now!

Test your skills in the urban league challenges and qualify for the finals at NeurIPS!
AI-DO 5

AI-DO 5 Update

Posted on | by Andrea Censi

AI-DO 5 Update

AI-DO 5 is in full swing and we want to bring you some updates: better graphics, more maps, faster and more reliable backend and an improved GUI to submit to challenges! 

Challenges visualization

We updated the visualization. Now the evaluation produces videos with your name and evaluation number (as below).

Challenges updates

We fixed some of the bugs in the simulator regarding the visualization (“phantom robots” popping in and out). 

We updated the maps in the challenges to have more variety in the road network; we put more grass and trees to make the maps more joyful!

We have updated the maps with more trees and grass

Faster and more reliable backend

The server was getting slow given the number of submissions, and sometime the service was unavailable. We have revamped the server code and added some backend capacity to be more fault-tolerant. It is now much faster!

Thanks so much to the participants that helped us debug this problem!

We overhauled the server code to make it much faster!

More evaluators

We brought online many more CPU and GPU evaluators. We now encourage you to submit more often as we have a lot more capacity.

We have many more evaluators now!

Submit to testing challenges

We also remind you that the challenges on the front page are the validation challenges, in which everybody can see the output. However what counts for winning are the testing challenges!  

To do that you can use dts challenges submit with the –challenges option

Or, you can use a new way using the website that we just implemented, described below.

Submitting to other challenges

Step 1: Go to your user page, by clicking “login” and then going to “My Submissions”.

Step 2: In this page you will find your submissions grouped by “component”. 

Click the component icon as in the figure.

Step 3: The page will contain some buttons that allow you to submit to other challenges that you didn’t submit to yet.

Join the AI Driving Olympics

Imitation Learning Approach for AI Driving Olympics Trained on Real-world and Simulation Data Simultaneously

Posted on | by Konstantin Chaika

Imitation Learning Approach for AI Driving Olympics Trained on Real-world and Simulation Data Simultaneously

The AIDO challenge is divided into two global stages: simulation and real-world. A single algorithm needs to perform well in both. It was quickly identified that one of the major problems is the simulation to real-world transfer. 

Many algorithms trained in the simulated environment performed very poorly in the real world, and many classic control algorithms that are known to perform well in a real-world environment, once tuned to that environment, do not perform well in the simulation. Some approaches suggest randomizing the domain for the simulation to real-world transfer.

We propose a novel method of training a neural network model that can perform well in diverse environments, such as simulations and real-world environment.

Dataset Generation

To that end, we have trained our model through imitation learning on a dataset compiled from four different sources:

  1. Real-world Duckietown dataset from logs.duckietown.org (REAL-DT).
  2. Simulation dataset on a simple loop map (SIM-LP).
  3. Simulation dataset on an intersection map (SIM-IS).
  4. Real-world dataset collected by us in our environment with car driven by PD controller (REAL-IH).

We aimed to collect data with as many possible situations such as twists in the road, driving in circles clockwise/counterclockwise, and so on. We have also tried to diversify external factors such as scene lighting, items in the room that can get into the camera’s field of view, roadside objects, etc. If we keep these conditions constant, our model may overfit to them and perform poorly in a different environment. For this reason, we changed the lighting and environment after each duckiebot run. The lane detection was calibrated for every lighting condition since different lighting changes the color scheme of the image input.

We made the following change to the standard PD algorithm: since most Duckietown turns and intersections are standard-shaped, we hard-coded the robot’s motion in these situations, but we did not exclude imperfect trajectories. For example, the ones that would go slightly out of bounds of the lane. Imperfections in the robot’s actions increase the robustness of the model. 

Neural network architecture and training

Original images are 640×480 RGB. As a preprocessing step, we remove the top third of the image, since it mostly contains the sky, resize the image to 64×32 pixels and convert it into the YUV colorspace.

We have used 5 convolutional layers with a small number of filters, followed by 2 fully-connected layers. The small size of the network is not only due to it being less prone to overfitting, but we also need a model that can run on a single CPU on RaspberryPi.

We have also incorporated Independent-Component (IC) layers. These layers aim to make the activations of each layer more independent by combining two popular techniques, BatchNorm and Dropout. For convolutional layers, we substitute Dropout with Spatial Dropout which has been shown to work better with them. The model outputs two values for voltages of the left and the right wheel drives. We use the mean square error (MSE) as our training loss.

Results

For the training evaluation, we compute the mean square error (MSE) of the left and the right wheels outputs on the validation set of each data source. 

The first table shows the results for the models trained on all data sources (HYBRID), on real-world data sources only (REAL) and on simulation data sources only (SIM). As we can see, while training on a single dataset sometimes achieves lower error on the same dataset than our hybrid approach. We can also see that our method performs on par with the best single methods. In terms of the average error it outperforms the closest one tenfold. This demonstrates definitively the high dependence of MSE on the training method, and highlights the differences between the data sources.

The next table shows simulation closed-loop performance for all our approaches using the Duckietown simulator. All methods drove for 15 seconds without major infractions, and the SIM model that was trained specifically on the simulation data only drove just 1.8 tiles more than our hybrid approach.

The third table shows the closed-loop performance in the real-world environment. Comparing the number of tiles, we see that our hybrid approach drove about 3.5 tiles more than the following in the rankings model trained on real-world data only.

Conclusion

Our method follows the imitation learning approach and consists of a convolutional neural network which is trained on a dataset compiled from data from different sources, such as simulation model and real-world Duckietown vehicle driven by a PD controller, tuned to various conditions, such as different map configuration and lighting. 

We believe that our approach of emphasizing neurons independence and monitoring generalization performance can offer more robustness to control models that have to perform in diverse environments. We also believe that the described approach of imitation learning on data obtained from several algorithms that are fitted to specific environments may yield a single algorithm that will perform well in general.

 —
 JBRRussia1 team
Join the AI-DO!

Join the AI Driving Olympics, 5th edition, starting now!

Posted on | by Ivano Marocchi

Compete in the 5th AI Driving Olympics (AI-DO)

The 5th edition of the Artificial Intelligence Driving Olympics (AI-DO 5) has officially started!

The AI-DO serves to benchmark the state of the art of artificial intelligence in autonomous driving by providing standardized simulation and hardware environments for tasks related to multi-sensory perception and embodied AI.

Duckietown hosts AI-DO competitions biannually, with finals events held at machine learning and robotics conferences such as the International Conference on Robotics and Automation (ICRA) and the Neural Information Processing Systems (NeurIPS). 

 The AI-DO 5 will be in conjunction with NeurIPS 2020 and have two leagues: Urban Driving and Advanced Perception. 

Urban driving league challenges

This year’s Urban League includes a traditional AI-DO challenge (LF) and introduces two new ones (LFP, LFVM).

Lane Following (LF)

Get the LF kit

The most traditional of AI-DO challenges: have a Duckiebot navigate a road loop without intersection, pedestrians (duckies) or other vehicles. The objective is traveling the longest path in a given time while staying in the lane.

Lane following with Pedestrian (LFP)

Get the LFP kit

The LFP challenge is new to AI-DO. It builds upon LF by introducing static obstacles (duckies) on the road. The objectives are the same as for lane following, but do not hit the duckies! 

Lane Following with Vehicles, multi-body (LFVM)

Get the LFVM kit

In this traditional AI-DO challenge, contestants seek to travel the longest path in a city without intersections nor pedestrians, but with other vehicles on the road. Except this year there’s a twist. In this year’s novel multi-body variant, all vehicles on the road are controlled by the submission.

Getting started: the webinars

We offer a short webinar series to guide contestants through the steps for participating: from running our baselines in simulation as well as deploying them on hardware. All webinars are 9 am EST and free!

Introduction

Learn about the Duckietown project and the Artificial Intelligence Driving Olympics.

  • Nov. 9, 2020
ROS baseline

How to run and build upon the “traditional” Robotic Operation System (ROS) baseline.

  • Nov. 11, 2020
Local development

On the workflow for developing and deploying to Duckiebots, for hardware-based testing.

  • Nov. 13, 2020
RL baseline

Learn how to use the Pytorch template for reinforcement learning approaches.

  • Nov. 16, 2020
IL baseline

Introduction to the Tensorflow template, use of logs and simulator for imitation learning.

  • Nov. 18, 2020
Synch Webinar Calendar

Advanced sensing league challenges

Previous AI-DO editions featured: detection, tracking and prediction challenges around the nuScenes dataset.

For the 5th iteration of AI-DO we have a brand new lidar segmentation challenge.

The challenge is based on the recently released lidar segmentation annotations for nuScenes and features an astonishing 1,400,000,000 lidar points annotated with one of 32 labels.

We hope that this new benchmark will help to push the boundaries in lidar segmentation. Please see https://www.nuscenes.org/lidar-segmentation for more details.

Furthermore, due to popular demand, we will organize the 3rd iteration of the nuScenes 3d detection challenge. Please see https://www.nuscenes.org/object-detection for more details.

AI-DO 5 Finals event

The AI-DO finals will be streamed LIVE during 2020 edition of the Neural Information Processing Systems (NeurIPS 2020) conference in December.

Learn more about the AI-DO here.

Participate in AI-DO 5 now
Get your AI-DO hardware kit
Join our Slack Community

Thank you to our generous sponsors!

The Duckietown Foundation is grateful to its sponsors for supporting this fifth edition of the AI Driving Olympics!

IROS2020: Watch The Workshop on Benchmarking Progress in Autonomous Driving

Posted on | by Ivano Marocchi

What a start for IROS 2020 with the "Benchmarking Progress in Autonomous Driving" workshop!

The 2020 edition of the International Conference on Intelligent Robots and Systems (IROS) started great with the workshop on “Benchmarking Progress in Autonomous Driving”.

The workshop was held virtually on October 25th, 2020, using an engaging and concise format of a sequence of four 1.5-hour moderated round-table discussions (including an introduction) centered around 4 themes.

The discussions on the methods by which progress in autonomous driving is evaluated, benchmarked, and verified were exciting. Many thanks to all the panelists and the organizers!  

Here are the videos of the various sessions. 

Opening remarks

Theme 1: Assessing progress for the field of autonomous vehicles (AVs)

Moderator: Andrea Censi

Invited Panelists:

Theme 2: How to evaluate AV risk from the perspective of real world deployment (public acceptance, insurance, liability, …)?

Moderator: Jacopo Tani

Invited Panelists:

Theme 3: Best practices for AV benchmarking

Moderator: Liam Paull

Invited Panelists:

Theme 4: Do we need new paradigms for AV development?

Moderator: Matt Walter

Invited Panelists:

Closing remarks

You can find additional information about the workshop here.

The Workshop on Benchmarking Progress in Autonomous Driving at IROS 2020

Posted on | by Liam Paull

The IROS 2020 Workshop on Benchmarking Autonomous Driving

Duckietown has also a science mission: to help develop technologies for reproducible benchmarking in robotics.  

The IROS 2020 Workshop on Benchmarking Autonomous Driving provides a platform to investigate and discuss the methods by which progress in autonomous driving is evaluated, benchmarked, and verified.

It is free to attend.

The workshop is structured into 4 panels around four themes. 

  1. Assessing Progress for the Field of Autonomous Driving
  2. How to evaluate AV risk from the perspective of real world deployment (public acceptance, insurance, liability, …)?
  3. Best practices for AV benchmarking
  4. Algorithms and Paradigms

The workshop will take place on Oct. 25, 2020 starting at 10am EDT. 

Invited Panelists

We have  a list of excellent invited panelists from academia, industry, and regulatory organizations. These include: 

  • Emilio Frazzoli (ETH Zürich / Motional)
  • Alex Kendall (Wayve)
  • Jane Lappin (National Academy of Sciences)
  • Bryant Walker Smith (USC Faculty of Law)
  • Luigi Di Lillo (Swiss Re Insurance), 
  • John Leonard (MIT)
  • Fabio Bonsignorio (Heron Robots)
  • Michael Milford (QUT)
  • Oscar Beijbom (Motional)
  • Raquel Urtasun (University of Toronto / Uber ATG). 

Please join us...

Please join us on October 25, 2020 starting at 10am EST for what should be a very engaging conversation about the difficult issues around benchmarking progress in autonomous vehicles.  

For full details about the event please see here.

Round 3 of the the AI Driving Olympics is underway!

Posted on | by Liam Paull

The AI Driving Olympics (AI-DO) is back!

We are excited to announce the launch of the AI-DO 3, which will culminate in a live competition event to be held at NeurIPS this Dec. 13-14.

The AI-DO is a global robotics competition that comprises a series of events based on autonomous driving. This year there are three events, urban (Duckietown), advanced perception (nuScenes), and racing (AWS Deepracer).  The objective of the AI-DO is to engage people from around the world in friendly competition, while simultaneously benchmarking and advancing the field of robotics and AI. 

Check out our official press release.

  • Learn more about the AI-DO competition here.

If you've already joined the competition we want to hear from you! 

 Share your pictures on facebook and twitter. 

Duckietown Workshop at RoboCup Junior

Posted on | by Kirsten Bowser

Duckietown Workshop at RoboCup Junior

In collaboration with the RoboCup Federation, the Duckietown Foundation will be offering workshops at RoboCup 2019 in Sydney, Australia, providing a hands-on introduction to the Duckietown platform.

We will be hosting three one-day workshops as part of RoboCup 2019 from July 4-6, 2019  for teachers, students, and independent learners who are interested in finding out more about the Duckietown platform. Attendance is completely free and everyone is welcome to apply, even if you are not participating in RoboCup. There are no formal requirements, though basic familiarity with GNU/Linux and shell usage is recommended. 

If you would like to apply to attend a workshop, please complete this form. 

We will have Duckiebots and Duckietowns for participants to use. However, you are more than welcome to bring your own Duckiebots, available for purchase at https://get.duckietown.org. 

Congratulations to the winners of the second edition of the AI Driving Olympics!

Posted on | by Kirsten Bowser

Team JetBrains came out on top on all 3 challenges

It was a busy (and squeaky) few days at the International Conference on Robotics and Automation in Montreal for the organizers and competitors of the AI Driving Olympics. 

The finals were kicked off by a semifinals round, where we the top 5 submissions from the Lane Following in Simulation leaderboard. The finalists (JBRRussia and MYF) moved forward to the more complicated challenges of Lane Following with Vehicles and Lane Following with Vehicles and Intersections. 

Results from the AI-DO2 Finals event on May 22, 2019 at ICRA

If you couldn’t make it to the event and missed the live stream on Facebook, here’s a short video of the first run of the JetBrains Lane Following submission.

Thanks to everyone that competed, dropped in to say hello, and cheered on the finalists by sending the song of the Duckie down the corridors of the Palais des Congrès. 

A few pictures from the event

Don't know much about the AI Driving Olympics?

It is an accessible and reproducible autonomous car competition designed with straightforward standardized hardware, software and interfaces.

Get Started

Step 1: Build and test your agent with our available templates and baselines

Step 2: Submit to a challenge

https://challenges.duckietown.org

Check out the leaderboard

View your submission in simulation

Step 3: Run your submission on a robot

https://get.duckietown.org/

in a Robotarium