How Woven Leverages W&B to Drive Continuous Learning
The mobility market is constantly evolving, as new road users and vehicles like kick scooters, next-gen vehicles, electric bikes and Mobility as a Service (MaaS) systems are regularly introduced, and drivers and cars have to adapt accordingly.
According to ML engineer Evan Cushing, that constant state of evolution is what makes the projects him and his team are working on at Woven—a mobility technology subsidiary of Toyota Motor Corporation focusing on driving software transformation within the Toyota group—so fascinating.
“We work on really interesting, challenging problems on a scale that you can’t find with really any other space or company,” said Cushing. “What makes autonomous driving an interesting ML problem that requires tools like Weights & Biases is that it’s a highly dimensional problem. One of these challenges is the need for an ML system of continuous learning.”
The team’s mission is to deliver safe, intelligent, human-centered autonomous driving and mobility solutions. Missed detections represent huge safety risks to passengers and the community around them, putting huge pressure on the company and Evan’s ML team to deliver cutting-edge models and solutions.
Challenges with the “Autonomy 1.0” Workflow
Cushing and his team recognized several pain points with their “1.0” process and traditional ML workflow for autonomous driving, from data collection through to training and deployment. These included:
- A method of data collection—using test vehicles driving fixed routes without fully capturing the rich and diverse data distributions of real-world driving scenarios—that didn’t scale.
- Having to manually curate data from test vehicles to become part of the Test Set used to benchmark. Team members would review video to find and curate appropriate scenes for training, kaizen scenarios that might be challenging for ML models or AI systems, such as animals on the road. This was also not scalable.
- A slow and time-consuming data labeling process working with a third-party data annotation service, which can take up to a month.
That entire process from data collection through to data annotation, and then retraining and testing the models and deploying to production, historically took over a year from start to finish. Evan and his team felt like they had to 10x this process in speed and scale in order to deliver truly continuous learning and safe autonomous driving for Toyota.
10x Improvement Using the Principles of Kaizen
Their biggest improvement was moving away from test vehicles for data collection, to leverage their large fleet of customer vehicles with data-collecting sensors, which led to substantially greater volume of data collected in a much shorter time. Those sensors also used automated triggers to find Kaizen scenarios, for example if it sensed a huge increase in sudden g-force. That might potentially be an unsafe situation and a valuable scene to train models on.
Autolabeling and dataset queries during the data curation workflow also led to a huge improvement in time savings, from 20 months down to just 2 months. The team automated data annotation with machine learning, automatically assigning labels to data instead of humans manually curating and annotating scenes to train on. The team could now query over the dataset to search for scenes that might be Kaizen scenarios, such as “deer crossing a road” or “child on a bicycle” to pull up all the samples of this particular scenario.
Supporting all these efforts throughout the entire workflow was a best-in-class ML framework that included using Weights & Biases as a central system of record, PyTorch as their core framework, Hydra to manage configurations, and both on-prem (DGX) and cloud (AWS EC2 and SageMaker) infrastructure.
The Impact of Weights & Biases
Weights & Biases in particular has become an essential tool for Woven by Toyota for developing, communicating, and collaborating on ML projects across organizations and product applications.
“Experiment tracking has given us 10x velocity and enabled us to share results with each other much faster, with tractability and traceability,” said Cushing. “Sweeps has helped us automate previously tedious processes, and Reports has helped raise the standard for good ML at Toyota.”
The team has also come up with some unique ways to use Weights & Biases, coming up with what Cushing described as a “really fun Kaizen pro-tip”: using Tables to serve as a de facto models leaderboard.
The team shares models across teams and functions, and aggregates model runs that have trained on the same dataset version. They then use Tables to order them by evaluation metrics to see who’s doing what, which model versions are leading, and who on the team is leading the pack.
“It keeps the team aligned on project development,” said Cushing. “The leaderboard gives us cross-functional visibility into the data, techniques, and squads that are delivering great results.”
And the team isn’t resting on their laurels, both with their use of Weights & Biases and with the overall Autonomy 2.0 workflow they’re working towards. Cushing plans to further integrate into the W&B platform, particularly with Launch (for seamless connectivity to compute to run training and inference jobs at even greater scale) and the Model Registry (to centralize and organize all model management and deployment).
Meanwhile, Autonomy 2.0 will continue improving from large behavioral datasets, better offline simulation for validation, and more affordable hardware costs to promote greater scale. The near-future of autonomous driving is an exciting one, spearheaded by the likes of Cushing and his team at Woven by Toyota.