Wayve Implements End-to-End MLOps with W&B

Full Speed Ahead 

The race towards a self-driving future is fierce. In a market dominated by tech giants and traditional automakers, one AI company is putting the pedal to the metal to rise above the competition. 

Headquartered in London, UK with a second office in California, Wayve’s mission is to reimagine autonomous mobility through embodied intelligence. This means developing AI systems that enable vehicles to learn, interact and adapt to driving in complex, real-world environments. 

To get there is no easy feat. Wayve needs robust solutions and infrastructure for their machine learning (ML) engineers to move efficiently from experimentation to production. While in-house tools have proven successful, to ensure the end-to-end ML lifecycle is supported, Wayve partners with Weights & Biases. 

Optimizing Developer Experience 

Bringing models to production takes a lot of patience, effort, and resources. 

At Wayve, there’s an entire team dedicated to improving the productivity of their ML engineers. The team, led by Peter Matev, Engineering Manager at Wayve, focuses on developing tools and pipelines for their internal users – and it all starts with data exploration. 

Preparing data is a crucial step of the ML workflow. But, it can be difficult to readily spot patterns and relationships in a dataset, especially if it is in large volumes. “The tools we’re building help our ML engineers to quickly slice through the data in different ways,” explained Peter. “It allows for a better understanding of the data so they can easily generate high-quality training datasets.” 

Once the data is processed, Wayve moves onto model training – this is where W&B comes into play. 

ML is iterative in nature. It requires the diligent tracking of multiple variables. With W&B, evolving information such as hyperparameters, metrics, and artifacts are automatically logged and stored in one place, making it easy to compare runs and identify problem areas during training.

“We actively use W&B to track all our experiments, and it’s allowed us to dig deep into comparisons and monitor in real-time exactly what’s happening in training,” said Peter. 

Beyond that, W&B also helps Wayve monitor system utilization (GPU, CPU, Networking, IO, etc.) and presents it in multiple graphs. These visualizations give insight on possible training bottlenecks and ensure the team is making efficient use of computing resources. 

“W&B allows us to get the best utilization out of our GPUs and compute nodes,” said Peter. “It provides a level of monitoring over our infrastructure so we can continue to train our models at scale.” 

To document all that goes on behind-the-scenes and provide context for their work, Wayve uses W&B Reports extensively. The dynamic tool helps tell the full story of the experiments the team ran, including what worked well, what didn’t and what needs improvement. 

“Our team uses the Reports feature quite a lot,” said Peter. “It’s been useful for them to talk about a particular experiment they ran and share the details with a larger group.” 

Azure with Weights & Biases 

When it comes to powering their ML workflow for both training and inference workloads, Wayve leverages Microsoft Azure. The team uses a combination of AzureML, Azure Kubernetes Service, and Azure native databases and networking infrastructure. Microsoft Azure allows Wayve to deploy their models flexibly, securely, and at a tremendous scale. 

W&B integrates seamlessly with ML workloads running on Microsoft Azure, including all the training and inference workloads run on AzureML. The integration allows Wayve to quickly and easily leverage W&B throughout their entire ML workflow. 

The Road Ahead 

As the AV race intensifies, streamlining and optimizing the processes involved in developing and deploying AI systems become all the more important for Wayve. Along with its in-house tools, Wayve turns to W&B to support the operational necessities of the entire ML lifecycle. Integrating W&B into their workflow means Wayve can log and visualize experiments in real-time, improve resource consumption and drive down cost, and optimize the AI developer experience for productivity gains. 

“We definitely see a huge benefit from using W&B,” said Peter. “The number of experiments we can run in parallel has grown exponentially, and it’s been valuable in ensuring that we get the right insights from those experiments.”