In our AV report, we talked about the 30 autonomous vehicle companies researching self-driving technologies. And most of these companies are competing to provide a single service: driverless taxi service.
Waymo, GM, and Aptiv are well-known companies building self-driving vehicles for driverless taxi service. In fact, Waymo even successfully launched its driverless taxi services in limited areas.
Uber, being one of the largest ride-hailing companies, has a major stake in the mobility business. Given the competition and opportunities at stake, it becomes crucial for Uber to invest in Self-driving vehicles.
“If we are not tied for first, then the person who is in first, or the entity that’s in first, then rolls out a ride-sharing network that is far cheaper or far higher quality than Uber’s, then Uber is no longer a thing,” Travis Kalanick had told Business Insider in a 2016 interview.
The Curious Case of Uber ATG
To achieve self-driving solutions, in 2016 Uber began researching autonomous vehicles and invested heavily in its self-driving project, better known as Uber ATG. Uber spent a lot of money on self-driving research and related acquisitions, such as buying Otto for 1% of its valuation, which was worth $678 million at that time.
In April 2019, Bloomberg reported that Uber had spent over $1 billion on self-driving technology. Uber paid $457 million in 2018 on self-driving technologies, and the amount grew $73 million in 2019, i.e., to a total of $530 million. By the time they sold the business, its ATG unit had already cost them $303 million in 2020 alone.
By combining all the figures, we have a total investment of around $1.8 billion.
Despite spending billions of dollars, Uber wasn’t making progress and was quite behind its competitors.
One of the prime reasons is the infamous Arizona Crash. After that, Uber had to halt on-road activity at all four locations, including San Francisco. And when they started again, the competitors were already far ahead of Uber.
The truth of the day is Uber is still not making a profit, and the company has been facing huge losses. In 2019, even with $14.1 billion in earnings, the company suffered a $8.6 billion loss, almost 3x the 2018 loss.
The pressure from investors and the shortage of funds led Uber to sell its self-driving unit, Uber ATG, to Aurora for $4 billion, or a 26% stake in Aurora along with a board seat for its CEO, Dara Khosrowshahi.
The interesting part is that Uber also invested an additional $400 million in Aurora. This acquisition increased Aurora’s valuation to $10 billion.
Aurora already has a name in the autonomous vehicle industry, along with other startups such as Zoox, AutoX, and Argo.ai. And with Uber’s ATG, it would surely get an edge in self-driving.
Now the question is – What did Uber have to offer Aurora?
That’s the question we tried to find answers t,o which gave us an insight into Uber’s work in self-driving tech.
To achieve this, we examined Uber’s patent portfolio and research activities in autonomous vehicles to find what the company was up to.
What are the technologies that Uber has been researching?
In recent years, Uber has increased its R&D budget and has been intensively researching self-driving technologies such as image processing, pattern recognition, navigation systems, autopilot, and lidar.
The proof lies in the company’s patent portfolio, which offers a glimpse of its research across different technologies: Uber has 844 patent families and a total of 2497 patents.
These patent families can be classified into six research areas on which Uber was focusing.
The chart above clearly outlines the scope of the company’s self-driving research.
Uber began researching autonomous driving in 2015, when it created a subsidiary, Advanced Technologies Group (Uber ATG). Major players such as SoftBank, Toyota, and Denso had previously invested in Uber ATG to give it a boost.
Further, Uber spent around two billion dollars on its ATG unit. Uber was undoubtedly not the only one to spend this much on self-driving. Still, if they are successful in developing a self-driving vehicle in collaboration with Aurora, it would certainly be the most beneficial, since they are the world’s largest ride-hailing company.
Which self-driving technologies was Uber researching?
We wanted to go a tad deeper and figure out which were Uber’s core self-driving research areas, so we looked up its patents and research publications. Below, we have listed all the core technologies Uber focused on to make its self-driving ride-hailing cab service a reality.
Artificial Intelligence
Uber has a dedicated AI department, UberAI, which is essential for Uber’s future growth, as it underpins many of the platform’s technologies and services.
According to Ghahramani, these are some of the areas in which Uber AI is working.
“On the research side, what we’re trying to do is we invest in areas where we think Uber needs to be at the cutting edge. Those areas of research include reinforcement learning, which is an area of research around systems that make sequential decisions to optimize something in the real world, obviously relevant to Uber; deep learning, a technology that’s used very widely; probabilistic modeling; and evolutionary algorithms— algorithms that can optimize their behavior through generations of improvement.” Zoubin Ghahramani, Chief Scientist at Uber.
UberAI is divided into two teams: AI Engagements and AI Platform.
The AI Engagements team leverages various ML/AI models for forecasting to make data-driven decisions at scale, which include:
Marketplace Forecasting: Predicts user supply and demand to direct driver-partners to high-demand areas before they arise, thereby increasing their trip count and earnings.
Hardware Capacity Planning: Predicts hardware capacity requirements to avoid under-provisioning, which may lead to service outages, or over-provisioning, which can be costly due to underutilized infrastructure.
Marketing: Makes estimates on the marginal effectiveness of different media channels while controlling for trends, seasonality, and other dynamics (e.g., competition or pricing).
Setting Financial Goals: Predicts future values for time-dependent data, such as sales, weekly trip counts, available driver levels, app traffic, and economic conditions.
Its AI Platform team, on the other hand, researches technologies such as signal processing, computer vision, and Natural language processing to address users’ pain points, including crash detection, improved GPS, automatic driver license approval, etc.
Some of these research areas would be useful to Aurora for its self-driving initiative. However, two places we feel Aurora will benefit from, and where Uber AI is mainly focused, are Neural Networks and Machine Learning. Let’s look into both of them.
Neural Networks
Neural Networks are an essential aspect of artificial intelligence. Uber’s patents and research publications show that the company is heavily investing in neural networks to address key challenges in self-driving vehicles.
Patent Insights and Publications
Uber’s neural network patents can be categorized into training, architecture, neural network models, and artificial life. Further, Training and Architecture neural networks are more research-focused as per the patent filings.
In fact, there are numerous research publications that hint at Uber’s desire to use neural networks to improve self-driving.
- For example, one publication discusses how Uber’s deep neural network helps identify previously unseen objects. It’s a challenging task for a self-driving vehicle to segment known and unknown objects. But to make it happen, the ride-hailing giant proposed the Open-Set Instance Segmentation (OSIS) network.
Below is Uber’s note on how the OSIS network works:
The proposed deep convolutional neural network encodes points into a category-agnostic embedding space, where they can be clustered into instances.
As a result, the method can perceptually group points into instances, regardless of whether they belong to a known or an unknown class. We validate our method on two large-scale self-driving datasets and achieve state-of-the-art performance in the open-set setting.
- Another publication discusses the use of graph neural networks for joint interaction and trajectory prediction in Autonomous Driving.
In this research paper, Uber proposed a graph-based model for multi-agent trajectory and interaction prediction, which explicitly models discrete interaction types using programmatically generated weak labels and typed edge models.
The main advantages of this approach are: i) gaining a boost in performance without additional labeling costs when compared to the baseline, and ii) the model can effectively capture the multi-modal behavior of interacting agents while learning semantically meaningful interaction modes.
Uber was further researching training neural networks using different techniques, such as safe mutation based on output gradients (US10699195B2), learning methods using an evolutionary program (US20200151576A1), and training artificial neural networks using evolution-based strategies and novelty search (WO2019116353A1).
Machine Learning
Machine Learning is crucial for autonomous vehicles, as ML-based models improve specific driving systems. Machine learning relies heavily on data to enhance car function day by day. The higher the deal, the faster the improvement. And in terms of data, Uber was and is way behind its counterparts.
Thus, researching ML was a top priority to gain access to more data. However, like other companies, Uber kept the information about how its self-driving cars work hidden until March 2020.
On March 4, 2020, Uber revealed the machine learning technology behind its self-driving cars. Machine learning models were an essential part of Uber’s self-driving cars, handling tasks such as processing sensor input, identifying objects, and predicting where those objects might go.
Uber created a five-step life cycle, from data ingestion to model serving, for the training and deployment of ML models in its self-driving vehicles (illustrated below for reference).
Source – Uber
Uber also developed a tool called VerCd to provide versioning and continuous delivery for all machine learning code and artifacts for Uber ATG’s self-driving vehicle software.
The ML model life cycle and tools like VerCD help researchers manage complex ML models and iterate on them more quickly.
Patent Insights
While this information was released in 2020, the research on ML had been ongoing for quite some time.
While Uber shared the technology behind its driverless car and its machine learning-based models in the last few months, a lot of information could have been figured out if anyone had looked into the ride-hailing company’s published research papers and patents.
Most ML models predict different things for a car, like detecting and prioritizing objects, coordinates, safety incidents, speed limit awareness (EP3655727A1), etc. While other models address space-finding for parking (US20190094858A1), user awareness (US20180157984A1), an AV management system (US10248121B2), etc. All these activities could have been figured out if anyone had kept track of their patenting activity.
LiDAR
LiDAR is a basic necessity for a self-driving car (at least for most autonomous vehicle companies that use LiDAR, while Tesla uses a RADAR and Camera combination to compensate for LiDAR).
Uber has a history with LiDAR, which resulted in Uber paying $244 million and an 18-month sentence for Anthony Levandowski for stealing Waymo trade secrets. But that doesn’t mean Uber hasn’t done any research on their side.
Uber has been researching LiDAR technology, as suggested by its research publications and intellectual property data.
Patent Insights and Publications
Uber has filed multiple LiDAR patent applications, some of which have been granted.
Besides patents, Uber’s LiDAR-related publications since 2018 also hint at its research in this area. In 2020 alone, Uber published 3 LiDAR-related publications, which throw light on their research on real data simulation using LiDAR data and physical adversarial examples.
One of their recent publications on LiDAR provides a realistic LiDAR simulation (LIDARsim) by leveraging real data collected by Uber’s self-driving vehicles as they drive through several cities. A scene will be selected from an extensive catalog of 3D static maps and 3D driving objects, and a self-driving car will be placed there “virtually”. The company further says it developed a novel simulator that captures the power of both physics-based and learning-based simulation.
Uber said the LiDARsim sensor simulator has a small domain gap, enabling more confident testing of the full autonomy stack. Uber uses LiDARsim’s ability to detect unknown objects in the scene with perception algorithms. The company also uses LiDARsim to understand better how the autonomy system performs in safety-critical scenarios in a closed-loop setting that would be difficult to test without realistic sensor simulation.
Some other inventions by the company are basic yet crucial, such as object recognition and classification, for which Uber uses LIDAR sensors, as suggested by its patent (US10310087B2).
Also, one of the few granted LiDAR patents (US10677925B2) addresses the shortcomings of Fixed-beam LiDAR systems and provides solutions to overcome them.
In 2020, Uber filed two more patent applications with the USPTO related to LIDAR, one of which (US20200041622A1) talks about systems and methods, and devices to find the LiDAR cross-talk and the mitigation process, while the other one (US20200033449A1) discusses measuring distance using LiDAR sensors.
V2X Communication
The V2X market is still in its nascent stage, but it is a crucial communication technology for self-driving, and it’s no secret that every self-driving player is researching it.
Patent Insights and Publications
V2X communication is not a big part of Uber’s portfolio, but it does have a few patents that discuss the technology in one way or another, such as EP3679710A1, US20190222986A1, etc. It would be fair to say that Uber wasn’t heavily investing in research for this technology, unlike the other technologies.
However, Uber recognizes that V2X is essential, as it enhances the safety features of a self-driving car. Following the infamous road accident, Uber wanted to ensure it wouldn’t happen again.
Rather than patents, Uber has a few research publications on V2X communication.
In one publication, Uber hinted at using V2V communication for joint perception and prediction. The company explored using vehicle-to-vehicle (V2V) communication to improve the perception and motion-forecasting performance of self-driving vehicles.
By intelligently aggregating the information received from multiple nearby vehicles, Uber can observe the same scene from different viewpoints. It enables a self-driving car to see through occlusions and detect vehicles at long range, even when observations are very sparse or nonexistent.
Uber’s IP and RD efforts alone do not paint the entire picture of the company’s work in Self-driving. The company had also acquired a few self-driving startups to get an edge over other players in the domain. Let’s have a look at these acquisitions.
Uber Self-Driving Acquisitions
Uber’s self-driving research team got some help from its acquisitions. Uber made 12 acquisitions, of which 3 were directly related to self-driving solutions.
Otto
Uber’s first and most famous acquisition of Otto in 2016 was perhaps the most significant move by the company to expand its service beyond ride-hailing. Unfortunately, the acquisition of Otto was a stroke of bad luck for Uber, as the company had to fight Waymo over trade secret theft by Otto’s co-founder, Anthony Levandowski.
Uber even had to shut down the Otto project to focus more on self-driving cars and their main operation.
Otto was researching LiDAR technology, so even though Uber closed its operation, Otto’s technology could still be in use to improve Uber’s self-driving vehicles, or rather, Aurora’s AVs.
Geometric Intelligence
Uber acquired GI in December 2016 and had also built a dedicated AI lab for self-driving research. Geometric Intelligence pioneered machine learning through innovative, patent-pending techniques that learn more efficiently from less data.
“It extends to teaching a self-driven machine to safely and autonomously navigate the world, whether a car on the roads or an aircraft through busy airspace or new types of robotic devices.”,
wrote Jeff Holden, Uber’s chief product officer, in a blog post, announcing the creation of Uber AI Labs.
The acquisition helped Uber’s AI efforts more broadly, with a focus on basic research likely to impact a range of potential uses, including route management.
Mighty AI
Uber acquired Mighty AI, a startup that develops training data for computer vision models, in June 2019.
“The team at Mighty AI has built technology to the label at scale using the latest AI and user experience techniques,” Jon Thomason, vice president of software engineering for Uber’s Advanced Technologies Group, said in a statement.
Uber acquired Mighty AI’s intellectual property, tooling, tech talent, and labeling community, and a little more than 40 employees from Mighty AI joined Uber at its Seattle engineering office, operating as a team within Uber’s self-driving division.
When things don’t go to plan – What was the company’s vision for self-driving?
Uber had planned to make self-driving ride-sharing a reality as soon as possible.
It could have served as its gateway to profitability, but achieving it would definitely be a huge challenge, despite having Aurora.
Uber planned to launch its self-driving cars in cities where weather, demand, roads, and other conditions were most favorable. The newly devised unconventional strategy was designed to help Uber cut down costs and show investors a clear path to profitability.
“The vision for growth is absolutely there. But growth where it makes sense.”
Dara Khosroshahi, Uber’s CEO.
The strategy above hinted that Uber didn’t want to solve every self-driving problem in every city, and thus didn’t want to compete with the likes of Waymo, Tesla, and Lyft. Instead, it would launch its self-driving services in markets where they best suit technological feasibility, user safety, and cost-effectiveness.
According to Eric Meyhofer, the Head of Uber ATG, the company must go through three stages: development, piloting, and commercialization. But as fate would have it, Uber’s ATG unit was in development when it sold its self-driving unit.
Things don’t often go as planned, and Uber ATG’s sale to Aurora, way before its pilot, is a good example.
Concluding Notes
It would be fair to say that Uber was still quite behind other self-driving players like Waymo, Baidu, and Tesla in data and other technologies, such as LiDAR.
Waymo’s vehicles have driven more than 20 million miles, according to its safety report, and Tesla has driven well over 2 billion miles. Baidu, on the other hand, as of December 2020, had driven 4.35 Million Miles with zero accidents, and its growth poses a challenge for most of those working on the technology before Baidu.
By the time Uber had joined the race after the Arizona crash, the progress gap between Uber and other companies had become huge. Thus, the financial struggle was inevitable, leading the company to sell its self-driving unit.
However, this should bring some relief to the company, as the sale of its self-driving research will significantly reduce its expenditures, and the research will still live on through Aurora.
Now, only time could tell how Uber’s strategy to outsource its self-driving research would work out in making its self-driving cab service a reality. Knowing Aurora, I personally feel they might pull off what Uber has wished for.
What’s the reasoning behind the strong intuition? I know a little more about the startups than you do. Want to level up? Read about Aurora along with other Top Autonomous Vehicle Startups here.
