Driverless Car Report Card: Who Should Get Top Grades?


Tesla, Uber, Waymo, GM, and more: who's leading the class?

11 min read

The big-name tech companies and auto manufacturers that we know are developing driverless cars have kept many details about their progress under wraps, likely with dreams of a big consumer-facing reveal and to prevent competitors stealing their ideas.

Most news we do hear about self-driving cars comes from the companies building them, which means the news is likely to be framed in flattering ways that don’t necessarily provide the full picture of what is and is not working in their autonomous vehicle development.

But we all want the answer to the big question: how close are driverless cars to hitting the market?

To get as close as possible to an answer, we’ve looked at were these companies stand in regards to:

  • how the vehicles are performing
  • company funding (less relevant for established automakers than it is for new tech companies)
  • current and future innovation plans assessed by patent holdings

We’ve collected the stats, and leave the results open to your interpretation: which driverless car maker is top of the class?

First, How Are Driverless Cars Being Tested?

Even though 13 states (plus Washington, D.C.) have passed laws relating to driverless vehicles and their testing, only California requires annual reports from companies conducting tests on public roads. California requires each company running tests in the state (currently there are 11) to submit “disengagement reports” which note how many times the human safety driver had to take back control from the autonomous driving system from November 30, 2015 until December 1, 2016. Other than the reports submitted to California, the only details we have about how driverless cars are performing come from highly polished press releases and details leaked to news organizations.

The reports required by the California DMV provide more information than the public would otherwise have access to, and they are useful in their own way, but they alone cannot be considered scientific measurements of driverless vehicle progress, argues Wired, because the reports do not require companies to release information about a number of variables crucial to understanding why the disengagements occurred.

But these reports are limited, Wired notes. California doesn’t require that the disengagement reports include information about weather, where the disengagements happened, or whether the vehicles were exploring new terrain or following well-worn paths. They don’t include details about the human operators, who are perhaps the biggest variable of all, because at least some of the time, they are the ones deciding when to take back control.

Still, the disengagement reports offer the closest thing to comparable data among companies. To that information we’ve added the newest available details about funding, which tells us how the market values each company; patents, which tells us about each company’s commitment to innovation; how many vehicles with true autonomous technology are available for purchase; and even some leaked reports on vehicle performance. You might take our assessment with a grain of salt, but considering the info out there, here’s how today’s driverless cars are stacking up:

Driverless Car Report Cards

Uber

The California Department of Motor Vehicles revoked Uber’s driverless vehicles’ registrations in December of last year, and Uber quickly moved operations to Arizona. As such, Uber is not one of the 11 companies that submitted a disengagement report to the state of California at the end of 2016.

Uber doesn’t voluntarily offer comprehensive data about how often humans need to take control of its self-driving cars, but newly leaked documents reveal a first look at the vehicles’ performance. Uber’s internal performance report for self-driving car tests they ran during the first week of March 2017 in Arizona were made public when Recode obtained a copy of the report.

Bryant Walker Smith, a University of South Carolina law professor and a member of the U.S. Department of Transportation’s Advisory Committee on Automation in Transportation told BuzzFeed News of the reports, “it’s difficult to draw conclusions about the progress of Uber’s self-driving car program based on just one week of disengagement metrics, adding that the figures suggest that safety drivers appear to intervene regularly out of caution — even in cases where an accident may not be imminent.”

During the week in question, Uber tested its driverless vehicles on two different roads in Arizona. The first, called Scottsdale Road, is a straight, 24-mile stretch that runs through the city of Scottsdale, and the second, a “loop” at Arizona State University (ASU). The results:

  • Scottsdale Road: Uber’s autonomous vehicles were able to travel an average of 0.67 miles without any driver intervention and 2 miles, on average, without any “bad experiences.” A bad experience, as defined by Uber, is an incident “in which a car brakes too hard, jerks forcefully, or behaves in a way that might startle passengers.”
  • ASU Loop: The driverless vehicles traveled a total of 449 miles with no “critical interventions” – these are cases in which either the driver took control of the system or the system returned control to the driver to “prevent a likely collision.” The vehicles drove 1.4 miles, on average, without disengagement and 7 miles, on average, without a “bad experience.”

Though it’s difficult – and to be fair to Uber, probably impossible – to draw definitive conclusions about the progress of its driverless vehicles based on this report alone, Smith explained to BuzzFeed News that it was possible to use the results to generalize whether Uber would soon be able to remove the human operators from their autonomous vehicles.

To take out the safety drivers, you would want far better performance than these numbers suggest, and you’d want that to be consistently better performance,” Smith said. “If these are actual bad experiences for someone inside the vehicle, then that probably doesn’t compare very favorably to human driving. How often do people go 10 miles or 10 minutes and have a viscerally bad experience?”

The following details provide more insight into Uber’s progress with autonomous driving systems:

  • Patents filed related to autonomous cars: 138 (80 pending)
  • Vehicles with autonomous technology available for purchase: 0
  • Funding or endorsements published for autonomous tech development: $8.81 Billion*
  • Important partnerships: Uber just teamed up with Mercedes-Benz, which will build autonomous vehicles for Uber’s network

*Amount specifically allocated to driverless vehicle development unknown

Driverless Uber San Francisco | Should Driverless Cars Require Voter Approval?

 

For those companies which DID submit disengagement reports, here’s how they stacked up:

Tesla

  • Number of vehicles tested on California public roads: 4
  • Miles driven on public roads: 550
  • Disengagements: 182
  • Disengagement frequency: 3.01 miles

(Data from Tesla’s 2016 report to the California DMV)

Waymo (formerly the Google self-driving car project)

  • Number of vehicles tested on California public roads: 60
  • Total miles driven on public roads: 645,868
  • Disengagements: 124
  • Disengagement frequency: Every 5,208 miles

(Data from Waymo’s 2016 report to the California DMV)

  • Patents filed related to autonomous cars: 238
  • Vehicles with autonomous technology available for purchase: 0
  • Funding or endorsements published for autonomous tech development: $0

GM Cruise

  • Number of vehicles tested in California: 25
  • Total miles driven on public roads: 9,730
  • Total disengagements: 149
  • Disengagement frequency: every 214 miles

(Data from GM’s 2016 report to the California DMV)

  • Patents filed related to autonomous cars: 907
  • Vehicles with autonomous technology available for purchase: 0
  • Funding or endorsements published for autonomous tech development: $18.8 Million

Ford

  • Total number of vehicles: 2
  • Total miles driven on public roads: 590
  • Total disengagements: 3
  • Disengagement frequency: every 196 miles

(Data from Ford’s 2016 report to the California DMV)

  • Patents filed related to autonomous cars: 521
  • Vehicles with autonomous technology available for purchase: 0
  • Funding or endorsements published for autonomous tech development: $0

BMW

  • Number of vehicles tested in California: 1
  • Total miles driven: 638
  • Total disengagements: 1
  • Disengagement frequency: Every 638 miles

(Data from BMW’s 2016 report to the California DMV)

  • Patents filed related to autonomous cars: 442
  • Vehicles with autonomous technology available for purchase: 0
  • Funding or endorsements published for autonomous tech development: $0

Bosch

  • Number of vehicles tested in California: 3
  • Total miles driven by all vehicles on public roads in 2016: 983
  • Total disengagements: 1442
  • Disengagement frequency: Every .7 miles

(Data from Bosch’s 2016 report to the California DMV)

  • Patents filed related to autonomous cars: 2655
  • Vehicles with autonomous technology available for purchase: 0
  • Funding or endorsements published for autonomous tech development: $0

Delphi Automotive Systems

  • Number of vehicles tested in California: 2
  • Total miles driven by all vehicles on public roads from December 1 2015 Nov 30 2016: 3,125
  • Total disengagements: 178
  • Disengagement frequency: Every 17.5 miles

(Data from Delphi’s 2016 report to the California DMV)

  • Patents filed related to autonomous cars: 32
  • Vehicles with autonomous technology available for purchase: 0
  • Funding or endorsements published for autonomous tech development: $0

Honda:

  • Did not test on public roads in California in 2016 and did not submit a report about disengagements to the California DMV
  • Patents filed related to autonomous cars: 952
  • Vehicles with autonomous technology available for purchase: 0
  • Funding or endorsements published for autonomous tech development: $0

Nissan

  • Number of vehicles tested in California: 5
  • Total miles driven by all vehicles on public roads from December 1 2015 Nov 30 2016: 4,099
  • Total disengagements: 28
  • Disengagement frequency: Every 146 miles

(Data from Nissan’s 2016 report to the California DMV)

  • Patents filed related to autonomous cars: 1169
  • Vehicles with autonomous technology available for purchase: 0
  • Funding or endorsements published for autonomous tech development: $0

Mercedes-Benz

  • Total number of vehicles tested in California: 1
  • Total miles driven on public roads: 673
  • Total disengagements: 336
  • Disengagement frequency: Every 2 miles

(Data from Mercedes-Benz’s 2016 report to the California DMV)

  • Patents filed related to autonomous cars: 961
  • Vehicles with autonomous technology available for purchase: 0
  • Funding or endorsements published for autonomous tech development: $0

Volkswagen

  • Did not test on public roads in California in 2016 and did not submit a report about disengagements to the California DMV.
  • Patents filed related to autonomous cars: 1140
  • Vehicles with autonomous technology available for purchase: 0
  • Funding or endorsements published for autonomous tech development: $0

car driving

Who Gets Top Grades on the Driverless Car Report Card?

As we’ve said, the data here is tricky, and there are so many unknowns! An established automaker like Mercedes-Benz doesn’t need funding in the way a new tech company does, and sheer number of patents held can’t tell the whole story about company innovation.

So which company is leading the race?

Waymo appears to have the fewest disengagements recorded as well as the most miles tested on public roads, but Uber’s partnership with Mercedes-Benz, among others, and its incredible amount of funding speak in its favor. However, Tesla actually has vehicles on the road with autonomous technology capable of driving the car (under limited conditions) and its soon-to-be-released affordable Model 3, which will also have autonomous technology, could make autonomous driving accessible to a wide variety of drivers.

For now, it’s a speculation game.

Would you pick a clear winner from the stats here? Which factors do you think matter most?