Elon Musk has been encouraging flawed crash
statistics to peddle the sanctified idea that not only are his cars more
environmentally friendly, but they are more likely to save you from a fatal
crash. Tesla motors now come with an Autopilot
suite of features, including one termed Autosteer. And a hastily written NHTSA
report from 2017 concluded that this (adding to the confusion, sometimes
they and now Tesla refer to it as Autopilot) would reduce your probability of a
major crash by 40%. But that’s bogus, as
we prove below, exploring four mathematical ideas from their underlying
data and analysis. Drivers (and
investors) beware! Here are the four
elements to our findings: (1) wrong mileage metric, (2) no paired analysis, (3)
conflated causality and counterfactuals, and (4) NHTSA didn’t endorse.
wrong mileage metric
After the 2016 crash of Tesla that was disconcertingly self-steered into a side of a semi-tractor trailer, Tesla provided the NHTSA (incidentally their parent organization The Department of Transportation follows our website) some basic mileage and airbag deployment data, which we show a highlight of further below. The focus has been on the misuse of a metric, total crashes per million miles). As we show below, in the first of many counterexamples, this is a senseless metric. It doesn’t tell us anything about your changed likelihood of crashing by using Autosteer.
The bottom line is that the bias and personal politics need to stop. The Department of Transportation needs to do a more thorough investigation and publish a better data set for us to comprehend. The public is being taken advantage of with hidden or misleading statistics, garnered without any analysis whatsoever, and making family safety decisions about their car purchase accordingly. Other companies such as UBER and Google are also wanting to jam into this space. We will eventually find out in court that these statistics were invalid, and perhaps it will be a watershed moment for Tesla investors as well on how to be more straight with their clientele.
wrong mileage metric
After the 2016 crash of Tesla that was disconcertingly self-steered into a side of a semi-tractor trailer, Tesla provided the NHTSA (incidentally their parent organization The Department of Transportation follows our website) some basic mileage and airbag deployment data, which we show a highlight of further below. The focus has been on the misuse of a metric, total crashes per million miles). As we show below, in the first of many counterexamples, this is a senseless metric. It doesn’t tell us anything about your changed likelihood of crashing by using Autosteer.
To confirm, indeed the government report only emphasized
these flawed metrics and nothing more.
From 1.3, to 0.8, and the elementary conclusion that the
post-installation crash “rate” is 40% lower than the pre-installation “rate”. Now we can begin to understand how fickle
such a metric is and how it should not be taken seriously.
no paired analysis
Now another issue is that in this case they had, but failed to utilize, the ability to look at paired variables. The individual driver data shows a dispersion of crash information, before and after installation. Individual differences can be far more variable, and hence alter the conclusion’s meaning, far more than looking at the less variable “summary aggregates” as they did (e.g., total crashes per millions of miles for all the Tesla drivers combined).
Now another issue is that in this case they had, but failed to utilize, the ability to look at paired variables. The individual driver data shows a dispersion of crash information, before and after installation. Individual differences can be far more variable, and hence alter the conclusion’s meaning, far more than looking at the less variable “summary aggregates” as they did (e.g., total crashes per millions of miles for all the Tesla drivers combined).
On an aside, the sample size would also be
sufficient given the many thousands of drivers, covering hundreds of millions
of travelled miles. And we know
by the crash rates would therefore come to thousands of crashes. Both Tesla and the NHTSA should have looked
at and published their statistical variation about their 1.3 and 0.8 statistic (let
alone show the individual driver data per year), else we are really left
with hollow and -as we see- easily manipulatable information.
conflated causality and counterfactuals
As noted in the initial section, Autopilot is not a single feature, but a suite of features that includes Autosteer and many others. An advanced brake system for example is also used in Autopilot. When many suite products are available and installed, all at about the same time, it is impossible to tease apart the individual contributions from any one feature (such as Autosteer) by looking at these aggregate crash data. What is the "40%" really in reference to? Further, it is even impossible -as we show below- to go further and place causality with the Autopilot system overall!
conflated causality and counterfactuals
As noted in the initial section, Autopilot is not a single feature, but a suite of features that includes Autosteer and many others. An advanced brake system for example is also used in Autopilot. When many suite products are available and installed, all at about the same time, it is impossible to tease apart the individual contributions from any one feature (such as Autosteer) by looking at these aggregate crash data. What is the "40%" really in reference to? Further, it is even impossible -as we show below- to go further and place causality with the Autopilot system overall!
We see insights above where some drivers have
turned off their Autosteer product for a more comfortable drive, and how no one
segregated the analysis to see those who use Autosteer only in certain ways (to
understand the driving context behind the crash events). Finally, nearly 70% of the drivers did not
have a pre and post installation driving data associated with them. A proper statistical design would have looked
at those who never installed Autosteer, and measure the counterfactual
crash data. So again, so many missed
opportunities (other than for misleading conclusions).
NHTSA didn’t endorse
Finally, it turns out that the NHTSA never analyzed anything! We find out last week that the government merely took the Tesla data and the now infamous “40%” statistic, and then stated that in their report. Tesla then turned around and heavily sold this apparent lie (see this thread as an example). We have no idea what is the quality of the data, what other questions should have been pursued, nor anything else from this very hands-off approach that the NHTSA took.
And unfortunately it’s not the first time that wild guesses were propagated. Steven Jurvetson, Tesla’s and Elon Musk’s main investor, enjoyed this scientific article here valuing Tesla’s battery performance. Something that had dramatic financial importance to Tesla’s financials in the early infant days of this company product.
The drama with poor analysis and communications continue to be an ire for the NTSB and others who are truly trying to get ahead of this. And we also see some small insurance companies that have begun to offer collision insurance discounts to Tesla Autopilot customers based on the “40%” statistic!
NHTSA didn’t endorse
Finally, it turns out that the NHTSA never analyzed anything! We find out last week that the government merely took the Tesla data and the now infamous “40%” statistic, and then stated that in their report. Tesla then turned around and heavily sold this apparent lie (see this thread as an example). We have no idea what is the quality of the data, what other questions should have been pursued, nor anything else from this very hands-off approach that the NHTSA took.
And unfortunately it’s not the first time that wild guesses were propagated. Steven Jurvetson, Tesla’s and Elon Musk’s main investor, enjoyed this scientific article here valuing Tesla’s battery performance. Something that had dramatic financial importance to Tesla’s financials in the early infant days of this company product.
The drama with poor analysis and communications continue to be an ire for the NTSB and others who are truly trying to get ahead of this. And we also see some small insurance companies that have begun to offer collision insurance discounts to Tesla Autopilot customers based on the “40%” statistic!
The bottom line is that the bias and personal politics need to stop. The Department of Transportation needs to do a more thorough investigation and publish a better data set for us to comprehend. The public is being taken advantage of with hidden or misleading statistics, garnered without any analysis whatsoever, and making family safety decisions about their car purchase accordingly. Other companies such as UBER and Google are also wanting to jam into this space. We will eventually find out in court that these statistics were invalid, and perhaps it will be a watershed moment for Tesla investors as well on how to be more straight with their clientele.
No comments:
Post a Comment