Pages

▼

Monday, July 3, 2017

Zillow misestimates, despite Big Data

Sequel to this article is here.

Despite millions of statistical models and proprietary data points behind Zillow’s price estimate (Z-est), the Z-est fails a typical real estate agent in predicting an ultimate home’s sales price.  How did we get here again with Big Data models, and how do such large-enough discrepancies occur?  In the end, even though over a hundred million homes have been seen on Zillow’s big data platform and at least ½ of them price analyzed on the cloud, the corresponding data points to fit a model suggest something to be desired.  What are you willing to pay for a home?  That’s something that is a bit of art in terms of assessing the ultimate transaction price, and yet Zillow’s models attempt to converge some available data and get to an answer, yet always lacking the data essential to do a better job than a human.  We see in the map below that there are broad differences in the level of Z-est errors, just as we move East/West across the plains.  The price estimates in their early days were so bad that it was not uncommon the estimates were off by more than +/- 20%!  Imagine that.  Say having $1/2 million dollars and finding out that the home you want is showing a $600k Z-est, or only getting $417k worth of home within a $1/2 million Z-est (or having to be in a significantly worse neighborhood, etc.)  Of course, Zillow has improved their model substantially in recent years, but it is still significantly worse and at times completely futile versus a human realtor estimate and Zillow has modeling accuracy that makes their home resale suggestions untrue (how do they know what are the marginal improvement drivers of price when -in their fat tails- they frequently have prices that are >40% erroneous in either direction?!)  Realtors tend to gauge the local nuances and flow data better, and interpret estimates that are far closer to the actual selling price.  However, their estimates are also biased downward (priced slightly to the economic disadvantage of the seller, to quickly realize their commission).  Consumers generally prefer though a reliably biased estimate versus the hassle of first-degree price discrimination (e.g., going into a car dealership and having no idea how much you will pay versus someone else and for what type of car). 

The best approach is to use a blend of a weathered realty service with the centralized and less biased, machine learned Z-est.  Both provide useful insight into the range and distribution of pricing one might be having to consider.  Look at the map below for the Zillow median absolute error (a statistical trick they use instead of contrasting the cumulative mean absolute error, or the squared errors, standard deviation, or other higher-order moments of Zillow's vicious forecasting tail-errors) for their homes sold across the 50 states.  There is a striking pattern where West Coast Z-estimates are much tighter fit versus on the East Coast.  Undoubtedly there is more story than simply state-level analysis, since there is not much of a reason for why simply buying a home somewhere in a large state would alone dictate how much more accurate a realtor is versus the Z-est.


People tend to look for features of communities within a state, to select where to live.  Manhattan real estate is very different from Upstate New York.  Chicago is very different from the rest of Illinois.  Philadelphia and Pittsburgh are different from the entire state between them.  The housing concentration and expensiveness of a county can both drive the Z-est’s accuracy down.  While larger sample sizes are helpful for model accuracy, it is the quality and uses of local data in integration with housing metrics that are also important.  People in über-popular Manhattan for example place less premium on garage size versus those who live in mansions outside of Houston.  And the latter may desire characteristics that are constantly being learned from human interactions and difficult to map out for every home (e.g., proximity to a recently-opened company headquarters).  Leading housing consultant Jonathan Miller reviewed these findings and agrees with the difficulties for Zillow estimates in dense cities, for example those with a towering skyline: "A 3rd floor condo in Manhattan called 3A and the exact same unit on 33rd floor in same building called 33A can't be distinguished in value."  And data scientist and Columbia University colleague Kaiser Fung responds: "Recognize the nature of your data.  Here: there are not that many transactions, especially when forecasting at the local level."

What we see below is the break-out of the Zillow information into the thousands of counties across the United States, and a measurement of the Z-est model accuracy based on the housing density (Zillow estimated homes per square mile) and the number of homes in the county (for consistency with model sample size).  The outcomes are highly compelling, evidencing that in the belly of the distribution (median density counties and median populated counties), we get the typical 7% median error.  But as we reduce the county population or increase the county density, or both, we see we get into a smaller niche of homes that Z-est does a dreadful job in properly differentiating.  And hence the still explosive errors of well over 12% (12% is simply the middling county in the red cells). 

And in the other corner (in green), we see the locations where Z-est shines.  Providing a good amount of higher quality price estimates to a certain sub-geography of the U.S.  The matching of locally available and “high quality” statistical data, with consumer appetite, simply works better in the West now.  But that could always change based on myriad of factors, competition, and artificial model-overfit.  And depending on when new performance tests are run in future testing month(s).  So the question will always remain whether the data for the East Coast improves to meet the high quality of the West Coast Z-est model, or if the West Coast models lose their strength and depreciate to the poor-quality we’ve seen on the East Coast?  And either way, the results show there is a long way to go before the Z-est can be seen as a sturdy, stand-alone pricing doctrine.

Side note.  This website recently grew to over 100 thousand twitter followers and recently had 4 thousand facebook likes from the same number of followers there.  See below.
at

No comments:

Post a Comment