RATINGS/DECODED

Explainers / RESEARCH

Do star ratings actually predict quality?

Weakly, where it has been tested. Ratings clearly change what people buy: one study tied a one-star Yelp increase to 5 to 9 percent more restaurant revenue. But a study of 1,272 products found average ratings did not converge with Consumer Reports scores. Ratings measure satisfaction, which overlaps with quality without being the same thing.

Two different questions hide inside the title. Do ratings matter, meaning do they change behavior? And are ratings right, meaning do they line up with some outside measure of how good the thing is? The research answers the first with a firm yes and the second with a hedge.

Ratings move money

The strongest evidence uses a quirk of display. Yelp shows a rounded average, so a restaurant at 3.74 displays 3.5 stars and one at 3.76 displays 4. The two are essentially identical in what reviewers think of them, and they look different to customers. Comparing businesses just either side of a rounding threshold isolates the effect of the displayed rating from the effect of actually being better.

Michael Luca did this with Washington State revenue data and found that "a one-star increase in Yelp rating leads to a 5-9 percent increase in revenue". The effect was "driven by independent restaurants"; ratings did not affect chains.11 Michael Luca, Harvard Business School Working Paper 12-016, Reviews, Reputation, and Revenue: The Case of Yelp.com

Anderson and Magruder applied the same idea to reservations and found that "an extra half-star rating causes restaurants to sell out 19 percentage points (49%) more frequently", with larger effects when other information about the restaurant was scarce.22 Anderson and Magruder, The Economic Journal 122(563), 2012, Learning from the Crowd: Regression Discontinuity Estimates of the Effects of an Online Review Database

For products, Chevalier and Mayzlin found that an improvement in a book's reviews at one retailer raised its relative sales there.33 Chevalier and Mayzlin, Journal of Marketing Research 43(3), 2006, The Effect of Word of Mouth on Sales: Online Book Reviews Northwestern's Spiegel center reports that displaying reviews raised conversion by 190% for a lower-priced product and 380% for a higher-priced one.44 Medill Spiegel Research Center, Northwestern University, How Online Reviews Influence Sales

Nobody disputes this side. Ratings are economically powerful, and most powerful for independent businesses with no brand to lean on.

Do they match independent quality measures?

Here the best study is de Langhe, Fernbach and Lichtenstein's analysis of 1,272 products across 120 categories. They compared average user ratings with Consumer Reports scores, which come from laboratory testing. Their findings:55 de Langhe, Fernbach and Lichtenstein, Journal of Consumer Research 42(6), 2016, Navigating by the Stars: Investigating the Actual and Perceived Validity of Online User Ratings

  1. Average user ratings "lack convergence with Consumer Reports scores".
  2. They are "often based on insufficient sample sizes".
  3. They "do not predict resale prices in the used-product marketplace".
  4. They "are higher for more expensive products and premium brands, controlling for Consumer Reports scores".

The fourth finding deserves a moment. Holding lab-measured quality constant, people rated pricier and more prestigious products higher. Ratings absorbed brand image and price as if they were quality.

The authors then showed that consumers lean heavily on the average anyway, weighting it more than price or the number of ratings when judging quality. They called this trust an "illusion of validity".

Reconciling the two

How can ratings drive sales so strongly while tracking quality so loosely? Because the two questions have different bars. To move sales, a rating only has to be believed. To track quality, it has to be accurate, and accurate about the thing a lab or an expert would measure.

Reviewers and testers measure different things. A reviewer reports an experience over days or weeks: did it arrive, did it work, did it please. A lab measures durability, safety and performance under controlled conditions. A blender that feels solid and dies in month fourteen collects a year of five-star reviews first.

It would be wrong to conclude that ratings are worthless. They are reasonable measures of short-run satisfaction, and for many purchases that is most of what a buyer wants. The research finding is narrower: do not treat the average as a measurement of technical merit.

What about local services?

No one has run the Consumer Reports comparison for electricians, because there is no Consumer Reports for electricians. So what follows is inference, and we mark it as such.

The gap between experience and technical quality is plausibly wider for skilled services than for products. A customer can evaluate whether the roofer was punctual, tidy and pleasant. Few can evaluate the underlayment. Problems with skilled work often surface years later, long after the review was written.

The demand studies carry one relevant clue. Luca found that consumers "respond more strongly when a rating contains more information", including more reviews and reviews from users Yelp certified as "elite".11 Michael Luca, Harvard Business School Working Paper 12-016, Reviews, Reputation, and Revenue: The Case of Yelp.com Readers already try to weigh review quality, and they are right to. For a skilled service, the reviews that carry information about workmanship tend to come from repeat customers, from customers reporting back after time has passed, or from people with relevant knowledge. They are a minority, and they are worth searching for.

One more piece of evidence concerns marketplaces. Nosko and Tadelis, working with eBay data, argued that reputation scores are biased and that buyers judge the whole platform by single transactions. They built an alternative measure of seller quality from data not shown to users and found the platform could benefit by promoting sellers on that basis.66 Nosko and Tadelis, NBER Working Paper 20830, 2015, The Limits of Reputation in Platform Markets: An Empirical Analysis and Field Experiment If a platform's own hidden signals can beat its public ratings, the public ratings are leaving information on the table.

What to do with this

  • Use ratings to screen out, not to rank. A low rating from many reviews is a reliable warning. Among businesses that pass, the ordering by tenths of a star means little.
  • Match the evidence to the risk. For a $30 purchase, satisfaction ratings are enough. For a $30,000 job, they are the start of the research.
  • Look for reviews written late. "Two years on, still no leaks" is quality evidence. "Showed up on time!" is service evidence.
  • Distrust price-flattered ratings. Premium brands were rated higher than their lab scores justified.55 de Langhe, Fernbach and Lichtenstein, Journal of Consumer Research 42(6), 2016, Navigating by the Stars: Investigating the Actual and Perceived Validity of Online User Ratings An expensive provider's glowing reviews may partly reflect customers justifying what they spent.
  • Add an independent check where one exists. Licensing boards record discipline. Building departments record permits and failed inspections. These are closer to a lab test than any review.

Questions readers ask

Is a higher-rated restaurant actually better?

Reviewers liked it more, and it will probably be busier. Whether a trained critic would agree has not been established by the studies cited here, which measured demand, not culinary merit.

Why do expensive products get better ratings?

The study that found this did not settle the cause. Candidates include buyers rationalizing a costly purchase and brand reputation coloring the experience.55 de Langhe, Fernbach and Lichtenstein, Journal of Consumer Research 42(6), 2016, Navigating by the Stars: Investigating the Actual and Perceived Validity of Online User Ratings

Should I ignore ratings then?

No. Ignoring them throws away real information about customer experience. Read them for what they measure and verify the rest another way.

Method We separate two bodies of research that are often blurred: studies of whether ratings change demand, and studies of whether ratings match an independent measure of quality. See how we source and check.

Sources

Every source below was opened and read for this page. Links go to the publisher's own copy.

  1. Michael Luca, Harvard Business School Working Paper 12-016. Reviews, Reputation, and Revenue: The Case of Yelp.com
  2. Anderson and Magruder, The Economic Journal 122(563), 2012. Learning from the Crowd: Regression Discontinuity Estimates of the Effects of an Online Review Database
  3. Chevalier and Mayzlin, Journal of Marketing Research 43(3), 2006. The Effect of Word of Mouth on Sales: Online Book Reviews
  4. Medill Spiegel Research Center, Northwestern University. How Online Reviews Influence Sales
  5. de Langhe, Fernbach and Lichtenstein, Journal of Consumer Research 42(6), 2016. Navigating by the Stars: Investigating the Actual and Perceived Validity of Online User Ratings
  6. Nosko and Tadelis, NBER Working Paper 20830, 2015. The Limits of Reputation in Platform Markets: An Empirical Analysis and Field Experiment