RATINGS/DECODED

Math / WORKED EXAMPLE

Why 4.8 from 12 reviews is weaker evidence than 4.6 from 900

Because the uncertainty around an average shrinks with the square root of the number of reviews. In this worked example, the 12-review business has a 95% range of about 4.4 to 5.0, wide enough to include the other business's entire range of 4.53 to 4.65. The lower number is the better-supported one.

Two electricians. One shows 4.8 stars from 12 reviews, the other 4.6 from 900. Most people pick the first. Here is the calculation that says not so fast.

4.8 12 reviews 4.6 900 reviews

The two sets of reviews

We need the individual ratings, not just the averages, so here are two plausible sets.

5 stars4 stars3 stars2 stars1 starTotalMean
Business A101100124.75, displayed as 4.8
Business B6901204518279004.587, displayed as 4.6

Check the means. A: (10 × 5 + 4 + 3) ÷ 12 = 57 ÷ 12 = 4.75. B: (3,450 + 480 + 135 + 36 + 27) ÷ 900 = 4,128 ÷ 900 = 4.587.

Step 1: how spread out are the ratings?

The standard deviation measures how far individual ratings sit from the mean.

  • Business A: 0.62 stars. Nearly everyone gave five.
  • Business B: 0.91 stars. A long tail of unhappy customers widens the spread.

On spread alone, A looks more consistent. Hold that thought.

Step 2: how uncertain is the average?

The standard error of the mean is the spread divided by the square root of the count.

  • Business A: 0.62 ÷ √12 = 0.62 ÷ 3.46 = 0.18
  • Business B: 0.91 ÷ √900 = 0.91 ÷ 30 = 0.03

B's average is six times more precisely known, even though B's customers disagree with each other more. Count beats consistency.

Step 3: the 95% range

Multiply the standard error by about 2 to get a range that would contain the long-run average 95 times in 100. For 12 reviews the correct multiplier is 2.20, a little larger, because small samples also misjudge their own spread. For 900 it is 1.96.

  • Business A: 4.75 ± 2.20 × 0.18 = 4.75 ± 0.39 → 4.36 to 5.14, which the scale caps at 5.0
  • Business B: 4.587 ± 1.96 × 0.03 = 4.587 ± 0.06 → 4.53 to 4.65
4.24.334.474.64.734.875A: 4.8 from 124.36 to 5.0B: 4.6 from 9004.53 to 4.65Plausible range for the long-run average rating (95%)
Dots mark the observed means. Bars mark the 95% ranges computed above; A's upper end is cut off at the top of the scale.

Reading the picture

B's whole range fits inside A's. Everything we know about B is compatible with what we know about A. The data cannot tell us A is better.

Look at the lower ends, which is where the risk lives. A could be a 4.36 business having a lucky first dozen. B is at least a 4.53. If you want to avoid the worse outcome, B has the higher floor.

A might truly be a 4.9 business. Nothing here says otherwise. The point is narrower: twelve reviews have not shown it yet.

One more review

A quick way to feel the difference is to add a single one-star review to each.

  • Business A: (57 + 1) ÷ 13 = 4.46. The display drops from 4.8 to 4.5.
  • Business B: (4,128 + 1) ÷ 901 = 4.583. The display stays at 4.6.

A's rating is one unhappy customer away from sitting below B's. A number that fragile is not yet a measurement.

What about the assumption that reviews are a random sample?

They are not, and this calculation does not fix that. Reviewers choose themselves, and platforms filter. The interval above captures only chance variation. If Business A's twelve reviewers are the owner's friends, the true uncertainty is far larger than the arithmetic shows, and in one direction.

So treat these ranges as the minimum uncertainty. Real uncertainty is this plus whatever bias came from how the reviews were gathered, which is why the research finding that consumers ignore sample size is troubling: people skip even the part that can be calculated.11 de Langhe, Fernbach and Lichtenstein, Journal of Consumer Research 42(6), 2016, Navigating by the Stars: Investigating the Actual and Perceived Validity of Online User Ratings

A platform that does this for you

Trustpilot's TrustScore includes seven neutral 3.5-star reviews in every calculation.22 Trustpilot Help Center, TrustScore and star rating explained Apply that to our two businesses, ignoring its recency weighting:

  • Business A: (7 × 3.5 + 57) ÷ 19 = 81.5 ÷ 19 = 4.29
  • Business B: (7 × 3.5 + 4,128) ÷ 907 = 4,152.5 ÷ 907 = 4.58

Under that formula B outranks A by a clear margin. On a plain-average platform such as Google, A is shown ahead.33 Google Business Profile Help, Understand review scores for local places and businesses Same reviews, opposite ordering, and both platforms are following their published rules.

Questions readers ask

At what number of reviews would the 4.8 become the stronger rating?

If Business A kept the same mix of ratings, its 95% range would tighten to roughly 4.63 to 4.87 at 100 reviews. At that point its lower end sits above nearly all of B's range and you could fairly call it the better-rated business.

Does this mean big businesses always win?

No. It means small samples cannot prove superiority yet. A small business with a real 4.9 will show it as reviews accumulate.

Why use 2.20 for the small sample?

With few observations the estimate of spread is itself shaky, so statisticians use a t multiplier, which for 12 observations is 2.20 instead of 1.96.

Method Both review sets are invented for illustration and are stated in full so every figure can be recomputed. The method is the standard error of a mean; the small sample uses a t multiplier. See how we source and check.

Sources

Every source below was opened and read for this page. Links go to the publisher's own copy.

  1. de Langhe, Fernbach and Lichtenstein, Journal of Consumer Research 42(6), 2016. Navigating by the Stars: Investigating the Actual and Perceived Validity of Online User Ratings
  2. Trustpilot Help Center. TrustScore and star rating explained
  3. Google Business Profile Help. Understand review scores for local places and businesses