Ratings can't be trusted

Other View vs Mainstream View

There is a non-mainstream point of view that "Ratings can't be trusted". Let's take a look into pros and cons of this statement:

1. Pros: Ratings can be manipulated by companies, interest groups, or biased individuals who artificially boost or lower scores, leaving people with a skewed perception of the true quality or satisfaction level.
   – Cons (Mainstream View): Most well-established rating systems have safeguards and methodologies in place to detect and prevent manipulation. The large sample size of genuine ratings tends to average out and minimize the statistical impact of a small number of biased reviews.

2. Pros: Ratings are inherently subjective and what one person enjoys or finds valuable, another may dislike or find lacking. An aggregate score doesn't tell you if you personally will enjoy the product, service, or experience based on your specific needs, tastes, and preferences.
   – Cons (Mainstream View): While not perfect, ratings provide a useful overall gauge of quality, performance, and satisfaction when taken in aggregate. Examining a representative sample of individual reviews or scores gives a more nuanced and well-rounded picture of common pros, cons, and experiences.

3. Pros: Ratings encourage companies and organizations to game the system and optimize for metrics and scores rather than focusing purely on delivering the best product, service, or experience. This can lead to perverse incentives and prioritizing the appearance of quality over actual quality.
   – Cons (Mainstream View): Companies and organizations have a strong incentive to get good ratings because it drives more business and enhances their reputation. Focusing on customer satisfaction to earn genuinely positive ratings tends to result in better offerings and experiences in the long run. Consistently low ratings will hurt the bottom line.

4. Pros: Selection bias means people who provide ratings are not necessarily representative of the entire customer or user base. People who had polarizing experiences (either very positive or very negative) are more likely to leave reviews than those who found it simply average or as expected.
   – Cons (Mainstream View): While extremes may be overrepresented, a sufficiently large sample size of ratings will include a broad spectrum of experiences and opinions. Shortcomings of selection bias can be mitigated by looking at the overall distribution of ratings and examining a diverse cross-section of reviews.

5. Pros: Ratings are not always a reliable predictor of future experiences because companies, products, and services can change in quality over time. What was true at the time of past ratings may no longer be accurate.
   – Cons (Mainstream View): Ratings are most useful as a directional guide and a snapshot in time based on recent experiences. Savvy consumers and decision makers look at the age of reviews and focus more on ratings and feedback from the past few months which best reflect the current state and trajectory of quality.

Terms

• Manipulation: The act of artificially boosting, lowering, or fabricating ratings and reviews through deceptive tactics like fake accounts, biased reviews, or cherry-picked data in order to mislead people and distort perceptions.

• Subjective: Based on or influenced by personal feelings, tastes, opinions, and preferences rather than objective facts and measurable attributes. Subjective evaluations can vary significantly from person to person.

• Aggregate score: The overall rating calculated from compiling and averaging many individual rating scores. It provides a high-level summary measure but lacks the nuance and context of individual reviews.

• Game the system: Exploiting loopholes, biases, or weaknesses in a system's rules, methodologies or incentive structure for gain and benefit rather than the intended purpose. Gaming ratings undermines their accuracy and reliability.

• Selection bias: A skew or distortion in ratings and reviews because the people who choose to provide feedback are not a truly random or representative sample of all users or customers. Certain segments are over or underrepresented.

Analogy

Saying ratings can't be trusted is like saying you can't trust weather forecasts because meteorologists could make mistakes, models could be flawed, or conditions can change unexpectedly. While you should maintain a healthy skepticism, weather forecasts can still provide useful information and insights directionally, especially in the short-term and when you look at multiple models. They give you a rough sense of what's likely or probable even if not a guarantee.

For example, if a movie has a 90% fresh rating from critics and a 95% audience score with tens of thousands of reviews, it's a pretty reliable signal that the movie is quite good, even if a few of the reviewers have biases or ulterior motives, and even if you account for the fact that a small percentage of people who disliked it may not have bothered to leave a negative review. You can be reasonably confident it's a good bet, but not 100% sure you'll enjoy the film.

History

1. 1905 - University of Missouri professor Max Meyer develops the first rating scale for judging the merit of food products

2. 1916 - The Michelin Guide begins awarding stars for fine dining restaurants

3. 1930s - Several market research and polling companies like Gallup and Nielsen emerge to gauge consumer opinions and behaviors

4. 1950s - Psychologist Rensis Likert develops the 5-point Likert scale which becomes widely adopted for surveys and ratings

5. 1968 - The Motion Picture Association of America introduces the film rating system to guide parents on age-appropriate content

6. 1979 - Zagat Survey launches, aggregating restaurant ratings and reviews from diners and popularizing the 30-point scale

7. 1990s - Consumer Reports expands its product testing and ratings, becoming highly influential in shaping buyer decisions

8. 2000s - With the rise of the internet, online ratings and reviews on sites like Amazon, Yelp, and TripAdvisor explode in popularity and importance, impacting almost every industry and domain

How to use it

1. When evaluating a high-stakes decision like choosing a doctor, college, or investment, look at the overall rating but also examine a representative sample of detailed reviews or testimonials to get a nuanced picture. Look for common patterns in praises and criticisms. Compare and cross-reference ratings and reviews across multiple trusted sources. Pay attention to the credibility and expertise of reviewers. Disregard outliers and extreme opinions without substantiation.

2. When choosing a restaurant, hotel, or local service provider, put significant weight on the aggregate rating but focus on the most recent reviews from the past few months. Quickly skim to get a sense of the key decision factors like quality, value, service, atmosphere, and what signature offerings people highlight. Make sure to factor in your specific needs and preferences. A 4-star hotel could be perfect for a budget-conscious traveler but underwhelming for a luxury trip.

3. When evaluating something subjective like art, fashion, books, or entertainment, acknowledge that ratings largely reflect group consensus and popular opinion, but not necessarily what you as an individual will enjoy. Spend more time examining a diverse set of critic and audience reviews to get a qualitative sense of the style, content, tone, and craftsmanship. See if the descriptions and reactions resonate with your personal tastes and sensibilities. Look for reviewers whose opinions have aligned with yours in the past. Accept that you might love something generally panned or feel lukewarm about a critical darling.

Facts

• Over 90% of people say ratings and reviews impact their purchase decisions

• 4.2 to 4.5 stars is the ideal average rating for purchase probability. Anything higher seems too good to be true.

• The average Metacritic score for movies is 57 out of 100 and for video games is 71 out of 100

• Customers are willing to spend 31% more on a business with excellent reviews

• A one-star increase in Yelp rating leads to a 5-9% increase in revenue for restaurants