In college, I devoured a book called Zen and the Art of Motorcycle Maintenance.
I know I’m dating myself here. But, the book reappears on college campuses every decade. The book delves into what defines quality. The protagonist’s philosophical investigations drove him insane, and he was ultimately treated with electroconvulsive therapy, which permanently changed his personality. Still, the quest to define quality remains a worthy goal, if elusive.
The online world has become the sounding board for consumer’s perceptions of quality. A vote up or down on the Internet does correlate with whether a business will make more revenue – or less. Researchers studied the effect of Yelp ratings on revenue from restaurants in Washington. They concluded that each ratings star added on a Yelp review translated to a 5 percent to 9 percent effect on revenues.
What about healthcare?
We were early skeptics that online reputation correlated with quality in healthcare. Many years ago, we believed that online reviews were useful, at best, for addressing the soft, subjective factors related to the patient experience – for example, did the doctor communicate well, was his bedside manner adequate, how long did the patient have to wait for an appointment, etc.
We assumed that patient reviews were not helpful for addressing the more important factors of patient care – namely safety and clinical outcomes. It doesn’t matter if the doctor communicates well if his patients are dying because of negligent care.
So, we studied it.
One the data was in, we concluded we were wrong.
Our group was the first to survey and connect online reputation with patient safety and quality of care. Segal J, Sacopulos M, Sheets V, Thurston I, Brooks K, Puccia R. Online Doctor Reviews: Do They Track Surgeon Volume, a Proxy for Quality of Care? J Med Internet Res 2012;14(2):e50
So, we were the first to provide empiric data suggesting online reputation was helpful in guiding patients on metrics that mattered the most, proxies for quality of care.
Which brings us to a collection of provocative articles recently published in the Journal of Consumer Research. I’m positive that journal is on everyone’s must-read list.
The flagship article, Navigating by the Stars: Investigating the Actual and Perceived Validity of Online User Ratings, compared quality measures for over 1,000 consumer products. The head to head comparisons were based on Consumer Reports and Amazon reviews. In the former, experts tested the consumer products using sophisticated measures and tools unavailable to most consumers. In the latter, consumers weighed in with their subjective take on the products.
The authors concluded average user ratings from Amazon:
- lacked convergence withConsumer Reports scores, the most commonly used measure of objective quality in the consumer behavior literature;
- are often based on insufficient sample sizes;
- do not predict resale prices in the used-product marketplace; and
- are higher for more expensive products and premium brands, controlling forConsumer Reports
“…when forming quality inferences and purchase intentions, consumers heavily weight the average rating compared to other cues for quality like price and the number of ratings. They also fail to moderate their reliance on the average user rating as a function of sample size sufficiency. Consumers’ trust in the average user rating as a cue for objective quality appears to be based on an “illusion of validity.”
Other papers in the same issue pushed back on the findings. One group said, so what, what does it matter? Quality is in the mind of the beholder. If the consumer believes a product delivers quality, so be it.
That same group pushed back arguing that helping consumers achieve their goals with uncorrelated data may be useful. One analogy. If my goal is to preserve wealth and grow my savings, I want a well-diversified portfolio exposed to different risks. The trajectory of stocks and bonds do not often correlate. I want investments that do not correlate. So, having a basket of both gets closer to achieving my goal of mitigating risk and preserving wealth. If I had all my eggs in one basket, I might grow my savings more, but at the expense of more risk.
The original authors responded that objective quality does matter, and may even trump subjective sentiment. They provide two examples:
- Beats Headphones. Apple purchased Beats, the high-end headphone company, in 2014for $3B. On Amazon, Beats has 4 star out of 5 star ratings with hundreds of reviews. Consumers seem to love the product. But, hardware engineer Avery Louie did a teardown analysis. He found that internal screws (which would add to production cost) were shunned in place of less expensive, and less durable snaps and plastic fasteners. More interestingly, Beats adds nonfunctional, heavy pieces of zinc to the headphones, potentially to give consumers the impression the construction is more solid than it really is. Consumer Reports ranks Beats as mediocre in quality and a bad deal at its premium price.
- Baby seats. While consumers can provide feedback, what matters most to purchasers? That the seat will protect their infant if the car is in an accident. The number of reviewers who can comment on that from experience is limited. But, Consumer Reports does have sophisticated equipment to rank one seat from another.
So, there you have it. Now I know why the protagonist in Zen and the Art of Motorcycle Maintenance lost his mind deconstructing quality.
Still, here are some take home points:
- If you are a physician, being defined by large number of online reviews makes it more likely you are fairly represented. You do not want to be defined solely by two noisy patients with a megaphone. The solution to pollution is dilution.
- Online reputation does correlate with revenue – in healthcare that means new patient volume.
- Online reviews are strong marketing signals; trusted by patients.
- Online reviews validate a patient’s decision to follow-through on a referral from friend, family, or another doctor.
- And our work still stands, for surgeons, online reputation correlates with proxies for objective quality of care.
In healthcare and other domains, online reputation does matter.