Gartner Blog Network


“Dammit Jim, I’m a doctor not a bricklayer!”

by Toby Bell  |  April 6, 2009  |  3 Comments

Doctors don’t want to be rated by users of Angie’s List and I
don’t blame them. Hear me out, as I’m sure it appears at first
blush that I’m much more sympathetic to doctors than I really am.
Of course, maybe it’s because I still have lingering pain in the
backs of my knees and I may need a specialist someday. I’m
generally fearful of inviting retribution. Maybe generally,
therefore, blogging isn’t such a good idea. Too late to stop now, I
suppose. I digress.

Scoring mechanisms for Internet reputation management are in
their infancy. In an earlier post, I suggested that typical “5
Star” systems fail to cast effective light on quality of goods,
services, companies, or even people. Sharp readers contacted me
indirectly and called out Gartner’s Magic Quadrants, Marketscopes,
and Vendor Ratings as examples where scoring is fundamental to the
overall reputation of technology providers. “Why,” they wondered,
“do you think average people can’t effectively rate technology as
well as analysts do?” And, perhaps adopting the same tone of voice
as any doctor, I might reply “because we’re experts in our field”,
or better still, “because I said so.”

Actually, scoring for our branded documents like Magic Quadrants
is transparent, comprehensive, dynamic, appropriately triangulated,
and effective. The merits of the vendors and products are compared
and then clearly articulated. It’s a beautiful thing. Especially
when conducted by me personally at events like Gartner Portals,
Content & Collaboration Summit
June 8 – 10 2009 in Orlando,
FL at JW Marriott Grande Lakes. But we have yet to invite public
participation in Magic Quadrant scoring – though that day may come
soon enough.

This is because Reality TV and the Internet have stimulated
belief that just about anything could soon be turned into a talent
contest. Rather than hearing “This is… American (or Bolivian or
Armenian) Idol!”, expect someday to hear instead the annoying
introduction of yet another episode of “Crazy Brain Surgeons” or
“The Fastest Colonoscopy in Canada” or “Hand Surgery on Roller
Coasters”. The key point being that comparisons make the most sense
when they’re apples-to-apples rather than… um… apples to
orifices. Sorry for that.

So, Angie’s List is the first major Web player to effectively
pit plumbers against heart surgeons with public judging of
reputation on a single scale. Is an ‘A’ for one the same as an ‘A’
for the other?

Uh… no. Out of a scoring alphabet of 26 letters, Angie’s List
has chosen only three to represent the spectrum of difference
between every professional (and every layperson): Price, Quality,
Responsiveness, Punctuality, and Professionalism. Fact is, the only
things that doctors and plumbers have in common is that they get
dirty, fix leaky things, take pride in their technical competency,
and have an apprentice program that involves a serious investment
of time. They don’t like to have to explain every detail of what
they’re doing. They routinely deliver bad news. They respond in
emergencies. But doctors and plumbers (and technology industry
analysts on occasion) might all agree that Price or Punctuality may
not be the best basis for comparison.

Ideally, there would be a scoring mechanism unique to each
profession that fairly provides evidence of qualifications and
experience and references without bias. Sure, doctors could still
score doctors, but so could the medical media, membership
organizations, patients, and even ex-spouses. Oh. That’s the point
I was searching for: the Internet isn’t meaningful as pieces/parts,
but more as a collective with various scores tied to roles, rules,
and identity. But as a professional, it is unfair to opt-out just
as the world begins to tune in.

Some physicians are asking their patients to sign waivers
indicating they won’t provide their feedback on sites like Angie’s
List. Think about it: when was the last time you were asked to fill
out a survey about your experience – at any point – while a patient
in the US healthcare system? According to doctors I interviewed,
the only people qualified to evaluate any doctor are other doctors.
I wonder what basis they would use to score each other? I seriously
doubt they’d emphasize “bedside manner” and “relating to family
members in the waiting room with sincere empathy”. I suspect it
would have something to do with accurate diagnoses, high recovery
rates, teaching, advanced education, etc. I should have asked the
doctors these and other questions but I didn’t have “interview
insurance” and they were called away on “emergencies”. But what
would you expect to see after searching on Angie’s List for a heart
surgeon?

Okay… I figure I could save a little by using a doctor with
mostly ‘B’s. Maybe a Chevy instead of a Mercedes cardiac surgeon.
And, maybe since most doctors I know seem to have common goals in
dealing with patients generally, these attributes could be the
basis for a new approach to scoring:

I don’t want to appear to be overly critical considering I am
very likely to get sick in the future and wouldn’t want to be
tended to by my plumber. But the simple fact is that scoring
everything on the Internet depends on simple rules: 1) the sum of
the business or individual reputation should include more than one
source; 2) the less anonymity in scoring mechanisms the better –
with visibility comes responsibility; and 3) scoring mechanisms
have to evolve to more fairly reflect qualified perspectives of
both the supply and demand sides of any goods or services.
Understanding how doctors would score themselves – as colleagues of
each other as much as in the role of patients – could serve us
better in dealing with them.

I also think I might finally give my personal physician an ‘A’
for trying hard to keep me from needing to score a more urgent care
provider. Maybe one factor is how well a doctor focuses us on
resiliency rather than recovery – terms already familiar to many of
my colleagues in technology. So, it’s clear the way we score may
have to take into account bias, experience, ability, and other
factors that qualify us to qualify others. I don’t propose to
certify doctors – but I would certainly be more interested in the
opinions of those qualified to do so. There are dozens of such
scoring mechanisms already in place for medical practitioners.
Learning to use all the resources to effectively judge reputation
is at present more art than science. And, can be a very interesting
topic for further research. Watch for more here. Or add some as
comments.

Additional Resources

Category: somewhat-serious  

Tags: 5-stars  angies-list  bones  doctors  healthcare  pcc  ratings  reputation  scoring  star-trek  

Toby Bell
Former Research VP
8 years at Gartner
25 years IT industry

Toby Bell is a former vice president in Gartner Research, responsible for a range of document and content management technologies for the High Performance Workplace group. Some key areas of coverage include vendors and trends in the enterprise content management (ECM) marketplace… Read Full Bio


Thoughts on “Dammit Jim, I’m a doctor not a bricklayer!”


  1. Cheryl Reed says:

    Mr. Bell,

    Cheryl Reed here, from Angie’s List.

    Thanks for opening the discussion on online health care ratings. At Angie’s List, we strongly believe that the more information consumers have, the better off they are – regardless of the service they’re seeking — and the better off service professionals (including those in the health care industry) are, too.

    Consumer reviews are only part of the toolkit patients need to make the best health care decisions, and we link our members to independent, regulatory sites where clinical proficiency indicators can be found. But the overwhelming response from our members, and our 14 years of experience in consumer ratings, shows us that real consumer experiences and information sharing is of great value.

    Most consumers don’t claim to be able to determine which heart surgeon will deliver the best outcome or which oncologist will best be able to manage an individual’s cancer, but they can tell you all about their experience with that physician. Did he communicate well? Were you comfortable? Was the billing process difficult? And finally, what was the outcome? While the answers to these questions may not directly address physician quality, they certainly impact the patient in ways that can affect the outcome of their treatment.

    Another important distinction is the reliability of the information consumers can access. Data integrity is a key part of our service, and we’ve invested heavily to ensure our information is as accurate and reliable as possible. We constantly monitor for attempts to stack the deck for or against any professionals; we alert service professionals to report and encourage their response and active participation in the dialogue.

    Consumers have more responsibility and direct control over their health care decisions today than in years past. They don’t just want information ahead of time: they deserve to be able to easily access it. As one member recently wrote: “Just because a doctor is a jerk doesn’t mean he isn’t a great oncologist, and when push comes to shove, I’ll take the healing powers over easy small talk… But it also doesn’t mean that I don’t want a head’s up before I go to the office that he’s a jerk.”

    Our members grade their experience with professionals on an A-F scale (so we actually use 5 of the 26 letters) but also offer rich detail about their experience. That combination of grades and stories helps consumers know something about the professional before they walk into the office. Taken in context with other ratings mechanisms, consumers have more information today than they did even a year ago. Hopefully, they’ll have dramatically more information in the future.

    At Angie’s List, we’re continually refining our service and will be enhancing the health care ratings very soon. We hope you’ll check on the changes and continue this important debate.

  2. Toby Bell says:

    Dear Cheryl,

    I appreciate your response and agree that this is dialogue worth having – and that Angie’s List is in no way at fault for providing scoring mechanisms extended toward consumers of any type of service. Just as I don’t blame doctors for not wanting to ‘opt in’, I don’t blame users for wanting detailed opinions related to others’ experiences. I will definitely check on upcoming changes to ratings of health care providers both as an analyst and as a user of your site. Which I recommend with 5 stars.

    The most interesting dimension of reputation scoring is what constitutes ‘local knowledge’. I’ve moved 7 times in 7 years and now live somewhere I share with no family or friends. Thus, when banking or buying a car or fixing a leak, I can’t count on Mom or Dad or cousin so-and-so to tell me which provider is best. Thus far, Angie’s List has become a key strategic resource across 4 of those recent “home towns” and unfailingly has been accurate in delivering positive experiences with providers recommended by others. Angie’s List should host pot luck dinners in each location so I can meet the nice folks in person who’ve set me on the right path so often. Just don’t score my casserole.

  3. […] higher overall score from reviewers. (Rotten Tomatoes, by the way, routinely delivers value with a reputation scoring system worthy of 100 on the Tomatometer itself… and of being studied by every other Web community […]



Comments are closed

Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.