Wes Rishel

A member of the Gartner Blog Network

Wes Rishel
VP Distinguished Analyst
12 years at Gartner
45 years IT industry

Wes Rishel is a vice president and distinguished analyst in Gartner's healthcare provider research practice. He covers electronic medical records, interoperability, health information exchanges and the underlying technologies of healthcare IT, including application integration and standards. Read Full Bio

Coverage Areas:

Usability Standards for EHRs

by Wes Rishel  |  July 17, 2011  |  3 Comments

Mark Frisse started an interesting thread on Google+ today based on Assessing the Effect of Standards in Digital Health Records on Innovation by Steve Lohr. He chose not to make it public so I can’t cite it directly here.

I now face the typical Google-plusser dilemma. I want to add to the thread but I also want my thought to be public. So I am posting the blog entry and will cite it there.

There is not yet enough science to determine the one right way to do user interfaces. I doubt that there is enough science to objectively conclude trade-offs between, for example, being able to see everything at once that is on a printed cover sheet vs. using smaller or less costly devices. (It would be easy to see a standard based solely on usability principles proscribing any devices other than wall-mounted HD screens with special glasses needed to prohibit accidental viewing by non-authorized people.) I *AM NOT* making light of usability principles; I am simply pointing out that they represent one set of factors in a trade-off in selecting an EHR.

If we do have some science now,  it should go into lifting the veil of obscurity that vendors place around their usability.

USABILITY MEASURES
I propose the notion of a “usability measure.” Like a “quality measure” it would be specific and might have specific exclusions. In order to be a “measurable measure” it could not usually be generic to the entire EHR. While “all references to a patient should include gender, age and unit number” might well be a good principle, the measure for this principle might enumerate four or five situations where the patient information is displayed such as patient selection, results lists, individual results viewing, ordering and documentation and NIST criteria would include scripts for determining yes-no answers in those situations.

Experience with clinical measures has shown that going from generally agreed-on principles to computable measures is a substantial investment of time.

A public dialogue with minority representation from the vendor community should review proposed measures and categorize them into these groups:

1) So obvious it would be unethical not require this of all EHRs. The criteria for inclusion in this group should be very high.

2) Likely to be important factors; submission to the measure should be required for EHR certification and the average across all submissions be published. If a certified product is purchased by a health delivery organization and the vendor failed to provide its own scores before the HDO was committed to the product, that HDO should be ineligible for receiving the benefit of meaningful use incentives (bonuses now, relief from penalties in the future).

3) Candidate measures; submission to the measure should be required for EHR certification but there is no requirement for EHR vendors to provide their scores. Anonymous scores would be published so that healthcare industry could see the spread of how products performed on the measure.

4) Worthy of further study; the measures should be published as fodder for study, no scripts are prepared and testing does not occur as a part of certification.

OTHER REGULATORY ACTIONS
Given threats to patient safety based on usability, any EHR that is purchased under a contract that includes a “usability gag clause” should be ineligible to benefit under the incentive program. Such a clause would penalize the client for speaking publicly about the usability of the product. Vendors have justifiable concerns about a minority of irresponsible who will make up or exaggerate such issues in order to put pressure on the vendor on some unrelated point. They also have justifiable concerns about “Internet warriors” who take every public dialogue as a no-holds-barred battle for attention and dominance. Nonetheless, the greater good for healthcare and patients is achieved by favoring transparency. All of us are Internet users and we are learning to glean the gems from the crud.

3 Comments »

Category: Healthcare Providers     Tags: , ,

3 responses so far ↓

  • 1 Mark Frisse   July 17, 2011 at 2:21 pm

    Wes,

    I am flattered that my brief posting on the NY Times Lohr piece stimulated such a wonderful exposition of the issues. A few years ago, I was fairly concerns about certification and standards processes in those early days. I am actually heartened by the progress, but believe the very significant concerns raised by the CSTB report and the impediments faced by practitioners adopting EHRs must be addressed.

    My only point was to ensure people distinguished between what is really important to care and what is merely interesting. These things have to evolve.

    I did provide the direct links to some of the references made in the NY Times piece.

    ACM Letter:
    http://usacm.acm.org/images/documents/USACM%20Letter%20Usability%20EMRfinal.pdf

    ONC Standards Committee Slides:
    http://healthit.hhs.gov/portal/server.pt/gateway/PTARGS_0_12811_954625_0_0_18/hitpc-cawg-usability-hearing-report-05-11-11.ppt

    AMA’s concern.
    http://www.ama-assn.org/resources/doc/hit/ama-ehr-usability-statement.pdf

    CSTB report:
    http://www.nlm.nih.gov/pubs/reports/comptech_prepub.pdf

    By legislative mandate, ONC has funded research in this area that complements the broad array of research efforts alluded to in the ACM letter. ONC-funded SHARP research is led by Jiajie Zhang at the University of Texas Health Science Center at Houston.
    http://www.uthouston.edu/nccd/

  • 2 Margalit Gur-Arie   July 17, 2011 at 7:01 pm

    Wes & Mark,

    If I may respectfully contribute my 2 cents here, I would like to submit that the usability debate seems to be suffering from slight confusion of terminology.
    The pain points that are cited in the NYT article and elsewhere pertain to patient safety, interoperability and general usefulness of the EHR. Perhaps usability standards can partially address all of the above, although I believe this is a major stretch of definitions, but they cannot fully address all three issues, and I doubt the value of bundling all these concerns under the term “usability”.

    I do agree with Wes that the science is not completely there, and in view of this, I would prefer that in a world of finite resources, and finite time frames, we spend our efforts and money on things where the science is fully there: interoperability standards from ONC and patient safety evaluation/approval from the FDA. Put together, these two should increase usefulness of EHRs, which is different than usability.

    I would also like to mention that, to my knowledge, studies have identified quite different impediments to EHR adoption – financial expenditures and tangible ROI – http://jamia.bmj.com/content/18/3/271.full.pdf
    I would be interested in a major study substantiating Dr. Shortliffe’s statement in the NYT.

  • 3 QA   July 21, 2011 at 12:50 pm

    Perhaps Margalit’s confusion arrises from the fact that there are many dimensions to usability. Or her lack of background in human factors.

    The science around measuring and evaluating various dimensions of usability is well established and has been applied in many important ways. Ways that have saved lives. That EHRs or this industry is too unique, complicated, exceptional, innovative. etc. for evaluation or improvement using established methods and techniques is hogwash.

    One aspect of usability / human factors that is quite mature – to the point that it’s the basis for a variety of regulatory programs – is critical aspects of usability related to safety.

    Before medical devices / medical device software are permitted to enter the market, the FDA requires that vendors of these systems demonstrate – through summative testing – that a representative sample of users of their systems will not make critical “use errors”. In addition to minimizing risk associated with electrocution, chemical spills and other hazards, the FDA wants to ensure that the design of medical devices / medical device software don’t contribute to harm.

    This is, of course, very different than the FDA designing medical devices or software. It’s about ensuring that vendors understand and mitigate risk of their products (“use error” and otherwise) prior to putting products on the market. And has resulted in manufacturers of products including human factors considerations much earlier in the design of their products.

    A convenient side effect is that following user-centered-design principles in the design of products tends to make them not only safer but more efficient, satisfying and effective.

    I couldn’t agree more that there is currently a lack of transparency around important aspects of usability of systems, including subjective ones (like whether users are satisfied).

    In my opinion, the market – and not the government – should demand this. Very, very few purchasers of EHR systems do a good job of incorporating human factors considerations into their acquisition processes.

    I also like the idea of disqualifying vendors that do not allow their customers to openly share incident information. Without sunlight, we are doomed to the systems that we have today.