Gartner Blog Network


Practical Concerns re: Certification Recommendation

by Wes Rishel  |  July 19, 2009  |  9 Comments

On 16 July the Certification and Adoption Workgroup (CAWG) of the U.S. HIT Policy Committee provided its recommendation to the full committee.  It calls for a 3-level approach to certification. There will be an opportunity to comment and I hope that readers of this blog will consider the thoughts here and then comment as they see fit. (Please see a subsequent clarification based on early feedback from blog readers at More on The EHR 3-Level Certification Approach.)

The current recommendation is

  1. ONC itself determine criteria for certification
  2. ONC identify an organization to accredit various certifying organizations
  3. As many organizations as feel there is a business in doing so become accredited and begin certification.

A corollary recommendation is that vendors should only need to be certified by one organization.This implies that all Federal programs and regulations that require certification would accept certification from any accredited organization. The main focus right now is the incentive payments but one must presume it applies equally to grant programs that provide assistance in implementing EHRs. Since Federal money contributes to many state programs they would also be bound, it seems. The bottom line is that vendors and CDOs that want to qualify with self-developed EHRs would each be free to pick one accredited certifier. The entities that are relying on certification to determine whether to give funds would not be able to say that “we feel that Sleaze Bag Certifiers, Inc.” is mainly in the business of helping vendors find loopholes and “Upright Certifiers, Inc” is diligently applying the criteria in a way that ensures we know what we are getting. They would have to accept certs from SBCI and UCI indiscriminately.  When the choice of certifiers is made only the vendor, not the organization that relies on the certificate, this creates an inevitable pressure to be the certifying organization that is the least thorough in its process.

At worst this guts the idea of certification. At best it creates a requirement that ONC construct a set of requirements for the accrediting organization that assures it not only pre-qualifies certifying organizations but also monitors their actions and provides a dispute mechanism for those that feel that a given certifier is too lax or too strict.

There have been those lobbying against the Bush-administration approach, which was embodied in CCHIT. Whatever the merits of the argument, it seems unlikely that a government agency can promote the notion of industry self-regulation given what has been learned from the financial services industry in the last year. Therefore, CCHIT has a very steep hill to climb if it to retain its status as the public consensus body for determining requirements.

Even so, there is a lot of valuable experience that can inform the specific adoption of the CAWG recommendations. I have worked at all three levels of the CCHIT workgroup, commissioner, and trustee. In my day job I am continually evaluating vendors and vendor claims, sorting out the information from the noise, at a company that became a billion-dollar company because users trusted our advice on vendors. Therefore I put myself forward as someone who has a good understanding of the process and a balanced view of vendors, neither demonizing nor glorifying them. I hope you will consider the material below in that light as you make your own decisions on commenting.

Questions to Consider

I have organized this into questions to consider and thoughts that might help. As always comments in the form of additional questions and further considerations are very, very welcome.

Why do we need certification at all, since we now have meaningful use criteria? In other words why not let physicians and small hospitals buy whatever product or build and assemble whatever combination of open-source code, commercial products and home-brew that makes sense. Either they qualify for meaningful use or they fail. They can reply on their trade associations and organizations that survey satisfied customers to give them guidance on what to choose.

The short answer is that the law requires certification for several big programs. The equivalent question in this context is should ONC minimize  the certification requirements and rely primarily on meaningful use going forward?This gives the marketplace the most opportunity to respond innovatively to the meaningful use measures, in other words to achieve the “outcome” rather than follow a prescribed approach.

This is a very attractive proposition. But it does mean that a practice or hospital buying a product now (and let’s face it, most of them will buy products rather than build) must correctly judge whether it does or will grow to meet the evolving meaningful use requirements in 2015. Customer surveys, whether commercial or conducted by specialist associations, primarily ask the question “are you satisfied with your product and the vendor that sold it?” They will no doubt have collected and published great data by 2016 on the ability to mean the 2015 meaningful use requirements, but they aren’t that much help to a CDO making a purchase decision in 2011 or 2013. This element of risk will contribute to a trend to defer purchases, even at the cost of giving up some incentive funds.

Is it possible to certify products to give solid buying advice to physicians? After all, CCHIT has failed to set advanced criteria for aiding the cognition of physicians, evaluate the quality and safety of user interfaces, or to evaluate the fiscal stability of vendors.

When CCHIT started there were about 200 EMR products that could be identified by scanning the Web, ads in trade magazines, etc. Currently about 80 of them are certified. To me this indicates that CCHIT has done a good job of finding the middle course between setting criteria that physicians won’t buy or can’t implement and being completely laissez faire. For example, all the so-called EMRs that were primarily systems for storing and retrieving scanned documents have either modernized or failed to be certified. Systems that only accept medication orders as text have similarly had to evolve. Systems that cannot record problems and allergies, and use med allergies in med orders have been eliminated.

No certifying organization can assess all the criteria that go into choosing a product.  For example:

  • Financial Viability. There are difficult issues in certifying financial viability, since almost no startup looks viable by any objective measure– or at least most people don’t think that startups should have to be sufficiently capitalized to look viable. CCHIT made a compromise: it allowed firms to provide some basic information about the size of their customer base and made it apparent when this information was not provided. Furthermore, it followed up with vendors that had qualified about every 90 days to see if they were still in business and answering their phone.  Would this be adequate, insufficient or overkill in the future? That would be up to ONC to decide in setting criteria.
  •  Quality and Safety. The quality and safety of user interfaces is subjective. Arguably one could establish some objective measures of what makes a “bad” or “unsafe” UI, but this would be incomplete and the converse of these measures would not define a “good” or “safe” UI. The experience building consensus on clinical measures has shown that even when the conceptual ideas can be easily accepted, nailing them down to the point of being objectively measured in real-world systems is a long process. It would be a substantial value if ONC were to start the long-term process now of building consensus on objectively measurable system characteristics that are unsafe. Perhaps it should kick this off with the Institute of Medicine. It seems unlikely that the consensus could be developed and reduced to practice in certification much before the 2013 round of criteria.
  • Cognitive Assistance.Advanced clinical support and point-of-care access to medical knowledge is an area where CCHIT has been criticized for being too strict and not strict enough. Vendors are often accused of watering this down because they can’t do it or beefing it up to try to restrict new entrances in the market. Having seen this debate in CCHIT I can say that the vendors tended to work to keep criteria matched with what they thought physicians would be willing to pay for. The pressure to raise criteria came from other stakeholders with righteous concern about safely and quality, hoping to use certification as a way to force physicians to practice better medicine. This is an area where any certification program should be closely tied to meaningful use criteria or other elements of healthcare reform. It is unlikely that the majority of physicians will accept these features unless they see near-term, highly credible pain if they don’t.

Should the accrediting organization evaluate howa certifier determines compliance or only whether it does? This 3-level approach is straight out of the playbook of ISO 9000, the capability and maturity model and the Common Criteria for security. It will be interesting to see what happens there. I am going to guess, however, that it is impractical to determine in advance whether a candidate certifying organization can be effective without reviewing its methods. 

In adopting the 3-level approach there are three strategic approaches. ONC can (1) leave the determination of methodology to each separate certification organizations; (2) strike out to establish a brand new approach to certification to be applied uniformly by all certifications; or (3) adapt the approach used by CCHIT for use by all certification organizations.

As discussed above, the 3-level approach will create market forces driving certification organizations to be the least intrusive on vendors. Arguably this means that the first approach, leaving the certification process to the certifying organizations, is equivalent to reducing the impact of certification to the legal minimum, and relying almost entirely on meaningful use.

The second and third approaches, ignoring the CCHIT approach or adopting it, represent the endpoints of a spectrum of approaches. Intermediate points on the spectrum arise based on how much attention is given to CCHIT experience. I strongly advocate paying close attention to the CCHIT experience with regard to certification because it achieved several important balances:

  • Criterion objectivity. Several rounds of certification have led CCHIT to a set of criteria that can be rated by judges with good inter-rater reliability. This is a value that the accrediting organization should consider in evaluating the methodology of organizations.
  • Criterion testability. The devil is really in the details in objectively testing functionality. This will be the biggest challenge for the 3-level approach, because it involves a give and take between the determination of criteria, the testing approach and the test scripts. In adopting the three-level approach the ONC should not consider how the HIT Standards Committee, or whoever else determines testing criteria should be able to participate on a quick-response basis as the certifying scripts that support a criterion are developed and executed. CCHIT also used a “beta certification.” Vendors could participate without charge and without their names being released to the pubic. This allowed the final scripts to be tuned up by experience before being put into play.
  • Certification costs.CCHIT adopted a budget of only testing as much as can be evaluated in a single day of testing (for ambulatory EHRs). This involves complex trade-offs between the selection of criteria and the methods of certification. It will also be more complicated in the 3-step approach.

Should the ONC reconsider the 3-level approach? I perceive that CCHIT actually achieved a balance between three different groups of stakeholders, those that must build to the requirements and sell that product to CDOs and physicians; the purchasers of products who are primarily focused on the short-term impacts directly oh themselves; and other stakeholders that want to ensure that funding EHRs leads to better and less expensive care. The closer observers were to seeing the process the more they will agree with me. However not everyone shares that view. I would urge those who write comments to consider this issue on their own, keeping in mind that if the outcome of establishing criteria is truly balanced, nobody will like all of it.

I would further argue that CCHIT met unbelievable deadlines, launching itself in a year and going through annual updates to its criteria.

Because of the current general public concern about industry self-regulation I doubt that the current governance of CCHIT will be acceptable. A separate question is whether the full bureaucracy of the 3-level process is needed,given the issues discussed above. I strongly suggest that commenters give consideration to whether a group such as CCHIT with a different approach to governance wouldn’t a better approach. Commenters might want to give some thought to how a governance approach can avoid the problems to the usual government “waterfall” process where requirements are developed by one entity and then become sacrosanct to the entities that need to figure out how to make them work in certification. (My clarification at More on The EHR 3-Level Certification Approach is particularly relevant to this section.)

Category: 

Wes Rishel
VP Distinguished Analyst
12 years at Gartner
45 years IT industry

Wes Rishel is a vice president and distinguished analyst in Gartner's healthcare provider research practice. He covers electronic medical records, interoperability, health information exchanges and the underlying technologies of healthcare IT, including application integration and standards. Read Full Bio


Thoughts on Practical Concerns re: Certification Recommendation


  1. […] Go read Practical Concerns re: Certification Recommendation. […]

  2. […] ← Practical Concerns re: Certification Recommendation […]

  3. M. Davis says:

    My biggest concern with having multiple certification agencies is the potential funding shortage to support them. What happens to the process if several certification agencies are created, and then exit the market after certifying EMR companies because they can’t fund their operations with the certification fees due to the competition to win market share. This would be disastrous to the industry. I believe the most prudent move, especially with the time frames of ARRA, is to fix what is wrong with CCHIT and move it forward.

  4. A. Wiesenthal says:

    Wes–Full disclosure–I am a current CCHUIT Commissioner (as you know) and have been one for 3 years. Separate from the fact that I would hate to see whatever work I did largely go down the drain, there is no excuse for not attending to the lessons learned by and from CCHIT. We ought to have only 1 certifying body (whether or not it is CCHIT. That body, needs to have a business model that will keep it alive, and it needs to look after the stakeholders you have described.It is hard for me to understand why we should not adopt the current CCHIT infrasturcture, people and all, create added or different governance to deal with some of the governance issues, and move on with iteratively improving certification over the next several years.

  5. Joe Heyman says:

    There are competing accrediting organizations with the Joint Commission. It is the most difficult, not the cheapest, and, yet, the most popular.

    Why look at the problem as one of foul human nature?

    I think Wes’ is looking for the dark side of people. His argument that, “It is unlikely that the majority of physicians will accept these features unless they see near-term, highly credible pain if they don’t”, speaks to that attitude. That is a terrible description of why physicians do what they do.

    The truth is just the opposite and actually is the problem: It is unlikely that the majority of physicians will accept these features unless they see near-term highly credible benefit if they do! Have we any evidence whatsoever that CCHIT certification has improved quality and safety for anybody?

    If they don’t see that benefit, then why should they purchase something? To satisfy someone else’s needs?

    Joe

  6. […] Heyman wrote a response to Practical Concerns re: Certification Recommendation that raised some important points. A dialogue between us may further illuminate the […]

  7. Dear Wes,

    Over the last couple of years I have had the pleasure to interact with you on health information technology standards and certification issues. I was co-chair of the CCHIT Ambulatory EHR Workgroup. Typically, I agree with your interpretations, but I cannot agree with this blog post.

    First, your criticisms make a large assumption that ONC and NIST will create a haphazard and poorly run accreditation process to select and monitor certifiers. This seems very unlikely.

    I sat at the HIT Policy Committee testimony throughout the entire day on July 14, and listened carefully to the recommendation proposal presentation on the 16 as well. I heard presenters show a deep understanding of the need for a robust and thoughtful accreditation process for HHS Certification. Based on those discussions it is unimaginable to think that a “Sleaze Bag Certifiers, Inc.” would ever meet the requirements by ONC and NIST to be a certifying entity.

    Secondly, you seem to make the assumption that there will be no choice of certifiers by the end-consumer or purchaser. I did not hear that at the meeting. In fact, I heard the opposite from Committee members: that other certifications should be allowed to exist and service the market. It is important to separate the certification from the certifying organization. By that I mean a certifying organization can have a certification for meaningful use and additional certifications for more advanced criteria. It seemed even possible for a certification entity to add advanced criteria to the base meaningful use criteria and only have one certification.

    Your arguments also assume that there is no choice with the HIT recommendation. In my opinion, the opposite is true in that with only one certification entity there is no choice. That was acceptable in the “Bush-administration approach” (as you put it), because certification was voluntary. There was no mandate or penalties for not using a certified EHR. This will not be the case in 2015 with meaningful use. This change to involuntary certification is one of the key issues requiring a new approach (see my testimony to HIT Policy Committee on Jul 14).

    Third, your arguments against the new approach and for CCHIT appear orthogonal to that point. For example you state, “But it does mean that a practice or hospital buying a product now…. must correctly judge whether it does or will grow to meet the evolving meaningful use requirements in 2015.” This would be true if a single certifier was used, be it CCHIT or other, and if multiple certifiers were used. It is a very important issue, but again orthogonal to the issue of how to certify.

    There are more comments that could be made regarding errors, at least how I see it, in your post, but my comment is already too long.

    Best,
    Steven E. Waldren, M.D.

  8. Paul Egerman says:

    Wes, I hope that you do not interpret our recommendations as a criticism of CCHIT. Your efforts, and the efforts of your colleagues are very much appreciated. ARRA created a new reality, however, and mandated that ONC establish criteria. ARRA also mandates the role of NIST in managing certification process.

    With regard to the issue of competition, I don’t think that competition will create a “race to the bottom”. We certainly haven’t seen such a race when hospitals compete against each other. Besides, a monopoly has its own significant problems, including the inevitable creation of consumer distrust.

    I would be happy to speak with you about these matters.

    Best regards,
    Paul

  9. Wes, I appreciate your raising the question of the ongoing role of EHR certification now that we have “meaningful use” criteria. Is there a need to certify EHRs with respect to the large set of functional characteristics in the current CCHIT criteria, when meaningful use addresses only a small portion of those functions?

    I believe there is a continuing need for EHR certification, particularly in the area of interoperability, but agree that this needs to be elucidated and refined so that it fits coherently into the new paradigm of how physicians are rewarded for using EHRs. I haven’t heard anyone else addressing that, so thanks for bringing it up…

    Regards,

    -Walter



Comments are closed

Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.