On 16 July the Certification and Adoption Workgroup (CAWG) of the U.S. HIT Policy Committee provided its recommendation to the full committee. It calls for a 3-level approach to certification. There will be an opportunity to comment and I hope that readers of this blog will consider the thoughts here and then comment as they see fit. (Please see a subsequent clarification based on early feedback from blog readers at More on The EHR 3-Level Certification Approach.)
The current recommendation is
- ONC itself determine criteria for certification
- ONC identify an organization to accredit various certifying organizations
- As many organizations as feel there is a business in doing so become accredited and begin certification.
A corollary recommendation is that vendors should only need to be certified by one organization.This implies that all Federal programs and regulations that require certification would accept certification from any accredited organization. The main focus right now is the incentive payments but one must presume it applies equally to grant programs that provide assistance in implementing EHRs. Since Federal money contributes to many state programs they would also be bound, it seems. The bottom line is that vendors and CDOs that want to qualify with self-developed EHRs would each be free to pick one accredited certifier. The entities that are relying on certification to determine whether to give funds would not be able to say that “we feel that Sleaze Bag Certifiers, Inc.” is mainly in the business of helping vendors find loopholes and “Upright Certifiers, Inc” is diligently applying the criteria in a way that ensures we know what we are getting. They would have to accept certs from SBCI and UCI indiscriminately. When the choice of certifiers is made only the vendor, not the organization that relies on the certificate, this creates an inevitable pressure to be the certifying organization that is the least thorough in its process.
At worst this guts the idea of certification. At best it creates a requirement that ONC construct a set of requirements for the accrediting organization that assures it not only pre-qualifies certifying organizations but also monitors their actions and provides a dispute mechanism for those that feel that a given certifier is too lax or too strict.
There have been those lobbying against the Bush-administration approach, which was embodied in CCHIT. Whatever the merits of the argument, it seems unlikely that a government agency can promote the notion of industry self-regulation given what has been learned from the financial services industry in the last year. Therefore, CCHIT has a very steep hill to climb if it to retain its status as the public consensus body for determining requirements.
Even so, there is a lot of valuable experience that can inform the specific adoption of the CAWG recommendations. I have worked at all three levels of the CCHIT workgroup, commissioner, and trustee. In my day job I am continually evaluating vendors and vendor claims, sorting out the information from the noise, at a company that became a billion-dollar company because users trusted our advice on vendors. Therefore I put myself forward as someone who has a good understanding of the process and a balanced view of vendors, neither demonizing nor glorifying them. I hope you will consider the material below in that light as you make your own decisions on commenting.
Questions to Consider
I have organized this into questions to consider and thoughts that might help. As always comments in the form of additional questions and further considerations are very, very welcome.
Why do we need certification at all, since we now have meaningful use criteria? In other words why not let physicians and small hospitals buy whatever product or build and assemble whatever combination of open-source code, commercial products and home-brew that makes sense. Either they qualify for meaningful use or they fail. They can reply on their trade associations and organizations that survey satisfied customers to give them guidance on what to choose.
The short answer is that the law requires certification for several big programs. The equivalent question in this context is should ONC minimize the certification requirements and rely primarily on meaningful use going forward?This gives the marketplace the most opportunity to respond innovatively to the meaningful use measures, in other words to achieve the “outcome” rather than follow a prescribed approach.
This is a very attractive proposition. But it does mean that a practice or hospital buying a product now (and let’s face it, most of them will buy products rather than build) must correctly judge whether it does or will grow to meet the evolving meaningful use requirements in 2015. Customer surveys, whether commercial or conducted by specialist associations, primarily ask the question “are you satisfied with your product and the vendor that sold it?” They will no doubt have collected and published great data by 2016 on the ability to mean the 2015 meaningful use requirements, but they aren’t that much help to a CDO making a purchase decision in 2011 or 2013. This element of risk will contribute to a trend to defer purchases, even at the cost of giving up some incentive funds.
Is it possible to certify products to give solid buying advice to physicians? After all, CCHIT has failed to set advanced criteria for aiding the cognition of physicians, evaluate the quality and safety of user interfaces, or to evaluate the fiscal stability of vendors.
When CCHIT started there were about 200 EMR products that could be identified by scanning the Web, ads in trade magazines, etc. Currently about 80 of them are certified. To me this indicates that CCHIT has done a good job of finding the middle course between setting criteria that physicians won’t buy or can’t implement and being completely laissez faire. For example, all the so-called EMRs that were primarily systems for storing and retrieving scanned documents have either modernized or failed to be certified. Systems that only accept medication orders as text have similarly had to evolve. Systems that cannot record problems and allergies, and use med allergies in med orders have been eliminated.
No certifying organization can assess all the criteria that go into choosing a product. For example:
- Financial Viability. There are difficult issues in certifying financial viability, since almost no startup looks viable by any objective measure– or at least most people don’t think that startups should have to be sufficiently capitalized to look viable. CCHIT made a compromise: it allowed firms to provide some basic information about the size of their customer base and made it apparent when this information was not provided. Furthermore, it followed up with vendors that had qualified about every 90 days to see if they were still in business and answering their phone. Would this be adequate, insufficient or overkill in the future? That would be up to ONC to decide in setting criteria.
- Quality and Safety. The quality and safety of user interfaces is subjective. Arguably one could establish some objective measures of what makes a “bad” or “unsafe” UI, but this would be incomplete and the converse of these measures would not define a “good” or “safe” UI. The experience building consensus on clinical measures has shown that even when the conceptual ideas can be easily accepted, nailing them down to the point of being objectively measured in real-world systems is a long process. It would be a substantial value if ONC were to start the long-term process now of building consensus on objectively measurable system characteristics that are unsafe. Perhaps it should kick this off with the Institute of Medicine. It seems unlikely that the consensus could be developed and reduced to practice in certification much before the 2013 round of criteria.
- Cognitive Assistance.Advanced clinical support and point-of-care access to medical knowledge is an area where CCHIT has been criticized for being too strict and not strict enough. Vendors are often accused of watering this down because they can’t do it or beefing it up to try to restrict new entrances in the market. Having seen this debate in CCHIT I can say that the vendors tended to work to keep criteria matched with what they thought physicians would be willing to pay for. The pressure to raise criteria came from other stakeholders with righteous concern about safely and quality, hoping to use certification as a way to force physicians to practice better medicine. This is an area where any certification program should be closely tied to meaningful use criteria or other elements of healthcare reform. It is unlikely that the majority of physicians will accept these features unless they see near-term, highly credible pain if they don’t.
Should the accrediting organization evaluate howa certifier determines compliance or only whether it does? This 3-level approach is straight out of the playbook of ISO 9000, the capability and maturity model and the Common Criteria for security. It will be interesting to see what happens there. I am going to guess, however, that it is impractical to determine in advance whether a candidate certifying organization can be effective without reviewing its methods.
In adopting the 3-level approach there are three strategic approaches. ONC can (1) leave the determination of methodology to each separate certification organizations; (2) strike out to establish a brand new approach to certification to be applied uniformly by all certifications; or (3) adapt the approach used by CCHIT for use by all certification organizations.
As discussed above, the 3-level approach will create market forces driving certification organizations to be the least intrusive on vendors. Arguably this means that the first approach, leaving the certification process to the certifying organizations, is equivalent to reducing the impact of certification to the legal minimum, and relying almost entirely on meaningful use.
The second and third approaches, ignoring the CCHIT approach or adopting it, represent the endpoints of a spectrum of approaches. Intermediate points on the spectrum arise based on how much attention is given to CCHIT experience. I strongly advocate paying close attention to the CCHIT experience with regard to certification because it achieved several important balances:
- Criterion objectivity. Several rounds of certification have led CCHIT to a set of criteria that can be rated by judges with good inter-rater reliability. This is a value that the accrediting organization should consider in evaluating the methodology of organizations.
- Criterion testability. The devil is really in the details in objectively testing functionality. This will be the biggest challenge for the 3-level approach, because it involves a give and take between the determination of criteria, the testing approach and the test scripts. In adopting the three-level approach the ONC should not consider how the HIT Standards Committee, or whoever else determines testing criteria should be able to participate on a quick-response basis as the certifying scripts that support a criterion are developed and executed. CCHIT also used a “beta certification.” Vendors could participate without charge and without their names being released to the pubic. This allowed the final scripts to be tuned up by experience before being put into play.
- Certification costs.CCHIT adopted a budget of only testing as much as can be evaluated in a single day of testing (for ambulatory EHRs). This involves complex trade-offs between the selection of criteria and the methods of certification. It will also be more complicated in the 3-step approach.
Should the ONC reconsider the 3-level approach? I perceive that CCHIT actually achieved a balance between three different groups of stakeholders, those that must build to the requirements and sell that product to CDOs and physicians; the purchasers of products who are primarily focused on the short-term impacts directly oh themselves; and other stakeholders that want to ensure that funding EHRs leads to better and less expensive care. The closer observers were to seeing the process the more they will agree with me. However not everyone shares that view. I would urge those who write comments to consider this issue on their own, keeping in mind that if the outcome of establishing criteria is truly balanced, nobody will like all of it.
I would further argue that CCHIT met unbelievable deadlines, launching itself in a year and going through annual updates to its criteria.
Because of the current general public concern about industry self-regulation I doubt that the current governance of CCHIT will be acceptable. A separate question is whether the full bureaucracy of the 3-level process is needed,given the issues discussed above. I strongly suggest that commenters give consideration to whether a group such as CCHIT with a different approach to governance wouldn’t a better approach. Commenters might want to give some thought to how a governance approach can avoid the problems to the usual government “waterfall” process where requirements are developed by one entity and then become sacrosanct to the entities that need to figure out how to make them work in certification. (My clarification at More on The EHR 3-Level Certification Approach is particularly relevant to this section.)
Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.