Thursday, December 20, 2012

Will Certification Kill the IHE Connectathon?

Summary: IHE Certification has been announced, is dubious but probably unavoidable; the for-profit ICSA Labs will execute it, possibly gutting the participation in future Connectathons and leading to their demise

Long Version.

With unseemly haste, and little time to go before the IHE North America Connectathon starts at the end of January 2013, IHE USA has announced a new IHE Certification program to be executed by ICSA Labs, a for-profit division of Verizon that has significant experience in security certification and is also a Meaningful Use ONC-Authorized Testing and Certification Body (ONC-ATCB), which competes with the likes of CCHIT.

There is no point in debating whether certification adds any value to the customer of interoperability products; that cat is already out of the bag, for MU at least, since ONC has a hard-on for certification (and asks standards organizations like IHE and DICOM uncomfortable questions when there is no corresponding formal certification program). Compared to all the other ludicrous costs of bring a product to market that do little if anything to improve quality or assure safety or interoperability, this is just yet another check box (and lifetime employment program) for the army of quality, safety and regulatory personnel. I don't have a strong opinion about the merits of certification per se; my anecdotal experience as a vendor and a customer and a worker in regulated industries would lead me to believe that it is neither necessary nor sufficient, unless it is conducted with such thoroughness (by which I mean depth and coverage of testing, rather than volume or rigor of documentation) as to be impractical. But I don't have any real evidence one way or the other.

I have no direct knowledge of the IHE USA Board's discussion of the matter, so can only guess as to their motivation. It isn't likely to be the potential for a significant revenue stream, since the price point precedent established by the ONC-ATCB market isn't very high, and the cost of doing the vast amount of ISO-standard inflicted paperwork isn't trivial. I am not sure whether or not IHE USA is getting a cut from ICSA Labs, or perhaps is even funding them.

The haste smacks of desperation, so another guess would be a desire by IHE USA to try to remain relevant in the US national "discussion" on what standards to adopt (if you can grace the ONC autocratic decision-making process with that term). Since a nice thing about standards is that there are so many of them to choose from, and anyone can make up a new one and call it a standard (as the DIRECT folks have clearly demonstrated), there is no question that choices need to be made. IHE USA's difficulty may be that, much to their chagrin, the ONC does not accept as a no-brainer that IHE's choices are always the best choices.

There is also little point in bemoaning what, as far as I can tell, has been a distinct lack of consultation with the IHE community at large. Almost nobody I have talked to knew anything about this before it was announced, and even those meetings that I am supposed to participate in (being Rad TC co-chair) like the Domain Coordination calls (which discuss such boringly bureaucratic and tedious topics that even I with my love of standards documentation can't stomach), were not, I am told, briefed about this. Nor was there a peep on the Testing and Tools mailing list. Clearly the lowly workers did not merit either the opportunity to comment nor advance warning. Strangely enough, the vendors I have talked to were not aware of it either, and indeed the vendors have a long history of opposing the additional burden of certification and have lobbied against it on the basis that self-certification has a demonstrated successful track record, and certification adds cost without value. Perhaps the IHE Board knew how controversial this would be and did not want their decision second-guessed, or perhaps they just got snowed by a plausible argument by a talented evangelist for the concept.

Or perhaps ONC just issued one of its edicts and the board had no choice ... conspiracy theories abound given the lack of transparency in the decision making process.

For better or for worse, unless the pilot ICSA Labs project overlaid on top of the IHE NA Connectathon somehow fails spectacularly, IHE Certification, now that it has been "announced" may be here to stay. If few vendors sign up for certification, then it might fail, but given the loss-leader price of $1,500 for each actor (in packs of 5) that they are offering, it seems unlikely that there won't be a sufficient critical mass of product managers who are tempted to get an additional potential marketing advantage over and above their normal Connectathon passing "gold star", or are just scared of being left behind if certification does become a trend. Today's briefing by ICSA Labs was certainly laced with dubious but blatent teasers for marketing folks. Besides, like UK National healthcare IT projects, it may be sufficient to "declare" victory even in the face of obvious failure, if the political motivation to proceed is strong enough.

So, with that background, on to the question of what this means for the upcoming and future Connectathons. One of my initial concerns, that the work product of the army of volunteer monitors who staff the Connectathon would somehow be "re-labeled" as a certification result, was clearly assuaged by ICSA Labs on today's call - their tests will be separate from the Connectathon tests, executed by their own staff (who they are calling "inspectors" to distinguish them from "monitors"), though they may repeat the same test, or add more steps or tests, and will certainly add more documentation. So, the volunteers will not be "exploited" in the sense of their work being reused for someone else's commercial gain. Nor will ICSA Labs interfere with the process; the certification work will be additional, and later in the week, though predicated on a Connectahon pass.

So far, so good, but what about in the long term? If I, as a vendor product manager, feel the need to obtain certification for any or all of my products, why would I bother to participate in the Connectathon at all? If I have to do the same amount of work, gain the same experience, and the additional cost is small related to the real or perceived marketing benefit, why do it twice? Though the certification testing will take place at the Connecthon itself for the pilot, in the long term ICSA Labs plan to do this separately as needed, virtually (over the Internet) if possible. Arguably, the Connectathon serves as dry run, but those can probably be done just as easily at home or informally.

Similarly, why would I ever volunteer to staff a Connectathon, when I could probably charge my hourly rate to ICSA Labs, or get a job with them, to do exactly the same thing (unless of course ICSA Labs treats its employees like parent company Verizon treats its linemen)? There is an educational component to volunteering, but for experienced folks, the supply is small and now there is a competing consumer.

That is why, at this date, I am opposed to IHE Certification process as announced; not because I think it is a waste of time and money (which I sort of do), but because it puts at risk what I think is much more important, the opportunity to gather and learn and test and even experiment at the Connectathon, in a relatively informal manner that encourages iteration, improvement and collegiality between competitors. In my opinion, there will inevitably be a dramatic reduction in Connectathon participation as a consequence of the need to become certified; regardless of how much the engineers might want to test at such an event, their marketing bosses who hold the purse strings just will not have the will or the resources to do both.

If I were one of the IHE decision makers, and felt bound to jump on the certification bandwagon to survive, I might have made a different choice, one which did not put at risk the Connectathon, but instead leveraged it, and found a means to certify Connectathon results, building on the experience and dedication of the existing staff, and added the necessary documentation (preferably automated) to satisfy the ISO processes. This sort of thing has been discussed before, including by the people who do the work. I dare say I would have been skeptical about the feasibility and initial cost of this, and tempted by the offer of an existing organization like ICSA Labs with the ISO infrastructure already in place, but I would have prioritized the future of the Connectathon, and seen the modest revenue stream from certification as a means to (belatedly) beef up the tests and tools that have languished somewhat for lack of resource investment over the years.

Not to mention avoiding the slap in the face to the core Connectathon staff, who, for well over a decade, have been putting on this event with diligence and dedication, and who are now probably destined for unemployment or a future as underpaid mistreated Verizon drones, in addition to seeing their many person-years of work in the form of openly available tests and tools handed off to a for-profit to reap the benefits, as well as being denied the opportunity to turn their vast talent to performing certification themselves. They are obviously not able or willing to speak about this, but observation of their reaction would suggest that they were not consulted or warned about this either; if that is the case, it is shameful treatment by the IHE organization that they have served so loyally.

Given my upcoming New Year's resolution to be more of a glass half full kind of guy though, perhaps I should view this as a precedent and take the opportunity to start a new initiative for DICOM Certification, probably the last thing the world really needs, but hey, there is no point in standing on principle in the face of a tidal wave, and if you can't beat them join them. Give the customer what they think they want, not what they actually need.

I still haven't decided whether or not to withdraw from volunteering as a monitor for the upcoming Connectathon. I would hate to miss it, but given that it is one of the few tangible ways to express my displeasure, I don't think I have much choice other than to boycott the event.


PS. An interesting side question, is why IHE USA did not select the non-profit CCHIT organization as a partner in this, particularly since they are doing the certification for Healtheway (formerly the Nationwide Health Information Network NWHIN). Perhaps it is just because Mike Nusbaum, a long time IHE player and expert has joined ICSA Labs as Program Consultant and seems, judging by today's briefing, to be a key player in this effort; they certainly have someone who knows what they are taking about. Not that I have any great love for what CCHIT does either, just wondering.


Herman O. said...

I could not have said it any better myself. Interestingly enough, IHE seems to go ahead with this at full speed even after the backlash when it was initially announced.
I wonder what the hidden agenda is as I have not talked with anyone who thinks that this certification makes sense at all.
I have talked with some who are potentially boycotting this as well, but the certification will probably happen despite these intentions.
But, who knows, maybe IHE is seeing the light, let's keep on dreaming...

Eichelberg said...

Interesting article. I don't know enough about the details of the certification program to really voice an opinion about its value. Being a long-term IHE connectathon monitor, I can certainly say that the "traditional" IHE process (including all the Gazelle test scripts) is not well suited for any kind of certification scheme, for a number of reasons:

- IHE mostly tests the practical interoperability of two or more systems, not their conformance to the specs. This is covered to some degree by the MESA tools, however, many vendors do not take these tests seriously and if IHE chose to kick everyone off the connectathon who failed some MESA test then the connectathon would still be a fairly small event. Most certification schemes I know actually test the conformance of one system, not interoperability of two systems, and that required totally different tools, methods and tests.

- IHE encourages vendors to come with system prototypes and to improve their software during the connectathon week. This is in my opinion one of the most important values of the connectathon: the prototypes are much better on Friday afternoon than they were on Monday morning. However, this is not really compatible with a certification "way of thinking", where a certificate is awarded to a specific product, version and patch level, and where every relevant change (e.g. to the networking code) would require re-certification. You cannot have both in one event: Either you force people to leave their systems unmodified and on product level, or you encourage prototypes and changing code.

- The connectathon week is always too short for vendors to actually do all interoperability tests they would want or need to do. If you now cut away 1-2 days for certification tests, that will make things worse.

David's other concerns also have their merit (such as having paid and unpaid people doing pretty much the same in the same room at the same time), but my biggest problem is that I cannot see how you can actually make both - the traditional connectathon and a product certification scheme - work within the same 5 days. Really a pity that I won't be there to see how it will work (or fail).

One last comment: If they are really willing to offer full product certification for 1500 per actor, then the certificate will either be not worth a lot (due to very little test coverage), or the company doing this will burn an enormous amount of money on each certificate, at least on actors such as Order Filler, Image Manager, or stuff like BIR or MAMMO.


David Clunie said...

On May 22, 2013, a spammer for commercial certification services left the following post, the text of which I have re-posted here after removing the self-serving advertising links, because it is a fair comment in its own right, though nothing at all to do with what IHE is considering:

"Achieving ISO 27001 Audit Certification visibly demonstrates that your business is conscious of the confidentiality and integrity of all information it creates, holds, moves and stores. Conformance to ISO 27001 also clearly proves a company's heightened awareness of any potential internal and external security threats that may lead to a breach of security of any information managed within your organisation."