How and Where Sebelius Screwed Up – A System Engineering View

| November 9 2013
David Leeper

sebeliusI’ve been watching Kathleen Sebelius’ struggles with a mixture of sympathy and scorn. I have some sympathy because after my 40 years in the engineering business I have a good feeling for how complex and difficult Obamacare is to implement.  I have scorn because she and her crew seem to have ignored what the industry has learned since 1975 about architecting and implementing complex systems.

What did she ignore? I think it’s best put in terms of classic aphorisms* from the art and science of system engineering.  Firstly –

If you want to win the company softball tournament, you cannot let the boss’s nephew pitch.

Sebelius’ process for choosing CGI and other contractors is generally a mystery, but there were apparently no-bid contracts awarded based on personal links between the contractors and administration insiders, including Michelle Obama and her Princeton ’85 classmate Toni-Townes Whitley. This may be standard Chicago-style cronyism, but for a nationwide mission-critical system like Obamacare it is grossly negligent and scandalous.

A second aphorism appears below, perhaps the most important I learned in my 40-year engineering career:

A complex system that works invariably began as a simple system that worked.
A complex system designed from scratch never works and cannot be patched to make it work.
You have to start over with a working simple system.

Obamacare-flow-chart

Look at the system diagram of Obamacare at the right. A small-scale pilot system is an absolute necessity on a project this large. It does not appear that Sebelius & Co. ever built one and learned from it. Instead it appears they built the whole thing, barely tested it end-to-end at all, and threw the switch on October 1. Failure and embarrassment were virtually certain. Consequently, there is a high likelihood that the whole thing will need to be thrown away.

Two additional, closely related aphorisms are:

The hardest part of building any system is deciding precisely what you want it to do.

Plan to throw one away. You will, whether you want to or not.

A pilot system does more than allow for simple “debugging”. The fact is that the client or customer rarely knows what he or she wants. The most important purpose of a pilot system is to allow the client to test the user interface and stabilize the requirements for the system. From news reports, it appears that system requirements were still in flux as late as the summer of 2013. Frequent and late-changing requirements are the kiss of death for a system project as complex as Obamacare.

As Fred Brooks, the famous IBM 360 system architect has said: “No other part of the conceptual work is so difficult as establishing the detailed technical requirements, including all the interfaces to people, to machines, and to other software systems. No other part of the work so cripples the resulting system if done wrong. No other part is more difficult to rectify later.”

Regarding actual execution:

Take no small slips.

When it’s obvious that a delivery date is going to be missed or (heaven forbid) a launch actually fails, it’s imperative that a manager not try to soften the impact by taking a small slip in the schedule, assuring the customers that there are just a few “glitches” that need to be addressed. Yet that is exactly what both the president and Sebelius appeared to do, immediately after October 1, and now they are betting heavily on a fix by Nov 30.

confusionAdding more people to a late project makes it later.

It’s human nature to believe that adding more hands will make the work go faster. So when Sebelius or the president talk about all the external talent they are bringing to bear, I cringe. These folks don’t know the system architecture, don’t know the people working on it, and have their own opinions and egos. They will start crashing into one another, arguing about the best course forward, and almost certainly delaying the whole thing even further.

 The greatest difficulties occur in the “white spaces” between the system elements.

The many elements that appear in the now-classic system diagram of Obamacare (above) have to communicate seamlessly with one another. Each of those elements will likely have been built by a different team. Some may be new, and some may be old legacy systems that cannot be modified easily if at all. Even if all those elements do their individual jobs well, it is the syntax and semantics of the information exchanges where the biggest problems occur. And these in turn almost always come from mismatched assumptions about the overall system requirements.

The best and longest-lived systems are the product of only one or two architects with resonant minds.

zeke1I have no idea who the Obamacare system architect is. It’s not clear that there is or ever was one. Zeke Emanuel is sometimes said to be Obamacare’s architect, but he is a “bioethicist”, an academician, and a classic Left-wing faculty lounge intellectual. I hope for everyone’s sake it is not him(!).

System engineering is part art and part science. But it is not new. It is remarkable that with all the experienced system-engineering talent available in America that Kathleen Sebelius chose instead to follow a series of politically expedient management decisions all the way to the launch date. The results should come as a surprise to no one.

——————————————————————-

*An aphorism is an adage — a tersely worded phrase or statement of a truth or opinion.  Most of the aphorisms in this article come from Fred Brooks, author of the classic Mythical Man Month and No Silver Bullettwo of the most important foundation documents in (software) system engineering ever written.

2 comments
Carl Reuter
Carl Reuter

As a retired systems engineer (35 years), I am not surprised at how badly Obamacare development has gone. What disturbs me is the lack of scrutiny from investigative reporters and news coverage leading up to the current debacle. No one seemed to be demanding to see the critical SE artifacts that must accompany any successful big system development. Why?

sspdavidmiller
sspdavidmiller

I am not surprised by this well written article. It seems to me that the reasoning of the author that was seemingly deliberately ignored by Sebelius et al at great taxpayer expense warrants criminal charges that, if proven, should result in prison terms for all involved. When will accountability for misdeeds for highly placed persons return to the US?