brad-gilbertson.com

Long considered a normal part of life in most software environments, the information and control provided by good configuration management (“CM”) practices increase in value when CM includes all Design Inputs and Design Outputs.

In this post, I’m going to tell you that, if you intend to maintain your product through multiple updates, or will develop many products, you should consider integrating hardware and software CM in the Design History File (“DHF”), and that the resulting configurations should be made easy to design transfer to the Device Master Record (“DMR”).

In subsequent posts, I’m going to describe how to handle specific CM and document publication techniques to make this approach happen, plus show you an idea for world-class software CM. This post lays out some basics.

BTW, this posting will open a discussion of a particular approach to CM, but won’t provide a tutorial on the general subjects of CM and its close relative, version control.

Here’s a Wiki page on CM in general: http://en.wikipedia.org/wiki/Configuration_management

Why Should You Bother Considering These Ideas?

The goal is to create an environment where written procedures are necessary only at the highest level of your Quality system, and that the work that people do tends to be compliant with those procedures without a lot of bureaucratic overhead.

It’s not hard to meet this goal, of course, but you want more than this. You also want everybody in the company to have an easily-accessible source for all information about requirements, design, risk management, and part specifications for manufacturing. If your people have to do manual searches of paper to find the information they need, they eventually will stop bothering. If they rely on common LAN directories, you won’t really have your data under control. FDA is likely to notice, and send you a company greeting that starts with the digits “483”.

Basics

The basic purpose of CM is to ensure that you can always replicate a particular “version” of your system or other complex artifact. In this blog, of course, the focus is always on medical device systems. A system is composed of many components that can each have a different version. The combination of all these components and versions is a “configuration.” It’s easy to get buried in the terms “revision,” “version,” and “release.” In software teams, there is usually no difference in meaning. In hardware teams, the word “revision” means “a formally approved version,” and a “version” is “identifiable specific content.” For this blog, I’ll use the words “revision” and “version” interchangeably, and use the word “release” to mean “a controlled configuration.”

Most medical devices comprise both hardware and software, with many also using chemical disposables (for the purpose of this post, I’m going to refer to chemical components as hardware, which might make you scream in pain if you’re a chemist, but keeps this post simpler without sacrificing much information). Each of these technologies has components that can be version-controlled. As development progresses toward 510(k) clearance or PMA, and Design Transfer begins, good configuration management will provide attachment points for snapshots such as the following:

·        Requirements to be implemented in a Scrum sprint, or other development increment, depending on your process;

·        Impact assessments to justify limiting testing to a part of the product’s features or characteristics in support of an on-market or development-stage change;

·        Development prototype configurations used to build up characterization or launch justification data;

·        Pilot configurations, including bills of material (BOMs), manufacturing methods, test methods, purchase specifications and other Design Transfer outputs.

The problem space for software is still somewhat different than for multi-technology systems, because:

·        Software tends to be more complicated, with more parts, than other technologies (see my Gleick On Software page);

·        Software lends itself very readily to automated controls and management, because the tools used to design and implement it were developed by people who are highly sensitized to software CM;

·        Folks who are more used to managing hardware are not as familiar with CM concepts and practices as are their software engineering counterparts.

The Tower Of Babel

It’s not a trivial question to consider pulling all your DHF artifacts into a single CM model. Each engineering and business discipline has its own toolkit, and the toolkits may integrate poorly, or be expensive to integrate.

All your manufacturing information will probably be contained in an Enterprise Requirements Planning system (“ERP”) and/or a Product Lifecycle Management System (“PLM”). Depending on your vendor, these two systems will probably represent everything as Bills Of Material. Their job is to manage changes to the versions to components, and track configurations. Manufacturing folks will use ERP to know what they’re building today. Hardware developers will use PLM to conglomerate hardware configurations into BOMs that can be handed off to the Design Transfer.

All the mechanical designs will be done in a mechanical CAD system, and electronics will be done in one or more electronics CAD systems (for schematics, board layouts, FPGA IP modules, and fabrication specifications) that may, or not, have their own version control and CM vaults. Software requirements and designs may be developed in a computer-aided software engineering system (“CASE”), and one or more interactive development environments (“IDEs”) that may also have independent repositories.

Source code might live in a software CM system, such as my favorite, Perforce, or Visual Studio, or one of the various freeware systems.

Don’t forget systems requirements; it’s not unlikely you have those stored, or at least their traceability managed, with a database system such as DOORS.

Thus, you have the problem of relating together the content of all these different repositories. You have this problem as soon as there’s a product complaint with a non-obvious cause, or when an FDA inspector shows up at your door. Although it depends on your scope of operations and the complexity of your device, you can raise your total capability, and dramatically reduce your risk of inconsistencies among Design Inputs and Design Outputs, by keeping track of all this stuff with a single tool.

Find An Appropriate Binding Platform, Such As A PLM System

My model for integrating the DHF and DMR uses a PLM system as the hub around which all information sources revolve. Development, and sustaining production changes, are entered into PLM as Change Requests (“CR”), each of which contains a statement of what has to be changed (the “Change Plan,” or “CP”).  The Change Plan can be either simple or complex, depending on the change, but the PLM system tracks each task and displays the artifacts that complete it. For example, if the task is “update a subassembly drawing”, it is completed by attaching an electronically approved drawing and the electronic change order (“CO”) that approved it. BOM relationships are approved as part of executing CO’s.

This is a good thing because all these drawings and approvals are electronically communicated to the ERP system when approved. Just thinking about the PLM end of things, the process looks like this:

Development Artifact=>CO=>Task=>Change Plan=>CR completed

(For each CO)          CO=>development BOM=>test=>CO=>production BOM

Each CO makes a version of a development part or a production part. For product tracking, an entire software release is modeled as a single part. In my particular case, each new release gets its own part number, because my product family needed to track software releases through field service, and maintain different instrument configurations in the field.

What I did was to create a PLM structure, modeled as a BOM, for each product’s DHF and DMR:

 Sorry for the grainy appearance – I’m still working on optimizing my Visio diagrams against the blog theme. Here’s a PDF of the diagram you can view: Integrated DHF & DMR

How It Comes Together

So, the product-description part of the DHF is a configuration-managed BOM, as is the product’s DMR. The artifacts contained in the DHF are the outputs of all the CAD and CASE tools described above.

Project members use the PLM system’s Change Order procedure to post development artifacts to the DHF, as part of the CR process shown above. Then the same Change Order process moves various parts and software releases into the DMR as a Design transfer activity, including whatever manufacturing processes are needed, when projects show that those parts and software releases are ready for market.

Some caveats and costs are:

·        Capital costs of enterprise platforms

·        Support costs of enterprise platforms

·        Systems analysis to ensure that your plan takes your entire process into account and has a growth path built in

The downside of this approach is that you need a strategic systems thinker in charge of the initial implementation, and that person needs to understand how your company works. PLM and ERP vendors may not provide this insight during the integration process, because they may have a plan in mind when they show up at your door.  Without a good internal strategist, you have a risk of buying what the vendor wants to sell you, instead of what you actually need.

The major benefits of this approach are:

·        Train everybody in the company on only one system

·        Global access to DHF and DMR information – supports knowledge-sharing and concurrent compliance assessment

·        Easy to integrate with ERP

·        Easy to maintain auditability, without a complex body of procedures

·        Removes the temptation to create multiple processes where one will do the job

·        If your company uses common parts among products, or develops derivative products, the total effort to design, implement, and test is reduced, because development and V&V efforts are not duplicated. Further, it’s easy to justify not doing that redundant work when the various information dependencies are baked in at the time of initial approval

·        With this model, it’s relatively easy for development projects and sustaining engineering to use the same design control practices and knowledge base, which helps prevent unintended effects of changes by putting your products’ design history and critical quality attributes front and center.

The Bottom Line

You can build a DHF-DMR structure that enables your development and sustaining engineers to concentrate on doing the real work, and get compliant document maintenance as a side effect. Your company and product line can grow quite extensively without accompanying increases in complexity. It’s analogous to what any database admin can tell you: a theoretically correct initial model will reduce the total cost over ownership – of your entire company, in this case! – including maintenance.

I’ll bring in more details in future posts to show how you supporting systems and even manual (ugh!) procedures can use CM to keep your troops’ eyes on the prize, and still hold the Bureaucracy Virus at bay.

Thanks for reading!

Brad

The first anti-bureaucracy therapy is The Operating Fiction Of Linear Thinking. If your troops learn this concept and practice it religiously, the FDA will like you, and your Quality System is less likely to grow into a writhing mass of poisonous, strangling, bureaucratic tentacles.

First, let’s nail down some ideas, so we can use them in this post, and others to follow.

What FDA wants is to know the following:

· You took the trouble to understand what behaviors and characteristics your product is to exhibit. You also know why you didn’t choose some behaviors or characteristics.

· Your processes are very highly likely to identify errors soon after they occur, including when making changes during development or manufacture. Also, you can prove it.

· Using your processes, you are very highly likely to devote your effort towards more important stuff than less important stuff.

· Somebody who doesn’t already know all your stuff can understand it, at least the high level, without having to live with you. Not only that, but this hypothetical observer will be able to learn this material in the course of an inspection, which usually lasts a few days.

I call this last point “Maintaining The Operating Fiction Of Linear Thinking.” The way many folks will talk about the requirements listed above will make you think that you’re required to actually do your work in a linear way. This is not true. They just want to understand your stuff in the limited time available.

When you’re creating plans, requirements, and designs (records of what you will do), and other documents such as test protocols and test results (records of what you did), FDA wants you to structure your material to tell a story.

So, here’s a metaphor: Think of how movies are made:

· First, there’s a “treatment”, which is a summary of what the movie will be like

· There’s a screenplay, which contains dialogue and basic elements of scenes

· Storyboards lay out the visuals of scenes

· A host of other documents describe costumes, props, and other materials

· Filming starts. This is the first place where The Operating Fiction Of Linear Thinking begins to manifest.

· After filming, the movie is edited. This is where The Operating Fiction Of Linear Thinking shines.

Let’s take the example of Castaway, starring Tom Hanks. Remember the early parts of the movie, when Tom is chubby and clean shaven? Then, later in the movie, after he’s been spearing fish for like three years, he’s slender, fit, muscular, and bearded, with a head of long hair.

God, there’s no justice. My high-school classmate is rich and famous, and he has his hair. Can you believe it?

Anyway, you could imagine that the director might have filmed the end of the movie first, so that Tom could have a couple of years to grow his hair and get ripped. Then, after filming, they could take six months to fatten the man, and give him a haircut. Okay, he could get the haircut any time, but hey, you know what I mean.

After all that mixed filming, the editing process would bring the movie together, to create the illusion that the narrative and filming occurred in the same order.

Having said all this, I cannot tell a lie. They actually filmed the movie in the same order as the major events, ie fat first, and muscles later.

I had you going, though, didn’t I? And you got the point!

When to apply The Operating Fiction Of Linear Thinking

This concept is an opportunity to replace Quality System content with skills. If you develop a cultural imperative to apply the concept, or create some training materials to inject the skill into the genetic code of your practitioners’ brain cells, then you can lighten up the amount of material you have to write to support each Quality System component. The troops are more likely to do a good job without detailed instructions.

The problem is that, if poor practices take hold, and your company attracts the attention of the agency, it’s a slippery slope to writing bigger and more complex procedures. One of the temptations is to try to legislate judgment by making perfecting the rules of engagement. While such an approach is theoretically possible, it’s very expensive and difficult, requiring a very high level of skill on the part of your procedures authors.

Wouldn’t it be cool if you could make your troops smart enough that they wouldn’t need detailed instructions for some common activities, such as writing?

Any writing that describes intended activities, or past events and activities, or their impacts, is a good candidate for the Tao Of The Operating Fiction Of Linear Thinking. Below, I’ve included some candidates. Please recognize that I haven’t exercised the fictitious examples for perfect writing and scenario construction, I’m just trying to make a helpful point about what issues to expose.

· Defect reports – During development testing or final Design Verification (not implying a process here, just putting the term “defect report” in context), your tester or other symptom observer may not have realized the actual behavior at the time it occurred. Maybe the person who investigated for cause was able to refine the or correct the description of the failure. In any case, provided you have an audit trail that lets each reader easily find or reconstruct the original description, it’s okay to go back and re-describe the description of the symptom, or investigation, or action plan so that a reader can make the mental connections among symptoms, causes, possible impacts to patients and on-market products, and the mitigations for those causes and impacts.

A defect report is a good example of how teaching the Tao Of The Operating Fiction Of Linear Thinking helps simplify your Quality System. If the troops know how the audience will think when reading defect reports, then they’ll do a good job of writing them. You won’t have to write a tutorial on defect reports as a CAPA for a 483 about your tortuous and garbled defect reports.

· Impact statements for on-market defect reports – your company may call this concept any of a number of different names, but if you have complaints against your device, you’ll have to assess the impact of your problem against your DMR, DHF, inventory, and installed base.

If you can afford a computer system that automates impact assessments, then you don’t need to worry about good authorship for them, but what if your company is too small, or too young, to afford the kind of systems analysis and software that such a system requires?

If the troops all understand the Tao Of The Operating Fiction Of Linear Thinking, then a “best practices” checklist might be all the support your Quality System needs in order produce acceptable impact statements. (I’m hoping to cover the pluses and minuses of checklists in another post, but for now, let’s just remember that, if a checklist is part of your Quality system, there will be a demand for checklist records that show every item to have a response, which can be onerous.)

If such a checklist is available, but not required, then an impact assessment author can write an understandable and complete description of what could be affected by the problem:

When the photon-density bracket fails due to accumulated cosmic irradiation, the gravity beam focus point is changed. The gravitron power sensor detects a change in the gravity beam power. The system reports an exception, then shuts down. Medical results are not affected, but there could be a delay of up to 1 hour in treatment or reporting for a procedure in progress.

Currently installed brackets that are not causing this gravitron power interruption can remain in service without harm to users or patients. Operations will do stock sweeps to remove and replace with rev B, any undelivered and field depot rev A brackets. For asymptomatic product units, a Field Service Bulletin will be issued (see Change Plan 42, task 3) to instruct Field Technicians to replace rev A brackets with rev B brackets at the next preventive maintenance visit.

Change Plan 42 introduces a modified bracket that withstands cosmic radiation for up to 14 years, instead of the current 6 month service life of the bracket.

Here’s what this fictitious example shows your inspector the following information, and demonstrates that you’re paying attention:

1. The condition(s) under which the problem occurs (accumulated cosmic irradiation)

2. The effect on the product in use (error message and shutdown)

3. The effect on operator and patient (delay of up to 1 hour, but no other medical effect)

4. Whether a latent hazard or problem is hiding in the field, waiting to harm customers.

5. How we’ll prevent this problem from recurring with parts already manufactured.

6. Because there’s a reference to a Change Plan, the inspector can see how the problem will be avoided for units not yet manufactured.

· Iterative development process – Unless your device’s method is well understood, you probably have a lot of feasibility development to do (OMG: If you’re a startup, your team might not yet even realize that getting the feasibility prototypes working are really the beginning of the work, not the end of it).

Your design team is going to make decisions based on data gathered during feasibility activities. Your QA folks are going to read the QSR, and conclude that FDA will regard data as suspect unless it was gathered in accordance with a pre-approved protocol (they don’t want business pressures to influence how the data were reported – they want the standards of success to be determined before the outcome must be interpreted).

So, fine, there’s a whole different proof-statement conversation about when to use characterizations and when to use Design Verification and Design Validation protocol execution results, and I’ll do another post on this topic. But, what about those situations where the original feasibility work was following a discovery path, and you’re pretty sure the data are solid, and you can’t afford to recreate the experiments later, just to satisfy a pre-approval requirement?

The key here will be to describe the rationale for using characterization data as a decision basis, and justify the rationale by showing that the characterization was rigorous. Or, maybe, using just the rigorously-derived outcomes as your decision basis. You’ll do this by writing up the informally-constituted steps that created the feasibility data, just as though there had been a pre-approved protocol in place, including expected and actual outcomes, the whole nine yards. In the introduction to this document, you’ll make clear that the material has been created in arrears. With a little forethought as to the acceptance standards for such a statement, your Quality System can allow the use of feasibility data as proof support for product design decisions or change impact determinations.

In conversation, your people will say “the gravity flux was not affected by the change because we were only changing the photo-density attachment bracket.” In a written description, each of the hidden assertions in this conversational tidbit will be parsed out and illuminated:

For the Anti-Gravity Generator to maintain the correct gravity field (see requirement 123.1), the gravity flux has to be X at time Y (see requirement 123.2). In March of 2008, prototypes 42 through 57 exhibited gravity flux of X in every experimental iteration (see pages 1001-5096 in redbook 456), irrespective whether the experimenter installed the production photo-density attachment bracket, part number 98765 rev A, or the experimental bracket, part number 1-98765, version 1.

Because this characterization study showed the gravity flux to be insensitive to differences in the photo-density attachment bracket when the gravity beam focus conformed to Gravity Beam Focus Specification, document 765, rev B, verification of the system with the modified photo-density attachment bracket, part number 98765 rev M, will consist of measuring the gravity beam focus in a shipment-ready Anti-Gravity Generator Subassembly, part number 91234, rev C. See Test Procedure 82, Gravity Beam Focus Measurement .

So what this quickly-crafted blurb says is

1. I know what the critical subsystem has to do (requirements), and how well it has to do those things (Critical Quality Attributes, CQAs, best expressed as requirements constraints, which are a subject of a different post).

2. In the past, somebody showed that the original part has been proven in the past to be interchangeable with other parts, without affecting the critical subsystem, provided that the CQAs are satisfied. This characterization was documented, including the inputs, procedure and outputs of it. The documentation is stored in a controlled location where I can prove that it hasn’t been lost or altered.

3. Here’s how I’ll use that historical information to devise a test of the modified part (this plan doesn’t involve setting up that horribly expensive Supernova With Black hole experiment we had to do during feasibility development). Now, I’ll just test the defining CQA using a procedure I’ve already approved, and that I can afford to use.

4. Without reading any other document, the reader can get a sense of the workflow, the intentions of the proof statement, and the level of rigor.

So, practice the Tao Of The Operating Fiction Of Linear Thinking. Focus more on what was challenged and proven and be careful to avoid concentrating only on from. The hidden philosophy in this idea is that your job is to ensure the outcome, and describe it in a way that someone else can visualize.

As with any mystical Tao, this one is not simple. With CAPA as an example, a moment of contemplation will lead you to realize that CAPAs need their whole record-investigate-risk-correction-corrective-action-preventive-action cycle to be executed. The point of this Tao is not that you can skip the thinking, is that your organization can run cleaner and lighter by communicating the thinking in a way that points the reader of your readers to real understanding of what happened and what the impact and significance of the events were.

Bureaucracy tends not to grow when the stakeholder groups, eg developers, testers, investigators, QA assessors, can develop justified confidence in each other.

So, emphasize outcomes and true understanding, not just the appearance of consistency.

In order develop your product, prove it, build some units, and get them out the door – ISO 13485 calls this journey “Product Realization” – there are 5 top-level processes you’re going to have execute. The Quality System Regulation (“QSR”) has words for the major Product Realization processes, including some terms that other industries don’t use:

1. Concept
2. Feasibility
3. Product Development  (I’m going to stash Design Validation and Design Verification inside Product development, for now)
4. Design Transfer
5. Manufacture
6. Launch or Release

It’s possible to either dramatically under-call your effort for each of these top-level processes, OR to get totally wrapped around the axle until your organization can’t move. In my first few postings, I’d like to mention some major problems that crop up in deciding where to put your development effort for Quality Systems and business processes.

The QSR and FDA Guidance Documents, plus some things you’ll hear inspectors say, might lead you to believe that FDA wants you to live your life in a waterfall model, surrounded by an immobilizing bureaucracy that demands perfect consistency in every dimension of human existence.

Do not be fooled! Nobody wants this, with the exception of a few misguided people who are suffering from having inherited a bureaucracy gene. This gene makes them like things that other people despise. As we progress through these posts, there will be therapeutic help for these folks, and for their co-dependent enablers.

You’ve already faced up to the fact that Medical Devices require greater information controls and more documentation than many other systems. The benefit of acceptance is that your total cost of operation will remain as low as possible, and you will first do no harm. Once you accept this requirement, however, the dark side manifests if your company falls in love with the ideas of documentation, control and consistency, and strangles itself.

It’s critical to strike the appropriate balance.

These therapeutic posts are aimed at helping you understand how to stay on the lean-and-mean side of the narrow and delicate boundary between compliant practice and slow, horrible death by bureaucracy.

More soon!

Brad