How QUELCE Works
QUELCE synthesizes scenario building, Bayesian Belief Network (BBN) modeling, and Monte Carlo simulation into an estimation method that quantifies uncertainties, allows subjective inputs, visually depicts influential relationships among change drivers and outputs, and assists with the explicit description and documentation underlying an estimate. It uses scenario analysis and design structure matrix (DSM) techniques to limit the combinatorial effects of multiple interacting program change drivers to make modeling and analysis more tractable. Representing scenarios as BBNs enables sensitivity analysis, exploration of alternatives, and quantification of uncertainty.
The BBNs and Monte Carlo simulation are then used to predict variability of what become the inputs to existing, commercially available cost estimation methods and tools. As a result, interim and final cost estimates are embedded within clearly defined confidence intervals. The method can be described as a series of eleven steps, summarized in the following sections1. Our recent SEI technical report, CMU/SEI-2011-TR-026, elaborates further on the method and its application.
Step 1: Identify Program Change Drivers
The identification of program change drivers is best accomplished by the experts who provide programs with information for consideration in cost estimation. Workshops with DoD contractors, domain experts, and former DoD program managers are used to identify drivers that could affect program costs. These experts should consider all aspects of a program that might change (and affect cost) during the program’s lifecycle—particularly given the new information developed during the Technology Development Phase in preparation of Milestone B. The Probability of Program Success (POPS) factors used by the Navy and Air Force can be used to start the discussion.
Step 2: Identify States of Program Change Drivers
In the workshops, experts are asked to brainstorm ideas about the status of each program change driver. The specific, assumed state as proposed by the Materiel Solution is identified and labeled as the nominal state. Experts then brainstorm about possible changes in the condition of each driver that may occur during the program lifecycle. The experts identify possible changes that might occur to the nominal state and use their best judgment for the probability that the nominal state will change.
Step 3: Identify Cause-and-Effect Relationships for Dependency Matrix
Once the changed condition— referred to as potential driver states—are fully identified, participants subjectively evaluate the cause and effect relationships among the drivers. Expert judgment is applied to rank the causal effects. A matrix is developed that provides the relationship between nominal and dependent states and contains the conditional probability that one will affect the other, but not the impact of the change. This exercise can result in a very large number of program change drivers and states identified for an MDAP.
Figure 2: Example Dependency Matrix After DSM Transformation
Step 4: Reduce the Dependency Matrix Using a Design Structure Matrix
Using the Design Structure Matrix (DSM) technique the change drivers can be reduced to an efficient set that has the most potential impact to cost. The DSM technique is a well-established method to reduce complicated dependency structures to a manageable size. An example of a dependency matrix after DSM transformation created during an SEI pilot workshop is provided in Figure 2.
Step 5: Construct a BBN Using the Reduced Matrix
Using the program change drivers derived from Step 4 and their cause and effect relationships established in Step 3, a BBN is constructed. This BBN is a probabilistic model that dynamically represents the drivers and their relationships, as envisioned by the program domain experts. Figure 3 depicts an abbreviated visualization of a BBN, in which the circled nodes represent program change drivers and the arrows represent either cause and effect relationships or leading indicator relationships. This example shows that a change in the Mission and CONOPS driver most likely will cause a change to the Capability Analysis driver, which in turn will likely effect a change in the Key Performance Parameter (KPP) driver and subsequently the Technical Challenge outcome factor. The three outcome factors (Product Challenge, Project Challenge, and Size Growth) are then used to predict some of the input values for traditional cost estimation models.
Figure 3: Example BBN
Step 6: Populate BBN Nodes with Conditional Probabilities
Conditional probabilities are assigned to the nodes (drivers) in the BBN. Each node can assume a variety of states, each of which has an associated likelihood identified by the domain experts. This allows the calculation of outcome distributions on the variables.