This is the foundation page for the topic of Verification Methods.
It seems that the classical methods of verification do not adequately serve their intended purpose when employed using modern system developmental management methods. To rectify this apparent oversight, a more complete list of soon-to-be standard verification methods has been compiled.
The Classical Methods1 are as follows:
1) Verification by Inspection. Employs direct observation of the item without the use of intrusive instrumentation. Applicable if and only if the characteristic specified is both quantifiable and externally observable.
2) Verification by Test. The indirect observation of the item using intrusive instrumentation, fixtures or external excitation that may alter the behavior of the item. Applicable if and only if the characteristic specified is quantifiable and the test method has a known effect on the item’s behavior.
3) Verification by Demonstration. The direct observation of the item for unquantified functionality without use of intrusive instrumentation, fixtures, or excitation. Applicable if and only if the characteristic specified is externally observable and need not be quantified2.
4) Verification by Analysis3. The algorithmic inference of a characteristic using models. Applicable if and only if the subject characteristic is verified to have valid dependencies on other characteristics that are themselves amenable to some combination of inspection, demonstration, or test4.
It should be obvious to any reasonably experienced engineer5 that the four classical approaches above share a common weakness in today’s flexible environment: they have explicit, technically relevant rules for applicability.
The following extended methods (more recently identified) attempt to redress this critical weakness. Apparently applicable under all circumstances, these have been shown to defer funding requirements beyond the current fiscal year, and, in many cases, result in significant promotion for the sophisticated engineers and managers capable of employing them.
5) Verification by Wishful Thinking: Application of concerted imagination on the part of the observer(s) to verify that the item is thought to meet the specified requirement.
6) Verification by Intimidation: Application of alpha-dominant characteristics to verify that no interested subordinate party considers it worth the risk of professional suicide to point out that the item does not meet the specified requirement.
7) Verification by Ridicule: An advanced variant of Verification by Intimidation employed to verify that a sufficient degree of humiliation has been reached by dissenting individuals for a consensus to be declared that the item has met the specified requirement.
8) Verification by Postponement: Employs classical guerrilla-warfare delaying tactics to verify that the requirement is really necessary before allocating the resources necessary to determine the ability of the item to meet it.
9) Verification by Anamorphosis: Logical abstraction of camouflage methods to verify that the item appears to meet the requirement when interrogated from pre-specified points of view.
Anamorphosis: n., 1. A drawing that presents a distorted image that appears in natural form under certain conditions, as when reflected from a mirror. 2. The method of producing such a drawing.
10) Verification by Neglect: Knowledge that verification is required coupled with an intent to ignore it and hope that no one discovers it.
The efficiency of this method can be measured in terms of “baumgartners” (a proprietary unit of idleness). One baumgartner-hour is the inverse of the number of calories expended by a 26 year-old, 220 pound male human while sleeping for one hour…but can only be accumulated while awake.
Note that, per the definition, a larger man (at a given level of cardiopulmonary fitness) can accumulate baumgartners more quickly than a smaller man. By that same token, a more fit man accumulates baumgartners more slowly than a less fit man. We also note that this unit of measure might apply only to males of the species. We are unsure if there is any appropriate measure for females, and would not presume to guess.
By the way…if you are keeping track of the rate at which you are accumulating baumgartners…you aren’t accumulating them as fast as you think you are.
11) Verification by Amphigory: Sufficient repetition of nonsensical freeform verse and tomfoolery to convince the audience that verification is adequately addressed with the current, non-existent approach.
Amphigory: n., 1. A meaningless rigamorole, as of nonsense verses. 2. A non-sensical parody.
12) Verification by Legerdemain: The artful manipulation of facts, data and emotion to contrive the appearance of verification in spite of common sense and the paucity of objective evidence.
Legerdemain: n., 1. skill in or practice of feats of magic, jugglery, etc.: sleight of hand. 2. Trickery, deception. 3. Any artful trick.
Based on a suggestion by Erik Illi.
13) Verification by Testimonial: Proposed acceptance of fervent expression of fondly held belief on the part of a recognized expert in the field. 6
14) Verification by Schedule Variance Analysis (SVA): Stipulation of adequacy on the basis of scheduled completion dates having been reached, and tacit acceptance of results to date. This is achieved by setting the BCWS7 equal to the BCWP8 on the due date, verifying that the schedule available for verification has been consumed, therefore all of the work must have been done because we’ve run out of time.
15) Verification by Cost Variance Analysis (CVA): Similar to Verification by SVA, this stipulates adequacy of verification when all of the available funds have been spent. In this case, BCWP9 is set equal to ACWP10 when the budget line goes to zero, verifying that all of the work must have been done because all of the money’s gone.
16) Verification by Surprise: This condition arises when an event occurs during test or analysis that reveals a previously unknown requirement. Subsequent assessment of the event demonstrates the degree of compliance.
Usually attended by remarks like “Whoa! How’d THAT fish get on the line?”
17) Verification by Coprophagy: This situation occurs when data of high technical credibility are produced (as by test or analysis) without regard (on the part of the producer) for their end-use. Verification occurs when the data are improperly used to show compliance without regard to the suitability of underlying assumptions and conditions under which the data were produced.
Copraphagous: adj., feeding on dung, as certain beetles. – coprophagy, n.
18) Verification by Sixpack: The tactical use of convivial inebriation to convince the auditor of the adequacy of the design as proffered.
19) Verification by Forn….nevermind. This one needs no further explanation, anyway.
20) Verification by OMS-1: The assertion that the requirements might as well be considered satisfied, since the system has been committed to operation.
OMS-1: first burn of the NSTS Orbital Maneuver System (OMS), which inserted the Orbiter Vehicle (shuttle) into orbit. Later burns of the OMS circularized and/or raised the orbit, then de-orbited at the end of mission.
Based on a suggestion by Ted Kenney.
21) Verification by Wheelstop: The operational demonstration that the requirements must have been met, since the system has successfully completed the mission.
Wheelstop: the event during landing that defined the end of a shuttle mission.
Based on a suggestion by Ted Kenney.
22) Verification by Divination: Art or practice that seeks to foresee or foretell future compliance events or discover hidden knowledge that the requirement is met; usually by the interpretation of omens or by the aid of supernatural powers.
The innards examined and knucklebones cast in this method are usually those of the cognizant engineering staff, resulting in an extraordinarily high on-schedule percentage.
23) Verification by Execution: “There comes a time in the life of every program when you have to shoot the engineers & go into production.”
24) Verification by Omphaloskepsis: An intersection of Verification by Divination with Verification by Testimonial involving the contemplation of their own navel (or other suitable body part) by a SME; that is, requiring no assets outside the practitioner’s own corporeal being in order to determine compliance.
Omphaloskepsis: n, contemplation of one’s navel as an aid to meditation.
Based on a suggestion by Mario D’Anunzio.
25) Verification by Revelation: A spontaneous variant of Verification by Divination; that is, requiring no art or practice beyond supplication to the revealing deity.
Based on a suggestion by James Hinkle.
26) Immaculate Verification: The requirement is claimed to have been verified, but no human being can be found who will admit to having done it.
Usually attended by remarks like “what happened in Test, stays in Test”.
Based on a suggestion by John Ward.
27) Verification by Revulsion: The product is so hideous and revolting, the entire cadre of auditors votes for compliance in order to relieve themselves of any further burden of review.
Based on a suggestion by Stan Jones.
28) Verification by Outlier: Repeated random draw from pre-production stores until a sample can be found that meets the specified requirement. (Sometimes implemented as a “selected set” of piece parts in order to create an assembly that meets the requirement.)
Derived from an evolution-based concept suggested by Derek van Alen.
29) Verification by Irredention: Verification is accomplished when it is determined that the offending component is not actually part of the Configuration Item being assessed for compliance with the specified requirements.
Irredentism: n, a political principle or policy directed toward the incorporation of irredentas within the boundaries of their historically or ethnically related political unit; irredenta: a territory historically or ethnically related to one political unit but under the political control of another. The term dates to 1888, and is derived from the Latin “Italia Irredenta” (Italy Unredeemed), referring to those ethnically Italian areas that were, for one reason or another, not joined into the modern Kingdom of Italy. Irrendention is, therefore, the act or principle of creating an irredenta (or, more properly, an irredentum) by removing a sub-unit from its historically or ethnically related parent.
30) Verification by Sub-sampling: Inverse implementation of Shannon’s Sampling Theorem in order to miss those data capable of reducing the signal-to-noise ratio such that an accurate determination of compliance can be made.
31) Verification by Sublimation: A departure from Wishful Thinking in which the reviewer expects to see a particular thing, and subconsciously disregards evidence (or lack thereof) to the contrary.
Sublimate: v (transitive),
1 a: SUBLIME
1b: to improve or refine as if by subliming (archaic)
2: to divert the expression of (an instinctual desire or impulse) from its unacceptable form to one that is considered more socially or culturally acceptable.
32) Verification at Random: Data produced without definite aim, direction, rule, or methodology. Typically achieved when verification requirements are written without regard to applicability of the Classical Methods given the discretionary technologies and functions available for selection into the design.
Usually attended by remarks like “what happened in Vegas, stays in Vegas”.
Based on a collective notion at the FCS Good-bye Luncheon for Bob Ahlquist, James Hinkle, and Alice Starratt in January, 2008.
33) Verification by Aphorism: A departure from Verification by Testimonial in which the repetition of irrelevant but pithy phrases distracts the reviewer into disregard for the detailed technical content submitted for review, leading to approval of the design on the basis of the submitter’s evident sagacity.
Aphorism: n, 1) a concise statement of a principle; 2) a terse formulation of a truth or sentiment : ADAGE.
Based on an unintentional suggestion by Terese Stevens.
34) Unrequited Verification: Verification of any requirement subsequently turning out to have been so irrelevant that no party can be found who is interested enough to review the results.
35) Verification by Attrition: All representatives of the purchasing or certifying authority retire from the field due to exhaustion, illness, or other incapacitation. Verification occurs when the customer’s contracts officer can no longer find anybody with enough energy to contend against the data tendered as verifying the requirement in question.
Based on a suggestion by Ralph Galetti.
36) Verification by Crapathy: No individual can be found who cares whether the requirement can be (or has been) verified…in fact, nobody can be found who cares whether the Design is successful at all.
Derived during an email exchange of utter despair with Ralph Galetti, in which we both realized we were so far beyond apathy that we didn’t even give a crap…hence “crapathy, crapathetic”.
37) Verification by Plan: Documentation of intent to satisfy an allocated requirement, where the contract stipulates that submittal of the plan is sufficient evidence of compliance.
Paraphrased from the “Working Together Agreement” between A Major Aerospace Manufacturer and (at least) one of its first-tier suppliers on a certain commercial aircraft. Yeah…I couldn’t believe it either…but apparently we were serious enough to put it in writing. When I read it, I thought I had died and gone to heaven…I am thoroughly convinced I walked among the angels that day!
38) Verification by Procedure: Documentation of the process by which an allocated requirement could be satisfied such that “…any person of the required level of competent arrive (sic) at the same results…”
Paraphrased from the “Working Together Agreement” between A Major Aerospace Manufacturer and (at least) one of its first-tier suppliers on a certain commercial aircraft. More angels11!
39) Verification by Design Review: No paraphrase will suffice…“When the Supplier is required to show compliance to requirements by discussing or presenting material…”
Quoted from the “Working Together Agreement” between A Major Aerospace Manufacturer and (at least) one of its first-tier suppliers on a certain commercial aircraft. A surfeit of angels11!
40) Verification by Auto-orchiectomy: The persistent, irrational refusal on the part of the customer to accept any evidence of compliance with the requirements until the proffering developer has performed some ultimate self-sacrificial act of contrite submission.
I cannot, for the life of me, understand how I failed to come up with this one before.
41) Verification by Categorical Cloning: Application of Category Theory12 to selection of verification details (including the method) based on perceived resemblance between the technical matter at hand and some other historical development activity.
Technically, not a verification method per se, but a process for identifying one; that is, a pointer to a method rather than the method itself. Identified by the proposed mis-use of qualification temperature requirements developed for the ISS Common Berthing Mechanism during development of a superficially similar item more than a decade later.
42) Verification by Buffer Thrashing: The energy required to manage changes in the interpretation of verification details exceeds the energy being spent on the verification activity itself; the requirement might as well be considered verified, since no more substantive work can ever get done.
Based on a suggestion by the esteemed (put possibly exasperated) John Hodges, with a semantic assist by Jeff Myer.
43) Verification by Reverse Specification: A sub-discipline within the concept of Reverse Engineering, in which the item is built before any requirements are written. The likelihood of successful verification is extremely high, but the hardware may not do anything originally envisioned when the project began.
Based on a suggestion by Russell Graves (“Verification by Post Qualification Specification”).
44) Verification by Morosophistry: A complex, detailed, precise closeout narrative based on unfounded assumption.
Based on a suggestion by Russell Graves (“Verification by Computer-aided Guess”). See also Matthew 7:26-27.
45) Verification by Thimblerig: The repeated reassignment of verification responsibility from team to team until one of them finally submits something for customer consideration.
Based on a suggestion by Russell Graves (“Verification by Customer Frustration”).
46) Verification by End Run: Obtaining formal concurrence of compliance from some obscure customer in the hope that the authorized representative will not notice the tactic until final contract action is complete.
Based on a suggestion by Russell Graves (“Verification by Alternate Customer Concurrence”).
47) Verification by Retrograde Motion: A frame-of-reference-shift from Verification by Anamorphosis in which the kinematic variable is distance rather than orientation such that a negative margin can be rounded out of perceived existence.
Based on a suggestion by Russell Graves (“Verification by Management Direction”). Not to be confused by management retreat from a position taken in public.
48) Verification by Request to Admit: Verification is achieved by requesting the development customer to concur with the supplier’s assertion of satisfaction based on no evidence whatsoever. Lack of timely objection by the development customer legally constitutes evidence of compliance.
Based on a suggestion by Tom Kozar and Dave Rollins, derived from the following supplier response: “Based on design review, the intent of the requirement is captured and no objective evidence of non-compliance has been found.”
The notion defined here relies on the legal concept that requests for admission (RFA) can obviate the need for presentation of actual evidence at trial. Unlike nolle prosequi (decision to not prosecute), the RFA can be initiated by defendant, but is binding on both parties once accepted by the court. The specific variant in Mr. Kozar’s original example also dances very close to a hoping-for-of-confusion between the concepts of “evidence of absence” (which is forensically acceptable) and “absence of evidence” (not forensically acceptable), but I was unable to invent a succinct phraseology that adequately captured that!
49) Verification by Blinn-Phong Reflection: Employment of state-of-the-art CGI to produce depictions of the intended or expected changes in the state1 of a system or component, based on the results of some underlying model. Verification occurs when the reviewer finds the animation so visually impressive as to obviate the need for model validation.
Based on a suggestion by Wes Bruner and Justin McFatter.
Footnotes- From informal definitions developed by the author in 1992 to demonstrate how to tell when each method should be (or could be) applied.[↩]
- That is, a pure functional requirement having no Measure of Performance.[↩]
- Simulation is sometimes broken out separately. The definition of Analysis given here is adequate to cover that concept.[↩]
- This second-order verification is often called “model validation”.[↩]
- And possibly to the occasional project manager, as well.[↩]
- Also known as “Verification by SME”.[↩]
- BCWS: Baseline Cost of Work Scheduled[↩]
- BCWP: Baseline Cost of Work Performed[↩]
- Baseline Cost of Work Performed[↩]
- Actual Cost of Work Performed[↩]
- See also “Verification by Plan”, above.[↩][↩]
- Sort of an odd-ball conglomeration of algebra, topology, set theory, and chicken soup…sometimes referred to as “general abstract nonsense”. (I kid you NOT!)[↩]