‘Real QA’

[Real Software and System Quality Assurance]
Manifesto
© By Tom at Gilb.com & [email protected] May 22 2009

Purpose of this document/manifesto:

Quality-Assurance (QA) in software worldwide has in fact degenerated into testing alone. Software/IT management has ignorantly allowed this to happen.
Of course many parts of the industry have been well-aware of more cost-effective ways of delivering required Quality in practice, but this has in fact been largely ignored; while granting very large Resources to testing alone.

It is time for a wakeup call!

This manifesto is here to tell the industry that;

1. testing alone is 10x more costly than doing Real QA.
2. testing alone is not good enough, this can and need not go on.
3. we know how to do real QA much much better than testing alone, using smarter upstream Engineering practices, based on design, prevention and upstream inspections.

QA Objectives

1. NO SURPRISES ASSURANCE: To allow management to understand release consequences fully, in relation to expectations, with the lowest costs, the lowest risks, and the lowest degree of surprises.
2. MEET EXPECTATIONS: To make sure that the project investors and sponsors get, at least, what they expect.
3. BUSINESS PERFORMANCE: To deliver the ‘business’ (or organisational) results envisaged and promised to project and programme supporters.
4. TECHNICAL PERFORMANCE: To Quantify and Measure the technical performance (including all System Quality Attributes) Attributes of the System.
5. CONSTRAINT COMPLIANCE: To ascertain that the System has not violated any specified constraints.
6. LONG TERM ASSURANCE: To give some assurance that the long term characteristics of the System are as planned. Things like adaptability, maintainability.
7. LEGAL COMPLIANCE: make sure the System is always compliant with legal and other compliance Policy items.

QA Strategies: valid QA Strategies include, but are never limited to:

1. Clear and quantified project and System Level objectives (Value-Requirements), at the Level of the organization, the Product (example a software package), and the System (all related aspects such as people, support, documentation, marketing). Using ‘Planguage’* to express the objectives clearly and fully.
2. Clear and detailed Impact and Cost estimated strategies for meeting the objectives. Rated and measured using Impact-Estimation* (Value-Decision) specification language. Strong Architecture design to meet multiple quantified objectives and constraints.
3. Quality-Control Reviews (Agile Inspections or Spec-QC *) of all forms of specification and documentation against sufficiently high Standards (Rules, Processes, and other types of Standards *) and higher Level Source Documents.
   The reviews will operate at early stages (leading measures) to
   1. motivate and teach engineers to utilize best practices.
   2. measure the degree to which every development and maintenance output actually meets the best practice adopted Standards.
   3. give a Basis for serious numeric Process entry and Exit conditions* for all Engineering Work outputs.
   4. give a partial data-driven Basis for continuous and immediate Process improvement (like Defect-Prevention-Process (DPP) and CMMI Level 5).
4. Rapid Evolutionary iteration, incremental value delivery to Stakeholders, data collection, feedback, analysis and change (as in Value-Delivery / Evo *).
   The purposes of this are to early, often, rapidly and throughout:
   1. be able to prioritise high value deliveries.
   2. validate that the teams are actually able to deliver value at all.
   3. learn about everything, and improve anything.
   4. enable all Quality-Assurance tactics to be tried and proven.
5. Automated and built in Measurement and detection of problems, including very direct real-time user and Stakeholder feedback to developers and engineers. Operating Before releases and eternally After releases.
6. Ongoing Fact Analysis: Software Engineering Accounting: capability of analysing, organisation wide and in the long term how all these things are working, so as to determine what works best and what the costs are.
7. oh yes, and not to forget, conventional testing, at its best.

- * as detailed in Gilb: Competitive-Engineering and Evo – or any better or equivalent specification method.
This manifesto is not about Gilb methods, they are mentioned here as examples so that no one can misunderstand what we mean by 'quality', 'inspections', and other concepts.

QA Principles (general ideas that guide us in detailed decisions)

1. Early Bird: QA must operate as early as possible to detect problems both in the current development/maintenance Process, and in the Product or System being engineered.
2. Quantification: quantification will always be preferred as a language to express any variable Idea associated with Product, System and Process ideas. It is always possible, and it is the only sound Basis for rational thinking by management, Engineering, and academic research.
3. Prevention: rather than tolerate eternal levels of problems to clean up later at 10x the Cost of doing it right in the first time, we will rapidly invest in a shift to defect prevention.
4. Rapid Feedback: we will position our activities to get rapid feedback (like this same week) so we can correct bad things as soon as possible, and so they cannot fester for months and years.

QA Values

1. Efficiency: we will always strive to find and implement the most Cost effective methods to assure long term System Quality.
2. Confidence: our assertions of confidence in System releases can be relied on totally, and will be very explicit about caveats, Assumptions, maximum deviations, and responsibilities if accepted.
3. Predictability: we will develop our ability to predict System Attributes based on early indicators (example: field bugs based on major defects in Requirement).
4. Leanness: we will constantly remove and avoid all activity that does not have clear measured value contributions in relation to its real Cost.
5. Perceptiveness: even when not directed by our Stakeholder to explicitly deal with them, we will anticipate all critical System Quality aspects.
6. Reality: we will not be driven by ideals, fads, customs, misunderstandings or illogical arguments. We will be directed by current local realities: how things Work for us now.
7. Delegation: avoiding micromanagement - we will practice extreme delegation for details and choice of design, and for Processes, to the people who do them daily. Management’s Role will be limited to setting clear measurable high Level objectives for products and Processes, and then enabling their staff to reach them.

The IT Management Role in Making Real QA Happen

The testers have not been leading the change to real and Complete QA. They ‘test’.
The developers have not done anything laudable either. They code.
Managers don’t seem to have a clue, but then nobody actually trained them to understand broad QA. They push deadlines, complain and pay.
The universities and their professors have no discernible honour in educating us in broad QA. They lag.

Nothing is going to change unless responsible management (CIO, CTO, if necessary CEO levels) is somehow illuminated about QA, and chooses to insist it is exploited to its fullest potential. This can be done in part through a QA Policy.

Definitions:

Manifesto

a public declaration of Policy and aims, esp. one issued Before an election by a political party or candidate.
ORIGIN mid 17th cent.; from Italian, from manifestare, from Latin, ‘make public,’ from manifestus ‘obvious’ (see manifest 1 ).

False QA

is calling your activity ‘QA’ when in fact you only do testing.

References:

Competitive Engineering

Gilb, Tom; A Handbook For Systems-Engineering, Requirements-Engineering, and Software-Engineering Using Planguage
ISBN 0750665076, 2005, Publisher; Elsevier Butterworth-Heinemann.

Evo

Gilb, Kai; Evolutionary Project Management & Product Development.

Signatories