"Quality is not an act. It is a habit." - Aristotle (384 BC - 322 BC)
"Testing is a skill. While this
may come as a surprise to some people, it is a simple fact."
- Mark Fewster & Dorothy Graham
Course Outline327 Course Info 2018.pdf
Office: Goodwin 534
Email: joshuad at cs [dot] queensu.ca
Office hours: Tue 13:00–14:00, Wed 13:00–14:00 + by appointment
Course Summary & Study Guide (PDF)
<keeler [at] cs.queensu.ca>
lb149 at queensu·ca
jhml at queensu·ca
dvl at queensu·ca
Some old exams are available in the Queen's Exambank
(Schedule subject to change as course progresses)
Software Quality Assurance
An introductory course in the practical aspects of software quality
Mon 15:30–16:30 Walter Light 205 Thu 16:30–17:30 Walter Light 205 Fri 10:30–11:30 Dupuis Auditorium
Lectures plus a range of library and web resources (for the main course content)
CISC 327 Course Readings—available at the Queen's Bookstore
(see also the list of reference books and websites below)
There are no formal tutorials.
Assignment advising times Mondays 14:00-15:30
There are no scheduled labs in CISC 327, but you will require significant team lab time outside class to carry out your project assignments
Introduction to Quality Assurance,
Introduction: Course info. What is Quality? What is Quality Assurance? Software Quality Assurance. Formal methods, testing, inspection, metrics. Achieving software quality.
Software Process I: Quality in context. Software process activities. The Waterfall model. The Prototyping model. Evolutionary development.
References: Kan ch. 1, Software Quality Page
References: Kan ch. 2
Optional reading: Royce 1970
(very old but interesting)
(links to papers may only be accessible from the Queen's network)
References: Sommerville ch. 2, 26
- Fill in the blank: "Know what you're doing", "know what you should be doing", "know how to ____________________________"
- What are the four fundamental process activities?
- What are some drawbacks and benefits of the waterfall model? the spiral model? etc.
Software Process Evaluation,
Software Process II: The Spiral model. The Iterative Development Process (IDP). The Object Oriented Development Process.
Software Process Evaluation: Software process improvement. The Defect Prevention Process (DPP). Software quality standards. Maturity models, CMM, SPR. Baldrige Quality Award, ISO 9000.
Extreme Programming I: What is XP? Why is it called extreme? Characteristics of XP. Addressing risks before they arise.
Course Project Kickoff: Course project phases. Details of course project requirements. Assignment #1.
References: Kan ch. 2 (2.7-2.8)
Assignment #0: Choose teams and platforms - due Tuesday week 3
References: Beck ch. 1
Reading: Beck ch. 2
Assignment #1: Create test suite—due Week 5
Optional reading: Curly braces and goto fail
- List two advantages of the XP practice "On-site Customer".
- XP practice 4, "Simplicity", favours designing for today over accounting for future needs. How might this lead to wasted work?
- We talked a little about safety-critical systems like Therac-25. Do you think XP would be a good process model for safety-critical software? Why or why not?
Intro to Systematic Testing
Extreme Programming II: XP in Practice: The planning game, small releases, metaphor, simplicity, refactoring, pair programming, standards.
Thu.: LECTURE CANCELLED :
Fri.: Introduction to Systematic Testing: Validation and Verification. Levels of Testing. Unit, integration, system, acceptance testing. + Review for mini-exam
Assignment #0 due Friday, 21 Sept.
References: Beck ch. 10
Reading: Beck ch. 11 and 12
References: Sommerville, ch. 8, The Software Test Page
Intro to Systematic Testing / Testing Methods
Mini-Exam #1, Monday, 24 Sept. in class
covers lectures 1–5 (QA, Process, XP)
Testing Methods - Black Box Methods: Black box vs. white box testing. Black box methods. Black Box method 1 - functionality coverage. Requirements partitioning. Experimental design. Choosing test inputs.
Black box method 2 - input coverage testing. Exhaustive testing. Input partitioning. Shotgun testing. Input partition/shotgun hybrid. Robustness testing. Boundary testing.
Assignment #1 due Fri. Oct. 5th
Assignment #2: Initial (untested!) implementation of Front End - due Mon. Oct. 15th
Testing Methods: Black Box Methods (cont'd), White Box Methods
Black box method 3 - Output coverage testing. Exhaustive output testing. Output partitioning. Handling multiple input/output streams/files. Black box methods at different levels. Gray box testing.
Black box unit testing (gray box testing). Test harnesses and stubs. Assertions in test automation, tools. Black box class testing (interface / object-oriented testing). Traces. Implementing assertions. Black box integration testing.
References: Lamb ch. 13, Trace specifications;
van Vliet ch. 13.6, Fault-based Techniques
Testing Methods: Code coverage
Thanksgiving Holiday Monday
(Prof. Cordy guest lecture.) Testing Methods - White Box Methods: White box vs. black box. Role and kinds of white box testing. Code injection. Implementation: source, executable and sampling. White box static analysis.
(Prof. Cordy guest lecture.) Code coverage methods. Statement analysis methods: statement coverage, basic block coverage. Decision analysis methods: decision (branch) coverage, condition coverage, loop coverage.
References: van Vliet ch. 13.5 Coverage-based Techniques
Mutation testing, Continuous testing
(Prof. Zulkernine guest lecture.) Code coverage - decision analysis methods (cont'd). Path coverage. Data coverage methods. Value coverage, data flow coverage, interface coverage.
Mutation testing: Definition and role. Mutants: value, decision, statement mutations. Examples and coverage.
Continuous Testing: Software maintenance: corrective, adaptive and perfective maintenance. Continuous testing methods: functionality, failure and operational testing.
REVIEW for Mini-Exam #2
Assignment #2 due
References: Sommerville ch. 8
References: Regression Testing Basics
Mini-Exam #2, Fall mid-term break
Mini-Exam #2, Monday Oct. 22nd,
covers Lectures 7-16 (black box, white box, code coverage)
Assignment #4: Back End initial implementation due Week 10
Oct. 29–Nov. 2
Regression testing, inspection (code smells), security
Regression testing: Purpose, method. Establishing and maintaining a regression test set. Observable artifacts: choosing, maintaining, normalizing, differencing. Version signatures. Regression test harnesses. Case Study: the TXL interpreter. Regression test organization, signatures and differencing for the TXL interpreter. Kinds of observed artifacts: functionality, performance, internal diagnostic. Advantages and disadvantages of regression testing.
Code inspection in XP: Pair programming, code refactoring. Refactoring process, catalogs (code smells) and rules. Continuous design improvement.
Introduction to Security: Technical and user security. The principle of least privilege. Examples of exploits. Buffer overrun (overflow) exploits. The 1980s and 1990s: Morris worm, early Mac viruses, macro viruses. The Heartbleed vulnerability.
Assignment #3 due Wednesday, Oct. 31st
References: Wake ch. 2 What is Refactoring?, Refactoring example.
Assignment #5 due Thursday, Nov. 22
Assignment #6: Integration and Delivery due Week 13
References: Sommerville ch. 11, 12, Dependability and security.
Code for Lecture 19-1: bufcopter.c
NO 327 LECTURE Friday, Nov. 9 (Remembrance Day)
The ongoing dumpster fire of buffer overruns: "INTEL-SA-00086". Severity ratings; the Common Vulnerability Scoring System.
Heartbleed in context: OpenSSL software process - lack of inspections, excessive scope, inadequate staffing. Language-based security: Memory safety, refinement types. Web applications: SQL code injection attacks; sanitizing input; parameter attacks. Character encodings: the fun never stops.
Assignment #5 : Back End testing due Week 12
References: Nullable Reference Types in C#
Mini-Exam #3; Inspections
Mini-Exam #3, Monday Nov. 12,
covers Lectures 17, 18, 19-1, 19-2, 19-3 (mutation testing, continuous testing, regression testing, security)
Software Inspection: Introduction, reviews, walkthroughs and inspections. Inspection in the software process. Formal (Fagan) inspections: roles, reviewers. Code inspections: efficiency, cost effectiveness. Benefits of inspection. Role of inspection in quality control.
Code Inspections: Techniques: checklists, paraphrasing, walkthroughs. Lightweight code inspection practices, XP. Heavyweight inspection practices, Cleanroom development.
References: Gilb & Graham ch. 3 Overview of Software Inspection, O'Regan ch. 2 Overview of Fagan Inspections
References: Java code inspection checklist, C++ code inspection checklist, Lions Commentary on Unix, The Story of Unix, Cleanroom tutorial.
Measurement and metrics
Inspection processes: Planning, orientation, preparation, review meeting, rework, verification. Inspection on your own - the Personal Software Process (PSP). Effective inspections.
Introduction to Software Metrics: Software quality metrics, what they are, what they are for. Measurement basics - entities, attributes, measures. Assessment and prediction. Prediction models. A framework for software measurement.
Product quality metrics. External metrics - faults, failures, defects. Defect density metric. Internal metrics - LOC, functionality, complexity. Complexity metrics - Halstead Software Science, McCabe cyclomatic complexity, flowgraph metrics.
References: Sommerville ch. 23, Project Planning. Otago Software Metrics Research Lab, U. Magdeburg Software Metrics Lab
References: Complexity Metrics and Models, Hacettepe U., McCabe and Associates Home Page
CSE COCOMO page, NASA COCOMO page, International Function Point User's Group
Process metrics / Review, Mini-Exam #4, Course Summary & Review
Process metrics - predicting software cost. COCOMO effort and time prediction. Regression based cost estimation. Specification-based size metrics. Function Points, FP analysis.
Mini-Exam #4, Thursday, 29 Nov. in class
covers lectures 19-0 and 20 through 27:
Course Summary & Review: Software Process - Software Testing - Software Inspection - Software Metrics - Web Application Security
Assignment #6 due Friday, Nov. 30th
(Scheduling may be subject to change as course progresses)
Lectures 1–5: Introduction and Process
Quality assurance definitions. Software process models - Waterfall, Prototyping, Evolutionary, Spiral, IDP, OOAD. Advantages and drawbacks. Software process evaluation - DPP, Baldrige,ISO 9000. eXtreme Programming.
Lectures 8–16: Testing
Systematic testing definitions. Black box methods. White box methods.
Lectures 16-18, 19, 19-1 through 19-3:
Mutation testing, continuous testing, regression testing; security.
Lectures 19-0 and 20 through 27:
Inspection and Metrics
(Scheduling may be subject to change as course progresses)
Assignments are organized into a multi-stage software project that will be carried out in teams.
Choose teams by Tuesday, Week 3
Project Advising: See "General Information" above
A1: Front End Requirements Tests
In XP fashion, precisely specify requirements for the Front End as a set of explicit test inputs and expected outputs.
A2: Front End Rapid Prototype
Quickly create first implementation of the Front End demonstrating basic functionality.
due Week 6
A3: Front End Requirements Testing
Refine Front End implementation to acceptable product, adapting to handle all A1 requirements tests.
due Week 8
A4: Back End Rapid Prototype
Quickly create first implementation of the Back End demonstrating basic functionality.
due Nov. 9th
A5: Back End Unit Testing
Practice unit testing on a subset of the units of the Back End implementation.
due Week 11
A6: Integration and Delivery
due Week 12
Refine Back End implementation to handle interaction with Front End, demonstrate on Front End requirements tests.
The University of Edinburgh guide to the Unix command line programming environment.
The Infionline guide to MS-DOS (Windows) command line script programming, by Terry Newton.
Steve Parker's online guide to Bourne shell / Bash shell scripting.
Kan, Metrics and Models in Software Quality Engineering, Addison Wesley 1995.
General reference on traditional software models, quality and metrics.
Gilb & Graham, Software Inspection, Addison Wesley 1993.
Reference on traditional software inspection.
Sommerville, Software Engineering, Addison Wesley 1996.
General reference on software engineering processes and procedures.
Succi & Marchesi, eXtreme Programming Examined, Addison Wesley 2000.
Wake, eXtreme Programming Explored, Addison Wesley 2000.
Additional references on eXtreme Programming ideas and methods.
Jeffries, Anderson & Hendrikson, eXtreme Programming Installed, Addison Wesley 2000.
Software Quality Web Sites
Software testing and quality links aimed at industrial Information Technology (IT) professionals.
Software Quality Assurance history and definitions page.
The NASA Software Quality Assurance website, with standards, procedures and checklists used at NASA.
The proceedings of WISE'01, the first international workshop on software inspection, from McMaster University.
ISO 9000 standard for computer software development and maintenance processes and procedures.
Maturity questionnaire used in SEI CMM assessments.