"Quality is not an act. It is a habit." - Aristotle (384 BC - 322 BC)
"Testing is a skill. While this
may come as a surprise to some people, it is a simple fact."
- Mark Fewster & Dorothy Graham
Course Outline327 Course Info 2017.pdf
Office: Goodwin 534
Email: joshuad at cs [dot] queensu.ca
Office hours: Wed 13:00–14:00 + by appointment
Course Summary & Study Guide (PDF)
nl48 at queensu·ca
jl245 at queensu·ca
cmt5 at queensu·ca
jw198 at queensu·ca
Old exams are available in the Queen's Exambank
(Schedule subject to change as course progresses)
Software Quality Assurance
An introductory course in the practical aspects of software quality
except Tuesday 14 Nov.: Kinesiology 100
and Wednesday 15 Nov.: Stirling Auditorium
Lectures plus a wide range of library and WWW resources (for the main course content)
CISC 327 Course Readings - available at the Queen's Bookstore
(see also the list of reference books and websites below)
There are no formal tutorials.
Assignment advising times Tuesdays 15:00-16:00,
Wednesdays 16:30-18:00 (+ 16:00-16:30 to be confirmed),
There are no scheduled labs in CISC 327, but you will require significant team lab time outside class to carry out your project assignments
Introduction to Quality Assurance,
Introduction: Course info. What is Quality? What is Quality Assurance? Software Quality Assurance. Formal methods, testing, inspection, metrics. Achieving software quality.
Software Process I: Quality in context. Software process activities. The Waterfall model. The Prototyping model. Evolutionary development.
Software Process II: The Spiral model. The Iterative Development Process (IDP). The Object Oriented Development Process.
References: Kan ch. 1, Software Quality Page
References: Kan ch. 2
Optional reading: Royce 1970
(very old but interesting)
(links to papers may only be accessible from the Queen's network)
References: Sommerville ch. 2, 26
- Fill in the blank: "Know what you're doing", "know what you should be doing", "know how to ____________________________"
- What are the four fundamental process activities?
- What are some drawbacks and benefits of the waterfall model? the spiral model? etc.
Software Process Evaluation,
Software Process Evaluation: Software process improvement. The Defect Prevention Process (DPP). Software quality standards. Maturity models, CMM, SPR. Baldrige Quality Award, ISO 9000.
Extreme Programming I: What is XP? Why is it called extreme? Characteristics of XP. Addressing risks before they arise.
Extreme Programming II: XP in Practice: The planning game, small releases, metaphor, simplicity, refactoring, pair programming, standards.
Course Project Kickoff: Course project phases. Details of course project requirements. Assignment #1.
References: Kan ch. 2 (2.7-2.8)
Assignment #0: Choose teams and platforms - due Tuesday week 3
References: Beck ch. 1
Reading: Beck ch. 2
Assignment #1: Create test suite - due Week 4
Optional reading: Curly braces and goto fail
- List two advantages of the XP practice "On-site Customer".
- XP practice 4, "Simplicity", favours designing for today over accounting for future needs. How might this lead to wasted work?
- We talked a little about safety-critical systems like Therac-25. Do you think XP would be a good process model for safety-critical software? Why or why not?
Intro to Systematic Testing
Tue.: Introduction to Systematic Testing I: Validation and Verification. Levels of Testing. Unit, integration, system, acceptance testing.
Wed.: More systematic testing + review for mini-exam
Mini-Exam #1, Friday, 29 Sept. in class
covers lectures 1-7 (QA, Process, XP)
Assignment #0 due Tuesday, 26 Sept.
References: Beck ch. 10
Reading: Beck ch. 11 and 12
References: Sommerville, ch. 8, The Software Test Page
Testing Methods - Black Box Methods: Black box vs. white box testing. Black box methods. Black Box method 1 - functionality coverage. Requirements partitioning. Experimental design. Choosing test inputs.
Black box method 2 - input coverage testing. Exhaustive testing. Input partitioning. Shotgun testing. Input partition/shotgun hybrid. Robustness testing. Boundary testing.
Black box method 3 - Output coverage testing. Exhaustive output testing. Output partitioning. Handling multiple input/output streams/files. Black box methods at different levels. Gray box testing.
Assignment #1 due Tue Oct. 10th
Assignment #2: Initial (untested!) implementation of Front End - due Thu Oct. 19th
Testing Methods: Black Box Methods (cont'd), White Box Methods
Black box unit testing (gray box testing). Test harnesses and stubs. Assertions in test automation, tools. Black box class testing (interface / object-oriented testing). Traces. Implementing assertions. Black box integration testing.
Testing Methods - White Box Methods: White box vs. black box. Role and kinds of white box testing. Code injection. Implementation: source, executable and sampling. White box static analysis.
Code coverage methods. Statement analysis methods: statement coverage, basic block coverage. Decision analysis methods: decision (branch) coverage, condition coverage, loop coverage, recursion depth coverage.
References: Lamb ch. 13, Trace specifications;
van Vliet ch. 13.6, Fault-based Techniques
Testing Methods: Code coverage, Mutation testing
Code coverage - decision analysis methods (cont'd). Path coverage. Data coverage methods. Value coverage, data flow coverage, interface coverage.
Mutation testing: Definition and role. Mutants: value, decision, statement mutations. Examples and coverage.
Continuous Testing: Software maintenance: corrective, adaptive and perfective maintenance. Continuous testing methods: functionality, failure and operational testing.
References: van Vliet ch. 13.5 Coverage-based Techniques
Assignment #2 due
Assignment #3 (PRELIMINARY): Front End acceptance due Week 8
Continuous Testing, Mini-Exam #2
Regression testing: Purpose, method. Establishing and maintaining a regression test set. Observable artifacts: choosing, maintaining, normalizing, differencing. Version signatures. Regression test harnesses. Case Study: the TXL interpreter. Regression test organization, signatures and differencing for the TXL interpreter. Kinds of observed artifacts: functionality, performance, internal diagnostic. Advantages and disadvantages of regression testing.
REVIEW for Mini-Exam #2
Mini-Exam #2, Friday Oct. 27th,
covers Lectures 8-16 (black box, white box, code coverage)
References: Sommerville ch. 8
References: Regression Testing Basics
Oct 30-Nov 3
Software Inspection: Introduction, reviews, walkthroughs and inspections. Inspection in the software process. Formal (Fagan) inspections: roles, reviewers. Code inspections: efficiency, cost effectiveness. Benefits of inspection. Role of inspection in quality control.
Inspection processes: Planning, orientation, preparation, review meeting, rework, verification. Inspection on your own - the Personal Software Process (PSP). Effective inspections.
Code Inspections: Techniques: checklists, paraphrasing, walkthroughs. Lightweight code inspection practices, XP. Heavyweight inspection practices, Cleanroom development.
Assignment #3 due Friday Nov. 3rd
Assignment #4: Back End initial implementation due Week 10
Inspection (cont'd), Review,
Code inspection in XP: Pair programming, code refactoring. Refactoring process, catalogs and rules. Continuous design improvement.
References: Gilb & Graham ch. 3 Overview of Software Inspection, O'Regan ch. 2 Overview of Fagan Inspections
References: Java code inspection checklist, C++ code inspection checklist, Lions Commentary on Unix, The Story of Unix, Cleanroom tutorial.
References: Wake ch. 2 What is Refactoring?, Refactoring example.
Miscellaneous bad things ("Piece of Crap"),
Measurement and Metrics
LECTURES IN DIFFERENT ROOMS:
Tuesday 14 Nov.: Kinesiology 100
Wednesday 15 Nov.: Stirling Auditorium
Introduction to Software Metrics: Software quality metrics, what they are, what they are for. Measurement basics - entities, attributes, measures. Assessment and prediction. Prediction models. A framework for software measurement.
Product quality metrics. External metrics - faults, failures, defects. Defect density metric. Internal metrics - LOC, functionality, complexity. Complexity metrics - Halstead Software Science, McCabe cyclomatic complexity, flowgraph metrics.
Process metrics - predicting software cost. COCOMO effort and time prediction. Regression based cost estimation. Specification-based size metrics. Function Points, FP analysis.
Assignment #4 due
Assignment #5 : Back End testing due Week 11
References: Nullable Reference Types in C#
References: Sommerville ch. 23, Project Planning. Otago Software Metrics Research Lab, U. Magdeburg Software Metrics Lab
References: Complexity Metrics and Models, Hacettepe U., McCabe and Associates Home Page
CSE COCOMO page, NASA COCOMO page, International Function Point User's Group
Introduction to Security: Technical and user security. The principle of least privilege. Examples of exploits. Buffer overrun (overflow) exploits. The 1980s and 1990s: Morris worm, early Mac viruses, macro viruses. The Heartbleed vulnerability. The ongoing dumpster fire of buffer overruns: "INTEL-SA-00086". Severity ratings; the Common Vulnerability Scoring System.
(Friday) USAT course evaluations
Heartbleed in context: OpenSSL software process - lack of inspections, excessive scope, inadequate staffing. Language-based security: Memory safety, refinement types. Web applications: SQL code injection attacks; sanitizing input; parameter attacks. Character encodings: the fun never stops.
Code for Lecture 28: bufcopter.c
(actually from the end of Week 10)
Reading: Thompson, Reflections on Trusting Trust
(1984 Turing Award lecture)
Assignment #5 due Thursday, Nov. 23
Assignment #6: Integration and Delivery due Week 12
References: Sommerville ch. 11, 12, Dependability and security.
Nov 27-Dec 1
Review, Mini-Exam #4, Course Summary & Review
Mini-Exam #4: Lectures 24–29b
Course Summary & Review: Software Process - Software Testing - Software Inspection - Software Metrics - Web Application Security
Assignment #6 due
(Scheduling may be subject to change as course progresses)
Lectures 1-7: Introduction and Process
Quality assurance definitions. Software process models - Waterfall, Prototyping, Evolutionary, Spiral, IDP, OOAD. Advantages and drawbacks. Software process evaluation - DPP, Baldrige,ISO 9000. eXtreme Programming.
Lectures 8–16: Testing
Systematic testing definitions. Black box methods. White box methods.
(Scheduling may be subject to change as course progresses)
Assignments are organized into a multi-stage software project that will be carried out in teams. This year's project TBA.
Choose teams by Tuesday, Week 3
Project Advising: See "General Information" above
A1: Front End Requirements Tests
In XP fashion, precisely specify requirements for the Front End as a set of explicit test inputs and expected outputs.
due Week 4
A2: Front End Rapid Prototype
Quickly create first implementation of the Front End demonstrating basic functionality.
due Week 6
A3: Front End Requirements Testing
Refine Front End implementation to acceptable product, adapting to handle all A1 requirements tests.
due Week 8
A4: Back End Rapid Prototype
Quickly create first implementation of the Back End demonstrating basic functionality.
due Week 10
A5: Back End Unit Testing
Practice unit testing on a subset of the units of the Back End implementation.
due Week 11
A6: Integration and Delivery
due Week 12
Refine Back End implementation to handle interaction with Front End, demonstrate on Front End requirements tests.
The University of Edinburgh guide to the Unix command line programming environment.
The Infionline guide to MS-DOS (Windows) command line script programming, by Terry Newton.
Steve Parker's online guide to Bourne shell / Bash shell scripting.
Kan, Metrics and Models in Software Quality Engineering, Addison Wesley 1995.
General reference on traditional software models, quality and metrics.
Gilb & Graham, Software Inspection, Addison Wesley 1993.
Reference on traditional software inspection.
Sommerville, Software Engineering, Addison Wesley 1996.
General reference on software engineering processes and procedures.
Succi & Marchesi, eXtreme Programming Examined, Addison Wesley 2000.
Wake, eXtreme Programming Explored, Addison Wesley 2000.
Additional references on eXtreme Programming ideas and methods.
Jeffries, Anderson & Hendrikson, eXtreme Programming Installed, Addison Wesley 2000.
Software Quality Web Sites
Software testing and quality links aimed at industrial Information Technology (IT) professionals.
Software Quality Assurance history and definitions page.
The NASA Software Quality Assurance website, with standards, procedures and checklists used at NASA.
The proceedings of WISE'01, the first international workshop on software inspection, from McMaster University.
ISO 9000 standard for computer software development and maintenance processes and procedures.
Maturity questionnaire used in SEI CMM assessments.