CISC 327 - Fall 2018

Contacts - General Information - Lectures - Mini-Exams - Assignments - Resources

Software Quality Assurance

"Quality is not an act. It is a habit." - Aristotle (384 BC - 322 BC)

"Testing is a skill. While this may come as a surprise to some people, it is a simple fact."
- Mark Fewster & Dorothy Graham

Course Outline

327 Course Info 2018.pdf

Contacts

Professor
        Joshua Dunfield

Office:    Goodwin 534
Email:     joshuad at cs [dot] queensu.ca
Office hours:     Tue 13:00–14:00, Wed 13:00–14:00 + by appointment

Final Examination:
none

Course Summary & Study Guide (PDF)

Teaching Assistants
       Chris Keeler
       Lucas Bullen
       Johnathan Lee
       Daniel Lucia

Email:
     <keeler [at] cs.queensu.ca>
lb149 at queensu·ca
jhml at queensu·ca
dvl at queensu·ca

Some old exams are available in the Queen's Exambank

General Information

  (Schedule subject to change as course progresses)

Course Title

Software Quality Assurance

Course Info    

An introductory course in the practical aspects of software quality

327 Course Info 2018.pdf

Lectures

Mon15:30–16:30Walter Light 205
Thu16:30–17:30Walter Light 205
Fri10:30–11:30Dupuis Auditorium

Textbook

Lectures plus a range of library and web resources (for the main course content)

CISC 327 Course Readings—available at the Queen's Bookstore

(see also the list of reference books and websites below)

Tutorials

There are no formal tutorials.

Assignment advising times Mondays 14:00-15:30
Wednesdays 16:30-18:00
Goodwin 241

Laboratories

There are no scheduled labs in CISC 327, but you will require significant team lab time outside class to carry out your project assignments

Lectures

Week 1
Sept. 6–7

Introduction to Quality Assurance,
Software Process

Lecture 1, Lecture 2

Introduction: Course info. What is Quality? What is Quality Assurance? Software Quality Assurance. Formal methods, testing, inspection, metrics. Achieving software quality.

Software Process I: Quality in context. Software process activities. The Waterfall model. The Prototyping model. Evolutionary development.

References: Kan ch. 1, Software Quality Page

References: Kan ch. 2

Optional reading: Royce 1970
(very old but interesting)

(links to papers may only be accessible from the Queen's network)

References: Sommerville ch. 2, 26

Study questions:
  • Fill in the blank: "Know what you're doing", "know what you should be doing", "know how to ____________________________"
  • What are the four fundamental process activities?
  • What are some drawbacks and benefits of the waterfall model? the spiral model? etc.

Week 2
Sept. 10-14

Software Process Evaluation,
Extreme Programming

Lecture 3, Lecture 4, Lecture 5,

Software Process II: The Spiral model. The Iterative Development Process (IDP). The Object Oriented Development Process.

Software Process Evaluation: Software process improvement. The Defect Prevention Process (DPP). Software quality standards. Maturity models, CMM, SPR. Baldrige Quality Award, ISO 9000.

Extreme Programming I: What is XP? Why is it called extreme? Characteristics of XP. Addressing risks before they arise.

Course Project Kickoff: Course project phases. Details of course project requirements. Assignment #1.

References: Kan ch. 2 (2.7-2.8)

Optional reading: NIST Baldrige report, ISO 9000-3, CMM

Assignment #0: Choose teams and platforms - due Tuesday week 3

References: Beck ch. 1
Reading: Beck ch. 2

Assignment #1: Create test suite—due Week 5

Optional reading: Curly braces and goto fail

Study questions:
  • List two advantages of the XP practice "On-site Customer".
  • XP practice 4, "Simplicity", favours designing for today over accounting for future needs. How might this lead to wasted work?
  • We talked a little about safety-critical systems like Therac-25. Do you think XP would be a good process model for safety-critical software? Why or why not?

Week 3
Sept. 17–21

Intro to Systematic Testing

Lecture 6, no Lecture 7, Lecture 8 / Lecture 8a (REVIEW)

Extreme Programming II: XP in Practice: The planning game, small releases, metaphor, simplicity, refactoring, pair programming, standards.

Thu.: LECTURE CANCELLED :

Fri.: Introduction to Systematic Testing: Validation and Verification. Levels of Testing. Unit, integration, system, acceptance testing. + Review for mini-exam

Assignment #0 due Friday, 21 Sept.

References: Beck ch. 10
Reading: Beck ch. 11 and 12

References: Sommerville, ch. 8, The Software Test Page

Week 4
Sept. 24–28

Intro to Systematic Testing / Testing Methods

Mini-Exam #1, Lecture 9, Lecture 10

Mini-Exam #1, Monday, 24 Sept. in class
covers lectures 1–5 (QA, Process, XP)
Hints!

Testing Methods - Black Box Methods: Black box vs. white box testing. Black box methods. Black Box method 1 - functionality coverage. Requirements partitioning. Experimental design. Choosing test inputs.

Black box method 2 - input coverage testing. Exhaustive testing. Input partitioning. Shotgun testing. Input partition/shotgun hybrid. Robustness testing. Boundary testing.

Assignment #1 due Fri. Oct. 5th

Assignment #2: Initial (untested!) implementation of Front End - due Mon. Oct. 15th

Week 5
Oct. 1–5

Testing Methods: Black Box Methods (cont'd), White Box Methods

Lecture 11, Lecture 12, Lecture 13

Black box method 3 - Output coverage testing. Exhaustive output testing. Output partitioning. Handling multiple input/output streams/files. Black box methods at different levels. Gray box testing.

Black box unit testing (gray box testing). Test harnesses and stubs. Assertions in test automation, tools. Black box class testing (interface / object-oriented testing). Traces. Implementing assertions. Black box integration testing.

References: Lamb ch. 13, Trace specifications;
van Vliet ch. 13.6, Fault-based Techniques

Week 6
Oct. 11–12

Testing Methods: Code coverage

Thanksgiving Holiday Monday

Lecture 14, Lecture 15

(Prof. Cordy guest lecture.) Testing Methods - White Box Methods: White box vs. black box. Role and kinds of white box testing. Code injection. Implementation: source, executable and sampling. White box static analysis.

(Prof. Cordy guest lecture.) Code coverage methods. Statement analysis methods: statement coverage, basic block coverage. Decision analysis methods: decision (branch) coverage, condition coverage, loop coverage.

References: van Vliet ch. 13.5 Coverage-based Techniques

Answers to questions about Assignment #2

Week 7
Oct. 15–19

Mutation testing, Continuous testing

Lecture 16, Lecture 17, Lecture 18, Lecture review2 / 19a

(Prof. Zulkernine guest lecture.) Code coverage - decision analysis methods (cont'd). Path coverage. Data coverage methods. Value coverage, data flow coverage, interface coverage.

Mutation testing: Definition and role. Mutants: value, decision, statement mutations. Examples and coverage.

Continuous Testing: Software maintenance: corrective, adaptive and perfective maintenance. Continuous testing methods: functionality, failure and operational testing.

REVIEW for Mini-Exam #2

Assignment #2 due

References: Sommerville ch. 8

References: Regression Testing Basics

Week 8
Oct. 22

Mini-Exam #2

Mini-Exam #2, Fall mid-term break

Mini-Exam #2, Monday Oct. 22nd,
covers Lectures 7-16 (black box, white box, code coverage)

Assignment #4: Back End initial implementation due Week 10

Week 9
Oct. 29–Nov. 2

Regression testing, inspection (code smells), security

Lecture 19, Lecture 19-0, Lecture 19-1

Regression testing: Purpose, method. Establishing and maintaining a regression test set. Observable artifacts: choosing, maintaining, normalizing, differencing. Version signatures. Regression test harnesses. Case Study: the TXL interpreter. Regression test organization, signatures and differencing for the TXL interpreter. Kinds of observed artifacts: functionality, performance, internal diagnostic. Advantages and disadvantages of regression testing.

Code inspection in XP: Pair programming, code refactoring. Refactoring process, catalogs (code smells) and rules. Continuous design improvement.

Introduction to Security: Technical and user security. The principle of least privilege. Examples of exploits. Buffer overrun (overflow) exploits. The 1980s and 1990s: Morris worm, early Mac viruses, macro viruses. The Heartbleed vulnerability.

Assignment #3 due Wednesday, Oct. 31st

References: Wake ch. 2 What is Refactoring?, Refactoring example.

References:
Heartbleed:
Wikipedia, xkcd #1354;
INTEL-SA-00086 vulnerabilities:
Intel's page,
security expert Matthew Garrett's tweets,
CVSS for the first vulnerability listed

Assignment #5 due Thursday, Nov. 22

Assignment #6: Integration and Delivery due Week 13

References: Sommerville ch. 11, 12, Dependability and security.

Code for Lecture 19-1: bufcopter.c

Week 10
Nov. 5–9

Security

NO 327 LECTURE Friday, Nov. 9 (Remembrance Day)

Lecture 19-2, Lecture 19-3, Lecture review3

The ongoing dumpster fire of buffer overruns: "INTEL-SA-00086". Severity ratings; the Common Vulnerability Scoring System.

Heartbleed in context: OpenSSL software process - lack of inspections, excessive scope, inadequate staffing. Language-based security: Memory safety, refinement types. Web applications: SQL code injection attacks; sanitizing input; parameter attacks. Character encodings: the fun never stops.

Assignment #5 : Back End testing due Week 12

References: Nullable Reference Types in C#

Reading: Thompson, Reflections on Trusting Trust
(1984 Turing Award lecture)

Assignment #4 due Friday, Nov. 9

Week 11
Nov. 12–16

Mini-Exam #3; Inspections

Mini-Exam #3, Lecture 20, Lecture 21, Lecture 22

Mini-Exam #3, Monday Nov. 12,
covers Lectures 17, 18, 19-1, 19-2, 19-3 (mutation testing, continuous testing, regression testing, security)

Software Inspection: Introduction, reviews, walkthroughs and inspections. Inspection in the software process. Formal (Fagan) inspections: roles, reviewers. Code inspections: efficiency, cost effectiveness. Benefits of inspection. Role of inspection in quality control.

Code Inspections: Techniques: checklists, paraphrasing, walkthroughs. Lightweight code inspection practices, XP. Heavyweight inspection practices, Cleanroom development.

References: Gilb & Graham ch. 3 Overview of Software Inspection, O'Regan ch. 2 Overview of Fagan Inspections

References: Java code inspection checklist, C++ code inspection checklist, Lions Commentary on Unix, The Story of Unix, Cleanroom tutorial.

Week 12
Nov. 19–23

Measurement and metrics

Lecture 24, Lecture 25, Lecture 26

Inspection processes: Planning, orientation, preparation, review meeting, rework, verification. Inspection on your own - the Personal Software Process (PSP). Effective inspections.

Introduction to Software Metrics: Software quality metrics, what they are, what they are for. Measurement basics - entities, attributes, measures. Assessment and prediction. Prediction models. A framework for software measurement.

Product quality metrics. External metrics - faults, failures, defects. Defect density metric. Internal metrics - LOC, functionality, complexity. Complexity metrics - Halstead Software Science, McCabe cyclomatic complexity, flowgraph metrics.

 

References: Sommerville ch. 23, Project Planning. Otago Software Metrics Research Lab, U. Magdeburg Software Metrics Lab

References: Complexity Metrics and Models, Hacettepe U., McCabe and Associates Home Page

CSE COCOMO page, NASA COCOMO page, International Function Point User's Group

Week 13
Nov. 26–30

Process metrics / Review, Mini-Exam #4, Course Summary & Review

Lecture 27, Lecture review4

Process metrics - predicting software cost. COCOMO effort and time prediction. Regression based cost estimation. Specification-based size metrics. Function Points, FP analysis.

Mini-Exam #4, Thursday, 29 Nov. in class
covers lectures 19-0 and 20 through 27:
inspections, metrics


Course Summary & Review: Software Process - Software Testing - Software Inspection - Software Metrics - Web Application Security

Assignment #6 due Friday, Nov. 30th

Mini-Exams

  (Scheduling may be subject to change as course progresses)

Mini-Exam #1

Lectures 1–5: Introduction and Process

Quality assurance definitions. Software process models - Waterfall, Prototyping, Evolutionary, Spiral, IDP, OOAD. Advantages and drawbacks. Software process evaluation - DPP, Baldrige,ISO 9000. eXtreme Programming.

40 minutes

Week 4

Mini-Exam #2

Lectures 8–16: Testing

Systematic testing definitions. Black box methods. White box methods.

40 minutes

Week 8

Mini-Exam #3

Lectures 16-18, 19, 19-1 through 19-3:
Mutation testing, continuous testing, regression testing; security.

40 minutes

Week 11

Mini-Exam #4

Lectures 19-0 and 20 through 27:
Inspection and Metrics

40 minutes

Week 13

Assignments

  (Scheduling may be subject to change as course progresses)  

Project
Description

QIES

Assignments are organized into a multi-stage software project that will be carried out in teams.

Project Description

Assignment #0

Choose teams by Tuesday, Week 3

Project Advising: See "General Information" above

Assignment #1

A1: Front End Requirements Tests

In XP fashion, precisely specify requirements for the Front End as a set of explicit test inputs and expected outputs.

Assignment #1

due 2018–10–05

Answers to questions about Assignment #1

Assignment #2

A2: Front End Rapid Prototype

Quickly create first implementation of the Front End demonstrating basic functionality.

Assignment #2

due Week 6

Answers to questions about Assignment #2

Assignment #3

A3: Front End Requirements Testing

Refine Front End implementation to acceptable product, adapting to handle all A1 requirements tests.

Assignment #3

due Week 8

Answers to questions about Assignment #3

Assignment #4

A4: Back End Rapid Prototype

Quickly create first implementation of the Back End demonstrating basic functionality.

Assignment #4

due Nov. 9th

Answers to questions about Assignment #4

Assignment #5

A5: Back End Unit Testing

Practice unit testing on a subset of the units of the Back End implementation.

Assignment #5

due Week 11

Answers to questions about Assignment #5

 

Assignment #6

A6: Integration and Delivery

Assignment #6

due Week 12

Answers to questions about Assignment #6


Refine Back End implementation to handle interaction with Front End, demonstrate on Front End requirements tests.

Resources

Command Line
Programming

University of Edinburgh Unix Guide

The University of Edinburgh guide to the Unix command line programming environment.

Infionline Windows Batch Command Line Programming Guide

The Infionline guide to MS-DOS (Windows) command line script programming, by Terry Newton.

Unix/Linux bash Shell Scripting Tutorial

Steve Parker's online guide to Bourne shell / Bash shell scripting.

Reference Books

Kan, Metrics and Models in Software Quality Engineering, Addison Wesley 1995.

General reference on traditional software models, quality and metrics.

Gilb & Graham, Software Inspection, Addison Wesley 1993.

Reference on traditional software inspection.

Sommerville, Software Engineering, Addison Wesley 1996.

General reference on software engineering processes and procedures.

Succi & Marchesi, eXtreme Programming Examined, Addison Wesley 2000.

Wake, eXtreme Programming Explored, Addison Wesley 2000.

Additional references on eXtreme Programming ideas and methods.

Jeffries, Anderson & Hendrikson, eXtreme Programming Installed, Addison Wesley 2000.

Software Quality Web Sites

CompInfo Software Testing and Quality Control Page

Software testing and quality links aimed at industrial Information Technology (IT) professionals.

Software Quality Assurance History and Definitions

Software Quality Assurance history and definitions page.

NASA Software Quality Assurance Page

The NASA Software Quality Assurance website, with standards, procedures and checklists used at NASA.

Workshop on Inspection in Software Engineering

The proceedings of WISE'01, the first international workshop on software inspection, from McMaster University.

International Standards Organization standard 90003 (IEC ISO 90003 2004)

ISO 9000 standard for computer software development and maintenance processes and procedures.

Software Engineering Institute Capability Maturity Model (CMM)

Maturity questionnaire used in SEI CMM assessments.