CISC 327 - Fall 2017

Contacts - General Information - Lectures - Mini-Exams - Assignments - Resources

Software Quality Assurance

"Quality is not an act. It is a habit." - Aristotle (384 BC - 322 BC)

"Testing is a skill. While this may come as a surprise to some people, it is a simple fact."
- Mark Fewster & Dorothy Graham

Course Outline

327 Course Info 2017.pdf

Contacts

Professor
        Joshua Dunfield

Office:    Goodwin 534
Email:     joshuad at cs [dot] queensu.ca
Office hours:     Wed 13:00–14:00 + by appointment

Final Examination:
none

Course Summary & Study Guide (PDF)

Teaching Assistants
      Niki Lin
      Julie Lycklama
      Christopher Thomas
      Jia Wan

Email:
     nl48 at queensu·ca
     jl245 at queensu·ca
     cmt5 at queensu·ca
     jw198 at queensu·ca

Old exams are available in the Queen's Exambank

General Information

 (Schedule subject to change as course progresses)

Course Title

Software Quality Assurance

Course Info    

An introductory course in the practical aspects of software quality

327 Course Info 2017.pdf

Lectures

Tue 08:30–09:30
Wed 10:30–11:30
Fri 09:30–10:30

Kingston 201
except Tuesday 14 Nov.: Kinesiology 100
and Wednesday 15 Nov.: Stirling Auditorium

Textbook

Lectures plus a wide range of library and WWW resources (for the main course content)

CISC 327 Course Readings - available at the Queen's Bookstore

(see also the list of reference books and websites below)

Tutorials

There are no formal tutorials.

Assignment advising times Tuesdays 15:00-16:00,
Wednesdays 16:30-18:00 (+ 16:00-16:30 to be confirmed),
Goodwin 241

Laboratories

There are no scheduled labs in CISC 327, but you will require significant team lab time outside class to carry out your project assignments

Lectures

Week 1
Sep 11-15

Introduction to Quality Assurance,
Software Process

Lecture 1, Lecture 2, Lecture 3

Introduction: Course info. What is Quality? What is Quality Assurance? Software Quality Assurance. Formal methods, testing, inspection, metrics. Achieving software quality.

Software Process I: Quality in context. Software process activities. The Waterfall model. The Prototyping model. Evolutionary development.

Software Process II: The Spiral model. The Iterative Development Process (IDP). The Object Oriented Development Process.

References: Kan ch. 1, Software Quality Page

References: Kan ch. 2

Optional reading: Royce 1970
(very old but interesting)

(links to papers may only be accessible from the Queen's network)

References: Sommerville ch. 2, 26

Study questions:
  • Fill in the blank: "Know what you're doing", "know what you should be doing", "know how to ____________________________"
  • What are the four fundamental process activities?
  • What are some drawbacks and benefits of the waterfall model? the spiral model? etc.

Week 2
Sep 18-22

Software Process Evaluation,
Extreme Programming

Lecture 4, Lecture 5, [there is no lecture 6], Lecture 7

Software Process Evaluation: Software process improvement. The Defect Prevention Process (DPP). Software quality standards. Maturity models, CMM, SPR. Baldrige Quality Award, ISO 9000.

Extreme Programming I: What is XP? Why is it called extreme? Characteristics of XP. Addressing risks before they arise.

Extreme Programming II: XP in Practice: The planning game, small releases, metaphor, simplicity, refactoring, pair programming, standards.

Course Project Kickoff: Course project phases. Details of course project requirements. Assignment #1.

References: Kan ch. 2 (2.7-2.8)

Optional reading: NIST Baldrige report, ISO 9000-3, CMM

Assignment #0: Choose teams and platforms - due Tuesday week 3

References: Beck ch. 1
Reading: Beck ch. 2

Assignment #1: Create test suite - due Week 4

Optional reading: Curly braces and goto fail

Study questions:
  • List two advantages of the XP practice "On-site Customer".
  • XP practice 4, "Simplicity", favours designing for today over accounting for future needs. How might this lead to wasted work?
  • We talked a little about safety-critical systems like Therac-25. Do you think XP would be a good process model for safety-critical software? Why or why not?

Week 3
Sep 25-29

Intro to Systematic Testing

Lecture 8, Lecture 9 / Lecture 9a (REVIEW)
Mini-Exam #1

Tue.: Introduction to Systematic Testing I: Validation and Verification. Levels of Testing. Unit, integration, system, acceptance testing.

Wed.: More systematic testing + review for mini-exam

Mini-Exam #1, Friday, 29 Sept. in class
covers lectures 1-7 (QA, Process, XP)
Hints!

Assignment #0 due Tuesday, 26 Sept.

References: Beck ch. 10
Reading: Beck ch. 11 and 12

References: Sommerville, ch. 8, The Software Test Page

Week 4
Oct 2-6

Testing Methods

Lecture 10, Lecture 11, Lecture 12

Testing Methods - Black Box Methods: Black box vs. white box testing. Black box methods. Black Box method 1 - functionality coverage. Requirements partitioning. Experimental design. Choosing test inputs.

Black box method 2 - input coverage testing. Exhaustive testing. Input partitioning. Shotgun testing. Input partition/shotgun hybrid. Robustness testing. Boundary testing.

Black box method 3 - Output coverage testing. Exhaustive output testing. Output partitioning. Handling multiple input/output streams/files. Black box methods at different levels. Gray box testing.

Assignment #1 due Tue Oct. 10th

Assignment #2: Initial (untested!) implementation of Front End - due Thu Oct. 19th

Week 5
Oct 9-13

Testing Methods: Black Box Methods (cont'd), White Box Methods

Lecture 13, Lecture 14, Lecture 15

Black box unit testing (gray box testing). Test harnesses and stubs. Assertions in test automation, tools. Black box class testing (interface / object-oriented testing). Traces. Implementing assertions. Black box integration testing.

Testing Methods - White Box Methods: White box vs. black box. Role and kinds of white box testing. Code injection. Implementation: source, executable and sampling. White box static analysis.

Code coverage methods. Statement analysis methods: statement coverage, basic block coverage. Decision analysis methods: decision (branch) coverage, condition coverage, loop coverage, recursion depth coverage.

References: Lamb ch. 13, Trace specifications;
van Vliet ch. 13.6, Fault-based Techniques

Week 6
Oct 16-20

Testing Methods: Code coverage, Mutation testing

Lecture 16, Lecture 17, Lecture 18

Code coverage - decision analysis methods (cont'd). Path coverage. Data coverage methods. Value coverage, data flow coverage, interface coverage.

Mutation testing: Definition and role. Mutants: value, decision, statement mutations. Examples and coverage.

Continuous Testing: Software maintenance: corrective, adaptive and perfective maintenance. Continuous testing methods: functionality, failure and operational testing.

References: van Vliet ch. 13.5 Coverage-based Techniques

Answers to questions about Assignment #2

Assignment #2 due

Assignment #3 (PRELIMINARY): Front End acceptance due Week 8

Week 7
Oct 23-27

Continuous Testing, Mini-Exam #2

Lecture 19, Lecture review2 / 19a, Mini-Exam #2

Regression testing: Purpose, method. Establishing and maintaining a regression test set. Observable artifacts: choosing, maintaining, normalizing, differencing. Version signatures. Regression test harnesses. Case Study: the TXL interpreter. Regression test organization, signatures and differencing for the TXL interpreter. Kinds of observed artifacts: functionality, performance, internal diagnostic. Advantages and disadvantages of regression testing.

REVIEW for Mini-Exam #2

Mini-Exam #2, Friday Oct. 27th,
covers Lectures 8-16 (black box, white box, code coverage)

References: Sommerville ch. 8

References: Regression Testing Basics

Week 8
Oct 30-Nov 3

Inspection

Lecture 20, Lecture 21, Lecture 22

Software Inspection: Introduction, reviews, walkthroughs and inspections. Inspection in the software process. Formal (Fagan) inspections: roles, reviewers. Code inspections: efficiency, cost effectiveness. Benefits of inspection. Role of inspection in quality control.

Inspection processes: Planning, orientation, preparation, review meeting, rework, verification. Inspection on your own - the Personal Software Process (PSP). Effective inspections.

Code Inspections: Techniques: checklists, paraphrasing, walkthroughs. Lightweight code inspection practices, XP. Heavyweight inspection practices, Cleanroom development.

Assignment #3 due Friday Nov. 3rd

Assignment #4: Back End initial implementation due Week 10

Week 9
Nov 6-10

Inspection (cont'd), Review,
Mini-Exam #3

Lecture 23, Lecture review3 / 24

Code inspection in XP: Pair programming, code refactoring. Refactoring process, catalogs and rules. Continuous design improvement.

Mini-Exam #3

References: Gilb & Graham ch. 3 Overview of Software Inspection, O'Regan ch. 2 Overview of Fagan Inspections

References: Java code inspection checklist, C++ code inspection checklist, Lions Commentary on Unix, The Story of Unix, Cleanroom tutorial.

References: Wake ch. 2 What is Refactoring?, Refactoring example.

Week 10
Nov 13-17

Miscellaneous bad things ("Piece of Crap"),
Measurement and Metrics

LECTURES IN DIFFERENT ROOMS:
Tuesday 14 Nov.: Kinesiology 100
Wednesday 15 Nov.: Stirling Auditorium

Lecture 24, Lecture 25, Lecture 26, Lecture 27

Introduction to Software Metrics: Software quality metrics, what they are, what they are for. Measurement basics - entities, attributes, measures. Assessment and prediction. Prediction models. A framework for software measurement.

Product quality metrics. External metrics - faults, failures, defects. Defect density metric. Internal metrics - LOC, functionality, complexity. Complexity metrics - Halstead Software Science, McCabe cyclomatic complexity, flowgraph metrics.

Process metrics - predicting software cost. COCOMO effort and time prediction. Regression based cost estimation. Specification-based size metrics. Function Points, FP analysis.

Assignment #4 due

Assignment #5 : Back End testing due Week 11

References: Nullable Reference Types in C#

References: Sommerville ch. 23, Project Planning. Otago Software Metrics Research Lab, U. Magdeburg Software Metrics Lab

References: Complexity Metrics and Models, Hacettepe U., McCabe and Associates Home Page

CSE COCOMO page, NASA COCOMO page, International Function Point User's Group

Week 11
Nov 20-24

Security

Lecture 28, Lecture 29a, Lecture 29b

Introduction to Security: Technical and user security. The principle of least privilege. Examples of exploits. Buffer overrun (overflow) exploits. The 1980s and 1990s: Morris worm, early Mac viruses, macro viruses. The Heartbleed vulnerability. The ongoing dumpster fire of buffer overruns: "INTEL-SA-00086". Severity ratings; the Common Vulnerability Scoring System.

(Friday) USAT course evaluations

Heartbleed in context: OpenSSL software process - lack of inspections, excessive scope, inadequate staffing. Language-based security: Memory safety, refinement types. Web applications: SQL code injection attacks; sanitizing input; parameter attacks. Character encodings: the fun never stops.

Code for Lecture 28: bufcopter.c
(actually from the end of Week 10)

Reading: Thompson, Reflections on Trusting Trust
(1984 Turing Award lecture)

References:
Heartbleed:
Wikipedia, xkcd #1354;
INTEL-SA-00086 vulnerabilities:
Intel's page,
security expert Matthew Garrett's tweets,
CVSS for the first vulnerability listed

Assignment #5 due Thursday, Nov. 23

Assignment #6: Integration and Delivery due Week 12

References: Sommerville ch. 11, 12, Dependability and security.

Week 12
Nov 27-Dec 1

Review, Mini-Exam #4, Course Summary & Review

Lecture review4

Mini-Exam #4: Lectures 24–29b

Course Summary & Review: Software Process - Software Testing - Software Inspection - Software Metrics - Web Application Security

Mini-Exam #4

Assignment #6 due

Mini-Exams

  (Scheduling may be subject to change as course progresses)

Mini-Exam #1

Lectures 1-7: Introduction and Process

Quality assurance definitions. Software process models - Waterfall, Prototyping, Evolutionary, Spiral, IDP, OOAD. Advantages and drawbacks. Software process evaluation - DPP, Baldrige,ISO 9000. eXtreme Programming.

45 minutes

Week 3

Mini-Exam #2

Lectures 8–16: Testing

Systematic testing definitions. Black box methods. White box methods.

45 minutes

Week 7

Mini-Exam #3

 

45 minutes

Week 9

Mini-Exam #4

45 minutes

Week 12

Assignments

  (Scheduling may be subject to change as course progresses)  

Project
Description

TBA

Assignments are organized into a multi-stage software project that will be carried out in teams. This year's project TBA.

Project Description

Assignment #0

Choose teams by Tuesday, Week 3

Project Advising: See "General Information" above

Assignment #1

A1: Front End Requirements Tests

In XP fashion, precisely specify requirements for the Front End as a set of explicit test inputs and expected outputs.

Assignment #1

due Week 4

Answers to questions about Assignment #1

Assignment #2

A2: Front End Rapid Prototype

Quickly create first implementation of the Front End demonstrating basic functionality.

Assignment #2

due Week 6

Answers to questions about Assignment #2

Assignment #3

A3: Front End Requirements Testing

Refine Front End implementation to acceptable product, adapting to handle all A1 requirements tests.

Assignment #3

due Week 8

Answers to questions about Assignment #3

Assignment #4

A4: Back End Rapid Prototype

Quickly create first implementation of the Back End demonstrating basic functionality.

Assignment #4

due Week 10

Answers to questions about Assignment #4

Assignment #5

A5: Back End Unit Testing

Practice unit testing on a subset of the units of the Back End implementation.

Assignment #5

due Week 11

Answers to questions about Assignment #5

 

Assignment #6

A6: Integration and Delivery

Assignment #6

due Week 12

Answers to questions about Assignment #6


Refine Back End implementation to handle interaction with Front End, demonstrate on Front End requirements tests.

Resources

Command Line
Programming

University of Edinburgh Unix Guide

The University of Edinburgh guide to the Unix command line programming environment.

Infionline Windows Batch Command Line Programming Guide

The Infionline guide to MS-DOS (Windows) command line script programming, by Terry Newton.

Unix/Linux bash Shell Scripting Tutorial

Steve Parker's online guide to Bourne shell / Bash shell scripting.

Reference Books

Kan, Metrics and Models in Software Quality Engineering, Addison Wesley 1995.

General reference on traditional software models, quality and metrics.

Gilb & Graham, Software Inspection, Addison Wesley 1993.

Reference on traditional software inspection.

Sommerville, Software Engineering, Addison Wesley 1996.

General reference on software engineering processes and procedures.

Succi & Marchesi, eXtreme Programming Examined, Addison Wesley 2000.

Wake, eXtreme Programming Explored, Addison Wesley 2000.

Additional references on eXtreme Programming ideas and methods.

Jeffries, Anderson & Hendrikson, eXtreme Programming Installed, Addison Wesley 2000.

Software Quality Web Sites

CompInfo Software Testing and Quality Control Page

Software testing and quality links aimed at industrial Information Technology (IT) professionals.

Software Quality Assurance History and Definitions

Software Quality Assurance history and definitions page.

NASA Software Quality Assurance Page

The NASA Software Quality Assurance website, with standards, procedures and checklists used at NASA.

Workshop on Inspection in Software Engineering

The proceedings of WISE'01, the first international workshop on software inspection, from McMaster University.

International Standards Organization standard 90003 (IEC ISO 90003 2004)

ISO 9000 standard for computer software development and maintenance processes and procedures.

Software Engineering Institute Capability Maturity Model (CMM)

Maturity questionnaire used in SEI CMM assessments.