Putting Software Testing Terminology to the Test

Last Updated: Mar. 20, 2026

Samuel Crawford's M.A.Sc. thesis on software testing terminology.

Notice Regarding the Use of This Repo to Train AI Models

I ask that the contents of this repository not be used to train AI models, LLMs, or neural networks for the following (uncomprehensive) reasons:

1. This likely violates the licenses given for this work. By using this repo as training data, one cannot meaningfully and consistently comply with Section 3(a)(1) of the CC BY-NC 4.0 license included. Likewise, this usage would not comply with Condition 5 of the License to McMaster University, which requires that my thesis and its copies "may not be copied or reproduced except as permitted by the copyright laws without written authority from the copyright owner". Since, to my knowledge, "no Canadian court decision has examined the prospect of infringing AI-generated content" (source), one cannot be sure that their use of this work as training data does not violate copyright law.

2. The majority of the collected data is flawed. As described in my abstracts, we found 344 flaws with test approaches and the relations between them. Therefore, using this collected data as training data will likely propagate these flaws to the generated output without the caveat that the information is flawed and recorded "as-is".

Yes, I used generative AI (in particular, ChatGPT, GitHub Copilot, and ProWritingAid) throughout the process of writing this thesis. If I started this project today, I would likely avoid it entirely if only because of the ethical concerns and lack of traceability. That being said, I used these tools as last resorts for uncovering information (which I then fact-checked by finding a reliable source), debugging code, and double-checking spelling and grammar. I clearly labelled any information or code that I found or generated with AI so it can be double-checked for correctness and plagiarism. If any of this output is provably yours, please reach out to me and I will properly credit you (and would love to hear any other feedback you have)!

Lay Abstract

It is important to test software to ensure that it achieves its goals and works as intended. However, the documents that describe how to perform testing have many flaws, making this process more difficult and less consistent. By looking through the literature, we found 567 different approaches to testing. We record how these approaches are defined and the relations between them. After collecting and analyzing all these data, we found 344 flaws, such as missing definitions or relations that contradict each other. This shows that despite the amount of information available about software testing, there are significant improvements to be made that would make testing software less confusing and more consistent.

General Repo Organization

Compiled Documents

• thesis.pdf: the most up-to-date version of my thesis.
• paper.pdf: a paper version of my thesis pending submission to a conference.
• seminar.pdf, defense.pdf, meeting.pdf, and poster.pdf: slidesets for degree-required presentations.

These documents have corresponding *.tex files which import:

• metadata from manifest.tex,
• bib references from *.bib,
• external assets (such as images, figures, tables, and generated visualizations of approach relations) from assets.tex,
• chapter information from front.tex, chapters.tex, and back.tex, all respectively for the front, main, and back matters,
• helpful macros from *macros.tex, and
• the McMaster University colour swatch from mcmaster_colours.tex.

Collected Data

• ApproachGlossary.csv: the list of identified test approaches, along with relevant information.
• QualityGlossary.csv: the list of identified software qualities (which may imply related test types), along with relevant information.
• SuppGlossary.csv: the list of terminology that is either shared by multiple approaches or too complicated to explain inline in another glossary.
• scripts: contains, among other things, many helper Python scripts for analyzing collected data; scripts/csvToGraph.py is the most prominent and calls all other relevant scripts.

Build

Before building, you will need to install:

• a working LaTeX installation,
• Pygments, and
• InkScape.

Please run make system_requirements for an up-to-date list of system requirements for building this thesis. Everything related to building the thesis is contained in the Makefile; example commands you may find useful are:

• make <X>: will build the corresponding X.pdf where X is one of thesis, paper, seminar, and defense, among others.
• make graphs: will build all visualizations of test approach relations, including those created manually or used as examples. (Note that this will take a while, mainly due to the size of approachGraph.pdf.)
• make docs: equivalent to running make paper thesis seminar defense.

Configuration

Please view manifest.tex for information about configuring the build.

Debugging

make thesis runs fast because it runs without interactivity. To run the TeX build of this thesis interactively, please run make debug instead of make thesis. This will allow you to debug any visual artifacts found in the PDF, or the lack thereof.

Credits and Acknowledgements

This thesis was based on Jason Balaci's McMaster thesis template which provided many provided many helper LaTeX functions; slides were also heavily based on a previous presentation of his. Christopher William Schankula also helped me with LaTeX and multiple friends discussed software testing with me, providing many of the approaches in Section 7.2. A special "thank you" to Dr. Stuart Reid for providing me with a paper he wrote that I could not find online, for his work with ISO, and for founding the International Software Testing Qualifications Board (ISTQB)! My family has also supported me in more ways than I can count, and I cannot thank them enough. Finally, Dr. Spencer Smith and Dr. Jacques Carette have been great supervisors and valuable sources of guidance and feedback.

ChatGPT was used to help generate supplementary Python code for constructing visualizations and generating LaTeX code, including regex. ChatGPT and GitHub Copilot were both used for assistance with LaTeX formatting. ChatGPT and ProWritingAid were both used for proofreading. ChatGPT also provided pointers to the potential existence of some of the approaches in Section 7.2.

Thesis Template Credits

Jason Balaci built his McMaster thesis template using "publicly available resources, such as Overleaf, LaTeX Wikibooks, as well as many works by individuals (for both inspiration and code snippets), in no particular order:

• Benjamin Furman's McMaster Thesis Example based on Steven Gunn and Sunil Patel's work.
• Geneva Smith's McMaster thesis template.
• Clemens Niederberger for this public StackOverflow response.
• Arash Esbati for this public StackOverflow response.
• Gabriel Devenyi's McMaster Thesis LaTeX, specifically for the bibliography code.
• 'tom' from TeXBlog."