Updated: 9/4/2024
Testing complex systems
“The goal of testing is to guarantee a system meets all of the needs of its creators over its lifetime.”
(Scroll down)
Website Summary
Who Is The Audience And What Are The Goals Of This Website?
Testing a complex software system is essential to ensuring its reliability, its stability, and its long-term operation.
Given "manual" and "automated" testing this website focuses on automated testing. Manual testing however usually "comes first". Attempts to automate testing that is not well understood will either fail or require a rewrite. If this happens accept the failure as a lesson and forge ahead. (See: Building Test Platforms | Development Phases). If a software system is well understood, tests for it can be started directly and proceed quickly.
This website coins the term "test platform". A test platform is a custom (Python) program designed to automatically operate, stress and verify the system under test in ways no manual testing can achieve. (If it doesn't do this the testing effort is missing the point.)
A test platform is designed to be used by QE, developers, DevOps, Systems Engineers and all the contributors to the system over its lifetime. Management can also use a test platform as a metric and as a guidepost.
This website's goal is to help the engineering staff and its testing practitioners build exemplary test platforms:
- By using the technology, commentary, Case Studies and Webinars found in this website.
- By improving this website based on user input.
- By training and consulting on the practice of testing complex systems.
Note that some engineering staff will be new to the task at hand - allocate and maintain time for them to learn and grow together.
What Is Chaos Testing And What Are Its Goals?
This website asserts that one way to test some complex computer systems is to automatically and continuously subject them to a random combination of functional, load, and stress operations, all the while observing and validating their proper behavior. This is often called "chaos testing".
Stress operations include hardware and software failures, configuration changes, upgrade and downgrade operations, site switchovers and failbacks. Chaos testing becomes more effective if the operation of the system under test is understood and the stress is engineered to target critical operations (e.g. fail a host during a downgrade operation).
Given the stressors above, validating the proper operation of the system should always involve looking for corrupt or missing data. (If not then, when?) This type of testing can also amplify the functional tests if they, or a subset of them are added to the test platform.
This website leaves it to the practitioners to decide if their system qualifies as "complex". It's also left to the practitioners to decide when to stop developing and running tests.
What Exactly Is A Python-based “Test Platform” And Why Build One?
The term “test platform” is a broad concept, similar to “operating system”. In the context of this website, a test platform is a Python program running on a Linux system. The Python / Linux pairing provides specific, easy-to-use functionality not readily available in other language / OS pairings.
A test platform is designed to interface with, drive, load, stress (fail parts of), and verify the complex system being tested. These activities are executed concurrently and in random sequences. It also provides a capable UI to help its test and development users get the most out of working with it.
Given many activities running at the same time a test platform functions like an operating system in many ways. Building a test platform does require attention to some of the concepts found in operating systems. These skills can be learned as needed.
The logic behind developing, running and maintaining a test platform often requires some amount of buy-in by the engineering team. People look at the engineering budget and ask why they need to build and support a complex piece of testing code along with the initial complex system. My experience has been:
- It doesn't require that many test practitioners to build the test platform given the tools and techniques described in this website.
- Test platforms find critical defects over the lifetime of the system under test.
- Test platforms have a certain logic and end up working in similar ways.
- Once up and running, test platforms are leveraged tremendously by their users. Getting the users' code up and running as soon as possible adds to the value the test platforms provide.
What Other Topics Are Covered In The Website?
Refer to the pull-down menus at the top of the website screen:
- The gDS Facility -- This section presents gDS, a "low code", Python-based, data handling facility that makes building test platforms much more straightforward. You can however use the data handling facility of your choice.
- Building Test Platforms -- This section covers building test platforms including how to manage concurrency.
- Case Studies -- Three case studies are an integral part of this website. The case studies come complete with running (Linux only) code housed in a public GitHub repository (browse to or “git clone” the URL below):
https://github.com/talborough/testingComplexSystems
- Free, interactive webinars -- 2 interactive Zoom webinars, aligned with the 3 case studies are available. Please chat or email with me to ask about them and how to schedule them:
In total this website gives engineering staff something to analyze, something to run, something to draw upon as they size up and build the automation needed for their particular testing situation.