Today we are going to talk about two critical aspects of a well-oiled software delivery workflow. Smoke testing and its close counterpart, sanity testing, are necessary to ensure a product is operating as expected after large or minor changes. These changes may be simple UI modifications that have no effect on the overall product or large-scale backend changes that have the potential to affect the entire application.
There are some misconceptions out there that smoke testing and sanity testing are the same. This is inaccurate when you look at the different cases in which each of these types of testing is used. Smoke testing is used to check the entire application. In many cases, a smoke test includes all the major functions of an application. Sanity testing takes a more targeted approach. It is more likely to be used to test a specific function of an application.
Knowing all about smoke and sanity testing will help you and your team implement the right verification at the right time. While similar, there are definitely times when one may be used over the other. We will also learn how they are also used to complement each other for a full end-to-end test of an application.
Smoke Testing Burns the Candle at Both Ends
The main thing that differentiates sanity testing from smoke testing is the level and type of testing needed for the feature that was released. Typically, smoke testing is done so that all critical aspects of the software product are tested and verified as “passed.” This could be through manual methods or through automated testing techniques. Failing just one of these during a smoke test means changes to code may have introduced a defect.
Smoke testing is usually completed by members of the QA staff. There are some cases where a developer may do a form of smoke testing, locally. In either case, smoke testing is meant to accomplish the following:
- Critical path testing of main application functions from start to finish. Often, this includes backend API testing via Postman or another automated testing method in a CI/CD pipeline.
- Provide a checkpoint for methods like gated checkins. This type of build verification testing is meant to protect the codebase from injected errors.
- Used as the first line of defense to identify obvious errors before going to QA.
From start to finish, smoke testing helps guide verifying aspects of the application that allow development to remain focused on feature requests, refactoring, or newly reported bugs. It is a simple addition to any team’s workflow to create a more stable product while increasing velocity.
Implementing smoke testing in addition to sanity testing is a good way to increase code quality for the application. In this type of testing, it is possible to automate testing on the developers’ workstations with scriptable testing products. Using this practice, much time can be saved by preventing simple problems from reaching environments used by other teams for testing and verification.
Automated Smoke Testing. When to Integrate Into a CI/CD Pipeline?
Smoke testing is meant to verify multiple portions of the product. As such, automation is often used to minimize the hours needed by QA staff to manually “check the boxes.” Scripted solutions like those mentioned earlier lend themselves to many places to inject an opportunity for code quality checks.
You will mainly see this used in the Deploy stage of a CI/CD process. Either to a development or known QA environment. Using a testing matrix, each critical portion of the application can be called out to be tested by either a front or backend method. Thorough knowledge of the product is critical to understand how to ensure each function is tested.
There are many ways teams can implement this type of automated smoke testing in their CI/CD processes. In the case of Azure, steps are available to run Visual Studio Test tasks. For those that use Jenkins, TravisCI, and others, adding a ‘run’ step may provide a quick win. In any case, adding smoke testing to a CI/CD pipeline will greatly advance the team’s ability to increase code quality.
Sanity Testing via Simplicity
While still an aspect of build verification like smoke testing, sanity testing is oftentimes more relaxed in its structure. Although it is meant to focus on a narrower testing matrix. This type of testing is often interchangeable with smoke testing. However, it has a more directed approach meant to dive deeper into a specific change.
When using sanity testing, engineers should be identifying any missing or broken aspects of the program related to an area that was changed. Additionally, any items that are even somewhat related to the code change should be tested more thoroughly to ensure they are still functional. This includes minor as well as major changes.
The main aspects of sanity testing include:
- A highly focused test matrix designed to verify front and backend functionality of a specific area of an application after a code change.
- A more simplified approach to verification versus a full suite of scripted or build testing.
- A way to identify issues prior to a full application regression test.
It may be a bit confusing to say that sanity testing is simplified, yet more narrowly defined. The simplification aspect comes from the fact that this verification can be tested by an a la carte approach. By using aspects already defined in existing smoke tests, testers can put the focus on specific portions of the product vs waiting for full regression testing to complete.
Conversely, some methods used for sanity testing can be extended into being a standard part of the application’s smoke testing. This well-placed use and re-use of testing methods helps create a full cycle of opportunities for verifying the application’s workings.
Smoke and Sanity Testing by Example
So far, you should be clear on the aspects of smoke and sanity testing that make them different. As an example of both, let us consider a couple of scenarios. In them, we will look at times where smoke testing vs sanity testing may be used and vice versa.
For both scenarios, we are doing development on a fictional web application for displaying various alert dashboards for a Network Operations Center (N.O.C.). The application is a standard dynamic web application that has the typical database connection requirements.
Scenario 1
A change is completed to modify the company logo across the entire product. Code modifications are completed and committed to source where CI/CD allows them to be built and packaged. These changes include style changes and a few other modifications that are not expected to affect the overall functionality of the product.
Best practice dictates that smoke testing should be completed to look for any glaring issues. These may or may not be related to the intended change thanks to several factors from simple mistakes to issues with complicated code merges. This is where automated sanity testing within the CI/CD pipeline can prove that the environment is not broken, but manual Smoke testing will ensure that the visual elements look as they should. By recording these Smoke tests, we can turn them into regression testing to ensure nothing is misplaced in future releases, where no UI changes are involved.
Scenario 2
Upgrades to the database used in the backend of our fictional application require a change to code related to data connections. Like the first scenario, this change is committed to source and ready to be built, deployed, and tested. Just as with other changes, our first line of defense may include smoke testing. Since we are modifying a specific portion of the product that deals with data input and output, we need to take a targeted approach and aim our Smoke testing towards the right direction. This may include manual testing to ensure the database connections exist, are stable, and not causing irreparable damage to existing data.
Sanity testing should be used to make sure the basic aspects of the application are intact. In many cases, one can run sanity testing before smoke to catch basic issues that would require sending back the version to the development team.
Alternatively, running Smoke testing prior to sanity enables the development team to move forward with a tested version, which had proven to correspond with the requirements (at the development level). Then sanity enters as a gate for the testing phase. This approach goes with the idea of ‘once the crazy is out of the way, sanity provides confidence to continue’.
Smoke and Sanity Testing for the Now, and the Later
Clearly, smoke testing is more widely used by default. Especially in automated software delivery pipelines but also in situations where some manual sanity testing is needed. The combination of the two is often the only approach to ensure the entire application is free of new errors.
Recognizing opportunities where additional sanity testing may be needed is often the main item to consider for software developers and testers. As stated earlier, portions of these sanity tests may evolve into being used as the basis of future smoke and regression testing. This combination of the two methods can be used to keep things operating at current levels as well as be ready to test future changes more efficiently.
