Using CTest Command (with examples)

CTest is a test driver program that comes with CMake, a popular build system generator. CTest provides a convenient way to run tests defined in a CMake project. In this article, we will explore different use cases of the ctest command and provide code examples to illustrate each use case.

Use Case 1: Running All Tests in Parallel

One of the main advantages of using CTest is the ability to execute tests in parallel, which can significantly reduce the overall testing time. The -j option allows you to specify the number of jobs to run in parallel.

Here’s an example of how to run all tests defined in the CMake project using 4 parallel jobs:

ctest -j4 --output-on-failure

Motivation: Running tests in parallel is useful when you have a large number of tests and want to speed up the testing process. By using multiple parallel jobs, the tests can be distributed across available CPU cores, resulting in faster execution.

Explanation:

• -j4: Specifies the number of parallel jobs. In this example, we set it to 4, meaning that 4 tests will be executed simultaneously.
• --output-on-failure: This option displays the standard output and error streams of each test only in case of failure.

Example Output: Running tests in parallel with -j4 and --output-on-failure options will display the output of failing tests while executing them concurrently. Other tests will only show the test name and result.

Running tests...
Test project /path/to/project/build
    Start 1: test1
1/5 Test #1: test1 ..................................   Passed    0.10 sec
    Start 2: test2
2/5 Test #2: test2 ..................................   Passed    0.11 sec
    Start 3: test3
3/5 Test #3: test3 ..................................   Passed    0.10 sec
    Start 4: test4
4/5 Test #4: test4 ..................................   Passed    0.12 sec
    Start 5: test5
5/5 Test #5: test5 ..................................   Passed    0.10 sec

100% tests passed, 0 tests failed out of 5

Total Test time (real) = 1.60 sec

Use Case 2: Listing Available Tests

CTest provides the -N option to list all the available tests defined in the CMake project without running them.

Here’s an example of how to display a list of available tests:

ctest -N

Motivation: Sometimes, it is helpful to have a list of available tests for reference, especially when dealing with a large CMake project with a significant number of tests.

Explanation:

• -N: This option tells CTest to only list the available tests without running them.

Example Output: Running ctest -N will display a list of all available tests, along with their test names and associated executables.

Test project /path/to/project/build
    Start 1: test1
1/5 Test #1: test1 ............................   Passed    0.10 sec
    Start 2: test2
2/5 Test #2: test2 ............................   Passed    0.11 sec
    Start 3: test3
3/5 Test #3: test3 ............................   Passed    0.10 sec
    Start 4: test4
4/5 Test #4: test4 ............................   Passed    0.12 sec
    Start 5: test5
5/5 Test #5: test5 ............................   Passed    0.10 sec

100% tests passed, 0 tests failed out of 5

Total Test time (real) = 1.60 sec

Use Case 3: Running a Single Test

In some cases, you may need to run a single test from your test suite, either by specifying its exact name or using a regular expression to filter the tests.

Here’s an example of how to run a single test based on its name:

ctest --output-on-failure -R '^test_name$'

Motivation: Running a single test can be helpful when you want to focus on a specific test, investigate a failure, or debug a particular scenario.

Explanation:

• --output-on-failure: This option displays the standard output and error streams of the test only in case of failure.
• -R '^test_name$': This regular expression filter allows you to run a test with an exact name that matches the specified regular expression pattern.

Example Output: Running ctest --output-on-failure -R '^test_name$' will execute the test with the exact name test_name. If the test fails, the output will be displayed.

Running tests...
Test project /path/to/project/build
    Start 1: test_name
1/1 Test #1: test_name ..........................   Passed    0.10 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) = 0.10 sec

Conclusion

In this article, we explored different use cases of the ctest command, which is part of CMake’s testing framework. We learned how to run tests in parallel, list available tests, and run a single test based on its name or using a regular expression filter. These examples demonstrate some of the powerful features that CTest provides for managing and executing tests in CMake projects. By taking advantage of these features, developers can ensure the quality of their code and detect issues early in the development process.