How to use the command 'ctest' (with examples)

CTest is the test driver program for CMake, a widely used system for managing the build process in a cross-platform manner. CTest is specifically designed to run tests on your project, ensuring that software components perform correctly before being integrated into a larger code base. CTest lets you automate and streamline the testing process, making it an invaluable tool for developers who prioritize software reliability and integrity. More information can be found on the official CTest Wiki .

Use case 1: Run all tests defined in the CMake project, executing 4 jobs at a time in parallel

Code:

ctest -j4 --output-on-failure

Motivation:

Running tests is a critical part of the build process in software development. When working on large projects, the test suite can become extensive, and running all tests sequentially can take a considerable amount of time. Therefore, leveraging parallel execution to run multiple tests simultaneously can significantly cut down the testing duration. This use case is ideal for developers who have multiple processor cores at their disposal and wish to optimize their development workflow by speeding up the testing phase.

Explanation:

• ctest: This command invokes the CTest test driver program, responsible for executing tests defined in the project.
• -j4: The -j option specifies the number of jobs to run concurrently. In this example, 4 indicates that four jobs (and hence four tests) will be executed simultaneously. This takes advantage of multi-core processing power to reduce the overall time spent on testing.
• --output-on-failure: This flag ensures that the output of a test is only displayed if the test fails. It is particularly useful for filtering out only the relevant information needed to diagnose test failures, thereby producing cleaner log outputs.

Example output:

Test project /path/to/your/build/directory
    Start 1: test_example1
    Start 2: test_example2
    Start 3: test_example3
    Start 4: test_example4
    .........
100% tests passed, 0 tests failed out of 4

Total Test time (real) =   0.32 sec

Use case 2: List available tests

Code:

ctest -N

Motivation:

Before executing tests on a project, especially in complex projects with numerous test cases, it is often beneficial to know what tests are available. Listing available tests provides an overview of what has been defined in the CMake project, facilitating better test management and execution planning. This is particularly useful for new team members or for maintaining consistent testing practices across development teams.

Explanation:

• ctest: Again, this is the command for executing the test driver program.
• -N: The -N option stands for “list-tests,” and its purpose is to list all tests that have been defined in the CMake project without executing them. This gives developers a comprehensive view of all the test cases they can run.

Example output:

Test project /path/to/your/build/directory
  Test #1: test_example1
  Test #2: test_example2
  Test #3: test_example3
  Test #4: test_example4

Total Tests: 4

Use case 3: Run a single test based on its name, or filter on a regular expression

Code:

ctest --output-on-failure -R '^test_name$'

Motivation:

In some scenarios, developers need to focus on a particular test—especially when they are troubleshooting or examining a specific piece of functionality. Whether for debugging purposes or when working within a test-driven development framework, isolating a single test helps developers concentrate on one aspect of the code without the overhead of running the entire test suite. This is particularly effective when the test suite consists of hundreds of tests, which running all at once can be impractical.

Explanation:

• ctest: The command-line utility for managing and running tests.
• --output-on-failure: This flag is used for the same purpose as described earlier, to only show output in the event of a test failure.
• -R '^test_name$': The -R option specifies a regular expression (in this case, a regex pattern) to filter tests by name. '^test_name$' matches exactly the test named test_name. Regular expressions provide powerful ways to search and select tests dynamically, particularly useful in projects with complex naming conventions.

Example output:

Test project /path/to/your/build/directory
    Start 1: test_name
1/1 Test #1: test_name ....................   Passed    0.04 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.07 sec

Conclusion:

The ctest command is an essential part of the CMake ecosystem, allowing developers to efficiently manage the testing process on their projects. Whether it is running all tests in parallel to save time, listing all available tests for better planning, or focusing on a specific test to resolve issues, each use case demonstrates the flexibility and power of CTest in ensuring your code maintains high quality and reliability. These examples highlight its utility in enhancing productivity and maintaining a seamless testing pipeline in modern software development environments.