How to use the command 'pio test' (with examples)

PlatformIO is a powerful, open-source ecosystem for professional embedded development. It offers an ideational and cross-platform build system and library manager. One of the crucial aspects of maintaining and developing embedded systems is the ability to write, execute, and keep track of tests. The command pio test is a versatile tool used to execute tests within a PlatformIO project environment. This command enables developers to ensure the reliability and correctness of their code, facilitating robust software development processes.

Use case 1: Run all tests in all environments of the current PlatformIO project

Code:

pio test

Motivation: Running all tests across all environments is critical when developers want to ensure that the software performs correctly irrespective of the setup. This covers all possible configurations, catching environment-specific bugs that might otherwise go unnoticed if the tests were limited to a single environment. Given the diverse nature of hardware platforms and configurations, this comprehensive testing approach mitigates risks related to code deployment on different devices.

Explanation: The command pio test without any additional arguments executes all test cases defined across all environments specified in the platformio.ini configuration file of the current project. This operation grants a universal check-up of the project’s readiness and ensures each piece of code is validated consistently.

Example output:

Using "native" environment
===========================
..F.

Run in 0.035 seconds
FAILED (failures=1)

Execution summary:
Root environment test result: "FAILED"

Use case 2: Test only specific environments

Code:

pio test --environment environment1 --environment environment2

Motivation: Testing specific environments can be useful in targeted development phases, where changes are made only in certain sections of the code meant to run on specific configurations. It helps to save time and computational resources when there is no need to test the entire range of environments and is particularly useful during iterative development of new features or patches meant for specific setups.

Explanation: By using the --environment flag followed by the names of the desired environments, developers restrict the execution of tests to those specified contexts. This precise control is beneficial for focusing on selective parts of the project setup, reducing overhead and focusing debugging efforts where they’re most needed.

Example output:

Test environment: environment1
===============================
All tests passed.

Test environment: environment2
===============================
.. All tests passed.

Use case 3: Run only tests whose name matches a specific glob pattern

Code:

pio test --filter "pattern"

Motivation: Filtering tests by name using a glob pattern is useful when developers are interested in assessing the performance or correctness of functionally related tests. This approach is simpler and quicker for a focused examination, for example, checking only network-related functionalities if a recent change was made in the networking module.

Explanation: The --filter option with a specific glob pattern restricts the test run to only those tests that match the given pattern. This pattern matching mechanism allows for customized test execution sequences, vital for efficient and targeted testing, especially in larger projects with a vast array of test cases.

Example output:

Starting tests filtered by pattern "network_*"
=============================================
.. All tests passed.

Use case 4: Ignore tests whose name matches a specific glob pattern

Code:

pio test --ignore "pattern"

Motivation: Ignoring certain tests might be beneficial temporarily during the development process when parts of the code are being refactored or are expected to fail due to known issues. By excluding these tests, developers can focus on other test cases that should be passing and avoid cluttering test results with known issues.

Explanation: The --ignore option allows developers to exclude specific tests from the execution based on a matching glob pattern. This exclusion can streamline test results presentation and focus debugging efforts only on unexpected failures.

Example output:

Ignoring tests matching pattern "legacy_*"
==========================================
.. All tests passed.

Use case 5: Specify a port for firmware uploading

Code:

pio test --upload-port upload_port

Motivation: Specifying a custom upload port is essential in a lab environment with multiple connected devices where the default port might not be suitable or when the automatic detection of the upload port fails. This ensures precise targeting of the testing device, thereby avoiding potential conflicts or accidental reprogramming of unintended hardware.

Explanation: The --upload-port option is used to manually designate the upload interface for the firmware. This specified port directly correlates to the particular device under test, allowing for controlled and deterministic test execution in multifaceted hardware environments.

Example output:

Uploading firmware to /dev/ttyUSB1
Testing...
All tests passed successfully.

Use case 6: Specify a custom configuration file for running the tests

Code:

pio test --project-conf path/to/platformio.ini

Motivation: A custom configuration file might be necessary when testing variations of the same project or when certain test runs need different compilation or linkage settings than those specified in the primary configuration file. This flexibility aids in creating tailored testing scenarios without altering the main project configuration, which is crucial for maintaining consistency in shared projects.

Explanation: The --project-conf flag, followed by the path to the desired configuration file, instructs PlatformIO to use an alternative .ini configuration. This allows developers to leverage a specific setup temporarily without making permanent changes to the primary project file, facilitating specialized testing needs.

Example output:

Using custom project configuration from path/to/platformio.ini
Starting tests...
All tests passed with custom configurations.

Conclusion

The pio test command is a powerful and flexible tool in the PlatformIO suite, tailored to enhance testing practices for embedded software developers. By offering a wide range of testing options, including environment specification, test filtering, port configuration, and more, it supports comprehensive and efficient testing strategies. Understanding these use cases allows developers to adapt their testing processes to meet their unique project requirements and ensure high software quality across diverse hardware platforms.