Skipping Tests

Pytest filtering options are used to run only the desired tests, but it may be useful to skip a test based on some external condition: for example if the product was built with a certain feature or not. If the feature is not supported, tests that are using this feature must be skipped.

It is not possible to accomplish this with the built-in pytest.mark.skipif marker since it is evaluated too soon and only takes expressions, it does not provide access to the fixtures. However, pytest-mh provides alternative solutions.

See also

Every project has its own specifics and it is not possible to implement generic feature detection and provide it out of the box. ldap.features property used in these examples is not part of pytest-mh. To see tips on how to implement feature detection for your project, see Features Detection.

Skipping individual tests

It is possible to skip individual tests with pytest.mark.require marker. This marker takes a callable as parameter, if the callable evaluates to True the test is run. If it evaluates to False, the test is skipped.

Parameters of the callable are all fixtures that are available to the test, including all MultihostRole objects required by the test, that is all dynamic fixtures defined by the topology.

Warning

The skip condition is evaluated before test setup is performed to avoid complex setup and teardown for a test that is going to be skipped.

Keep this in mind when accessing the role objects, since the role and its utilities setup method has not yet been called, therefore some properties may not be correctly initialized. It is up to you to make sure that you only access the properties that have been already set in your code.

However, it is perfectly fine to run commands and access properties that do not depend on the setup.

Examples of pytest.mark.require

Pytest code

# Use a simple lambda function
@pytest.mark.topology(KnownTopology.LDAP)
@pytest.mark.require(
    lambda ldap: "password_policy" in ldap.features,
    "Server is not built with password policy support"
)
def test_skip__lambda(client: Client, ldap: LDAP):
    pass


# Use a defined function
def require_password_policy(ldap: LDAP):
    return "password_policy" in ldap.features

@pytest.mark.topology(KnownTopology.LDAP)
@pytest.mark.require(
    require_password_policy,
    "Server is not built with password policy support"
)
def test_skip__function(client: Client, ldap: LDAP):
    pass


# Use a defined function that also returns a reason in a tuple
def require_password_policy(ldap: LDAP):
    result = "password_policy" in ldap.features
    reason = "Server is not built with password policy support"

    return result, reason

@pytest.mark.topology(KnownTopology.LDAP)
@pytest.mark.require.with_args(require_password_policy)
def test_skip__function_and_reason(client: Client, ldap: LDAP):
    pass

Note

Notice the usage of with_args in the third example test_skip__function_and_reason. Pytest marker does not allow single function as an argument and it must be worked around by using with_args.

See pytest documentation for more information: pytest.MarkDecorator.with_args()

Pytest run result

tests/test_passkey.py::test_skip__lambda (ldap) SKIPPED (Server is not built with password policy support)
tests/test_passkey.py::test_skip__function (ldap) SKIPPED (Server is not built with password policy support)
tests/test_passkey.py::test_skip__function_and_reason (ldap) SKIPPED (Server is not built with password policy support)

Skipping topology

Sometimes, it is not possible to run any tests from specific topology even though all hosts and roles required by the topology are available – for example when your program was not built with functionality required to correctly setup the topology. It is possible to achieve this by setting a skip condition by overriding the skip() method of TopologyController.

All dynamic fixtures defined by the topology are passed to the method, but this time they are instances of MultihostHost instead of MultihostRole since the role objects are only created for tests and are not available at this point.

Warning

The skip condition is evaluated before topology setup is performed to avoid complex setup and teardown for tests that are going to be skipped.

Keep this in mind when accessing the host objects, since the hosts and its utilities setup method has not yet been called, therefore some properties may not be correctly initialized. It is up to you to make sure that you only access the properties that have been already set in your code.

However, it is perfectly fine to run commands and access properties that do not depend on the setup.

Examples of TopologyController.skip()

Pytest code

class PasswordPolicyTopology(TopologyController):
    def skip(self, ldap: LDAPHost) -> str | None:
        if "password_policy" not in ldap.features:
            # Return reason to skip the tests
            return "Server is not built with password policy support"

        # Return None to run the tests
        return None

Pytest run result

tests/test_passkey.py::test_skip__lambda (ldap) SKIPPED (Server is not built with password policy support)
tests/test_passkey.py::test_skip__function (ldap) SKIPPED (Server is not built with password policy support)
tests/test_passkey.py::test_skip__function_and_reason (ldap) SKIPPED (Server is not built with password policy support)