Getting Started

Pytest-mh is not a plugin to support unit testing. It is a plugin designed to support testing your application as a complete product, this is often referred to as a black-box, application or system testing. The application is installed on the target host and tested by running commands on the host (and on other hosts that are required). These hosts can be virtual machines or containers.

See also

SSH, podman and docker may be used to execute commands on the remote hosts. See Running Commands on Remote Hosts for more information.

As such, it is often useful to write a high level API that will support testing your application and make your test smaller and more readable. This may require a non-trivial initial investment, but it will pay off in the long run. However, implementing such API is not required and it is perfectly possible to run commands on the host directly from each test. This approach makes tests usually larger and more difficult to understand and maintain – but every project is different and it is up to you to choose how are you going to test your application.

See also

Pytest-mh provides several building blocks to help you design your test framework. See Extending pytest-mh for Your Needs. It would be good to open this document and read it side by side.

We will use sudo as the application to test for this getting started guide as sudo is known by every power user so chances are that you are already familiar with it. Additionally, sudo tests allows us to show many pytest-mh features. Note, that these tests were written only as an example and sudo itself is not using pytest-mh for its tests at this moment and as far as we know there are no plans to do so.

See also

The example code can be found in the example folder of the git repository.

Example project: sudo

Sudo is a widely known tool that can elevate current user’s privileges by running command as different user – usually root. It is possible to write a set of rules to define which user can run which command and these rules can be stored either locally in /etc/sudoers or in LDAP database.

Our goals are:

• write basic tests

  • allow the user to run all commands, user must authenticate

  • allow the user to run all commands, without authentication

  • allow the user to run all commands if they are a member of a group, user must authenticate

  • allow the user to run all commands if they are a member of a group, without authentication

• these tests must be written for all possible sources of data

  • file: /etc/sudoers

  • LDAP pulled directly by sudo

  • LDAP pulled by SSSD

• write a simple test framework that will help us to extend the tests easily

• every change must be reverted after each test

As you can see, these goals require us to write 12 tests in total. But since the result is the same and only the data is fetched from different sources, we can use topology parametrization. Topology parametrization allows us to write only one test but run it against different backends and thus we will do less work but get more code coverage.

We will take the following steps to achieve it:

1. Prepare a file structure

2. Define multihost topologies

3. Write configuration file

4. Define MultihostConfig and MultihostDomain

5. Design and implement the framework

6. Enable pytest-mh in conftest.py

7. Write the tests

8. Run the tests

Prepare a file structure

The following snippet shows a recommended file structure for your test utilizing pytest-mh. Look at Extending pytest-mh for Your Needs to get more information about the meaning of individual classes.

.
├── framework/                    # Test framework, high-level API
│   ├── hosts/                    # Subclasses of MultihostHost
│   │   └── __init__.py
│   ├── roles/                    # Subclasses of MultihostRole
│   │   └── __init__.py
│   ├── utils/                    # Subclasses of MultihostUtility
│   │   └── __init__.py
│   ├── __init__.py
│   ├── config.py                 # Definition of MultihostConfig, MultihostDomain
│   ├── topology_controllers.py   # Custom topology controllers
│   └── topology.py               # Definition of multihost topologies
|
├── tests/                        # Tests
|
├── conftest.py                   # Pytest conftest.py
├── pytest.ini                    # Pytest configuration file
├── py.typed                      # Declare that this project uses type hints
|
├── mhc.yaml                      # Pytest-mh configuration file
|
├── readme.md                     # Tests readme
└── requirements.txt              # Tests requirements

Define multihost topologies

This is the first step when designing a test framework since it defines what hosts and roles your project needs. For sudo, we want sudo rules to be fetched from different sources. We can consider each data source to be a single topology.

• sudoers

  • only one host needed

  • users, groups and sudo rules will be created locally

• ldap

  • we need a host where we will run sudo and a host that runs an LDAP server

  • users, groups and sudo rules will be added to the LDAP database

  • sudo will read data from LDAP

• sssd

  • we need a host where we will run sudo and SSSD and a host that runs an LDAP server

  • SSSD will be connected to the LDAP domain

  • users, groups and sudo rules will be added to the LDAP database

  • sudo will read data from SSSD which in turn reads it from LDAP

These are the three topologies that we will define. We will also define a topology group as a shortcut for topology parametrization.

See the code

./framework/topology.py

"""Predefined well-known topologies."""

from __future__ import annotations

from enum import unique
from typing import final

from pytest_mh import KnownTopologyBase, KnownTopologyGroupBase, Topology, TopologyDomain, TopologyMark

from .topology_controllers import LDAPTopologyController, SSSDTopologyController, SudoersTopologyController

__all__ = [
    "KnownTopology",
    "KnownTopologyGroup",
]


@final
@unique
class KnownTopology(KnownTopologyBase):
    """
    Well-known topologies that can be given to ``pytest.mark.topology``
    directly. It is expected to use these values in favor of providing
    custom marker values.

    .. code-block:: python
        :caption: Example usage

        @pytest.mark.topology(KnownTopology.LDAP)
        def test_ldap(client: Client, ldap: LDAP):
            assert True
    """

    Sudoers = TopologyMark(
        name="sudoers",
        topology=Topology(TopologyDomain("sudo", client=1)),
        controller=SudoersTopologyController(),
        fixtures=dict(client="sudo.client[0]", provider="sudo.client[0]"),
    )

    LDAP = TopologyMark(
        name="ldap",
        topology=Topology(TopologyDomain("sudo", client=1, ldap=1)),
        controller=LDAPTopologyController(),
        fixtures=dict(client="sudo.client[0]", ldap="sudo.ldap[0]", provider="sudo.ldap[0]"),
    )

    SSSD = TopologyMark(
        name="sssd",
        topology=Topology(TopologyDomain("sudo", client=1, ldap=1)),
        controller=SSSDTopologyController(),
        fixtures=dict(client="sudo.client[0]", ldap="sudo.ldap[0]", provider="sudo.ldap[0]"),
    )


class KnownTopologyGroup(KnownTopologyGroupBase):
    """
    Groups of well-known topologies that can be given to ``pytest.mark.topology``
    directly. It is expected to use these values in favor of providing
    custom marker values.

    The test is parametrized and runs multiple times, once per each topology.

    .. code-block:: python
        :caption: Example usage (runs on AD, IPA, LDAP and Samba topology)

        @pytest.mark.topology(KnownTopologyGroup.AnyProvider)
        def test_ldap(client: Client, provider: GenericProvider):
            assert True
    """

    AnyProvider = [KnownTopology.Sudoers, KnownTopology.LDAP, KnownTopology.SSSD]

Write configuration file

The topology defines which hosts and roles are needed to run sudo tests. We can convert it into a configuration file that can be used to run all sudo tests.

The configuration file will define one domain with two hosts - one client which will run sudo and SSSD and one ldap which will run the LDAP server.

See also

The full format of the configuration file can be found at Configuration File.

See the code

./mhc.yml

domains:
- id: sudo
  hosts:
  - hostname: master.ldap.test
    conn:
      type: ssh
      host: 172.16.200.3
    role: ldap

  - hostname: client.test
    conn:
      type: ssh
      host: 172.16.200.4
    role: client
    artifacts:
    - /var/log/sssd

Define MultihostConfig and MultihostDomain

These two classes are required to correctly map the configuration file into your Python code. Look for more information at Multihost Configuration and Multihost Domains. It is possible to extend these classes in order to add custom configuration options, use different topology mark and so on. In this example, they only provide the mapping from configuration file to Python classes.

See the code

./framework/config.py

from __future__ import annotations

from typing import Type

from pytest_mh import MultihostConfig, MultihostDomain, MultihostHost, MultihostRole

__all__ = [
    "SUDOMultihostConfig",
    "SUDOMultihostDomain",
]


class SUDOMultihostConfig(MultihostConfig):
    @property
    def id_to_domain_class(self) -> dict[str, Type[MultihostDomain]]:
        """
        All domains are mapped to :class:`SUDOMultihostDomain`.

        :rtype: Class name.
        """
        return {"*": SUDOMultihostDomain}


class SUDOMultihostDomain(MultihostDomain[SUDOMultihostConfig]):
    @property
    def role_to_host_class(self) -> dict[str, Type[MultihostHost]]:
        """
        Map roles to classes:

        * client to ClientHost
        * ldap to LDAPHost

        :rtype: Class name.
        """
        from .hosts.client import ClientHost
        from .hosts.ldap import LDAPHost

        return {
            "client": ClientHost,
            "ldap": LDAPHost,
        }

    @property
    def role_to_role_class(self) -> dict[str, Type[MultihostRole]]:
        """
        Map roles to classes:

        * client to Client
        * ldap to LDAP

        :rtype: Class name.
        """
        from .roles.client import Client
        from .roles.ldap import LDAP

        return {
            "client": Client,
            "ldap": LDAP,
        }

Design and implement the framework

This step is more complicated and can not be treated universally as every project has different needs. It is possible to use multiple building blocks provided by pytest-mh in order to build a high-level API for your tests, see Extending pytest-mh for Your Needs and Life Cycle and Hooks to get a good grasp of all the classes and how to use them.

For the sudo tests, we have implemented several hosts, roles and utility classes and one topology controller for each topology. The following table describes the main idea behind each of these classes.

See the table

Class name/Subclass of	Description
ClientHost MultihostBackupHost	Implements backup and restore methods for the client.
LDAPHost MultihostBackupHost	Implements backup and restore methods for the LDAP server. Opens and maintains connection to the LDAP server using python-ldap library.
SudoersTopologyController BackupTopologyController	Configures environment for the sudoers topology Sets expected content of /etc/nsswitch.conf Creates backup of this setup and automatically restores its state when a test is finished
LDAPTopologyController BackupTopologyController	Configures environment for the LDAP topology Sets expected content of /etc/nsswitch.conf Configures SSSD for identity and authentication Configures /etc/ldap.conf that is read by sudo Creates backup of this setup and automatically restores its state when a test is finished
SSSDTopologyController BackupTopologyController	Configures environment for the SSSD topology Sets expected content of /etc/nsswitch.conf Configures SSSD for identity, authentication and sudo rules Creates backup of this setup and automatically restores its state when a test is finished
Client MultihostRole	Implements GenericProvider which defines interface for managing users, groups and sudoers. The implementation uses local files to store the content.
LDAP MultihostRole	Implements GenericProvider which defines interface for managing users, groups and sudoers. The implementation uses LDAP to store the content.
LocalUsersUtils MultihostUtility	Provides shareable implementation of local users and groups management. Every user and group added during testing is automatically removed.
SUDOUtils MultihostUtility	Implements methods to execute sudo and assert the result

See also

Look at the example code to see how this was implemented.

Enable pytest-mh in conftest.py

When the test framework is written and ready to use, we can tell pytest to start using it in our tests. First, configure pytest to load pytest-mh plugin and then inform pytest-mh which config class it should instantiate.

See the code

./conftest.py

# Configuration file for multihost tests.

from __future__ import annotations

from framework.config import SUDOMultihostConfig

from pytest_mh import MultihostPlugin

# Load additional plugins
pytest_plugins = ("pytest_mh",)


# Setup pytest-mh
def pytest_plugin_registered(plugin) -> None:
    if isinstance(plugin, MultihostPlugin):
        plugin.config_class = SUDOMultihostConfig

Write the tests

The example code shows four tests in total, but 12 tests are executed when pytest is run because each test is run once per topology against different data sources. See Writing Tests to get more information on how to write the tests.

• allow the user to run all commands, user must authenticate

• allow the user to run all commands, without authentication

• allow the user to run all commands if they are a member of a group, user must authenticate

• allow the user to run all commands if they are a member of a group, without authentication

See the code

./tests/test_user.py

from __future__ import annotations

import pytest
from framework.roles.base import GenericProvider
from framework.roles.client import Client
from framework.roles.ldap import LDAP
from framework.topology import KnownTopologyGroup


@pytest.mark.topology(KnownTopologyGroup.AnyProvider)
def test_user__passwd(client: Client, provider: GenericProvider):
    u = provider.user("tuser").add(password="Secret123")
    provider.sudorule("test-rule").add(user=u, command="ALL")

    # LDAP and SSSD topology uses SSSD for id and/or sudo rules
    # Since sudo rules are fetch in periodic task, we must start SSSD after
    # the rule is created in LDAP to avoid race conditions.
    if isinstance(provider, LDAP):
        client.svc.start("sssd.service")

    assert client.sudo.list(u.name, "Secret123", expected=["(root) ALL"])
    assert client.sudo.run(u.name, "Secret123", command="ls /root")


@pytest.mark.topology(KnownTopologyGroup.AnyProvider)
def test_user__nopasswd(client: Client, provider: GenericProvider):
    u = provider.user("tuser").add(password="Secret123")
    provider.sudorule("test-rule").add(user=u, command="ALL", nopasswd=True)

    # LDAP and SSSD topology uses SSSD for id and/or sudo rules
    # Since sudo rules are fetch in periodic task, we must start SSSD after
    # the rule is created in LDAP to avoid race conditions.
    if isinstance(provider, LDAP):
        client.svc.start("sssd.service")

    assert client.sudo.list(u.name, "Secret123", expected=["(root) NOPASSWD: ALL"])
    assert client.sudo.run(u.name, command="ls /root")

./tests/test_group.py

from __future__ import annotations

import pytest
from framework.roles.base import GenericProvider
from framework.roles.client import Client
from framework.roles.ldap import LDAP
from framework.topology import KnownTopologyGroup


@pytest.mark.topology(KnownTopologyGroup.AnyProvider)
def test_group__passwd(client: Client, provider: GenericProvider):
    u = provider.user("tuser").add(password="Secret123")
    g = provider.group("tgroup").add().add_member(u)
    provider.sudorule("test-rule").add(user=g, command="ALL")

    # LDAP and SSSD topology uses SSSD for id and/or sudo rules
    # Since sudo rules are fetch in periodic task, we must start SSSD after
    # the rule is created in LDAP to avoid race conditions.
    if isinstance(provider, LDAP):
        client.svc.start("sssd.service")

    assert client.sudo.list(u.name, "Secret123", expected=["(root) ALL"])
    assert client.sudo.run(u.name, "Secret123", command="ls /root")


@pytest.mark.topology(KnownTopologyGroup.AnyProvider)
def test_group__nopasswd(client: Client, provider: GenericProvider):
    u = provider.user("tuser").add(password="Secret123")
    g = provider.group("tgroup").add().add_member(u)
    provider.sudorule("test-rule").add(user=g, command="ALL", nopasswd=True)

    # LDAP and SSSD topology uses SSSD for id and/or sudo rules
    # Since sudo rules are fetch in periodic task, we must start SSSD after
    # the rule is created in LDAP to avoid race conditions.
    if isinstance(provider, LDAP):
        client.svc.start("sssd.service")

    assert client.sudo.list(u.name, "Secret123", expected=["(root) NOPASSWD: ALL"])
    assert client.sudo.run(u.name, command="ls /root")

Run the tests

The example code provides a set of containers that can be started and used as hosts for testing. See the example readme.md to get the instruction on how to start the containers and install requirements.

When the containers or virtual machines are ready, it is possible to run the tests with the pytest command that you are already familiar with. The only additional thing needed to run pytest-mh tests is to provide the path to the pytest-mh configuration file with --mh-config.

$ pytest --color=yes --mh-config=./mhc.yaml -vvv

Multihost configuration:
  domains:
  - id: sudo
    hosts:
    - hostname: master.ldap.test
      conn:
        type: ssh
        host: 172.16.200.3
      role: ldap
    - hostname: client.test
      conn:
        type: ssh
        host: 172.16.200.4
      role: client
      artifacts:
      - /var/log/sssd

Detected topology:
  - id: sudo
    hosts:
      ldap: 1
      client: 1

Additional settings:
  config file: ./example/mhc.yaml
  log path: None
  lazy ssh: False
  topology filter:
  require exact topology: False
  collect artifacts: on-failure
  artifacts directory: artifacts
  collect logs: on-failure

============================= test session starts ==============================
platform linux -- Python 3.11.9, pytest-8.3.3, pluggy-1.5.0 -- /home/runner/work/pytest-mh/pytest-mh/.venv/bin/python3
cachedir: .pytest_cache
rootdir: /home/runner/work/pytest-mh/pytest-mh/example
configfile: pytest.ini
collecting ...

Selected tests will use the following hosts:
  client: client.test
  ldap: master.ldap.test

collected 12 items

example/tests/test_group.py::test_group__passwd (ldap) PASSED            [  8%]
example/tests/test_group.py::test_group__nopasswd (ldap) PASSED          [ 16%]
example/tests/test_user.py::test_user__passwd (ldap) PASSED              [ 25%]
example/tests/test_user.py::test_user__nopasswd (ldap) PASSED            [ 33%]
example/tests/test_group.py::test_group__passwd (sssd) PASSED            [ 41%]
example/tests/test_group.py::test_group__nopasswd (sssd) PASSED          [ 50%]
example/tests/test_user.py::test_user__passwd (sssd) PASSED              [ 58%]
example/tests/test_user.py::test_user__nopasswd (sssd) PASSED            [ 66%]
example/tests/test_group.py::test_group__passwd (sudoers) PASSED         [ 75%]
example/tests/test_group.py::test_group__nopasswd (sudoers) PASSED       [ 83%]
example/tests/test_user.py::test_user__passwd (sudoers) PASSED           [ 91%]
example/tests/test_user.py::test_user__nopasswd (sudoers) PASSED         [100%]

============================= 12 passed in 24.80s ==============================