Artifacts Collection
Collecting logs and other artifacts from a test is a very important task,
especially if the test fails. Most test frameworks allows you to collect
artifacts that are explicitly configured. pytest-mh has this feature
as well but it also takes this a step further and allows you to collect
and even produce artifacts dynamically after a test is finished.
This is especially useful if you do not want to rely on each test to produce
artifacts that require additional commands to be run (for example a database
dump). With pytest-mh, it is possible to implement this on a different level and
therefore each test can focus solely on testing functionality, pytest-mh
will take care of producing and collecting the extra artifacts.
See also
This feature is used to capture AVC denials and coredumps in
Auditd and
Coredumpd. You can check out the source
code to get some examples.
Example source code
def __init__(
self,
host: MultihostHost,
*,
avc_mode: Literal["fail", "warn", "ignore"],
avc_filter: str | None = None,
) -> None:
"""
``avc_mode`` values:
* ``ignore``: all failures are ignored
* ``warn``: test result category is set to "AVC DENIALS" and the test is
marked as such in a test summary, however test outcome and pytest exit
code is kept intact
* ``fail``: test result category is set to "AVC DENIALS" and the test is
marked as such in a test summary, if a test outcome is ``passed`` it
is set to ``failed`` and pytest will return non-zero exit code
:param host: Multihost host.
:type host: MultihostHost
:param avc_mode: Action taken when AVC denial is found in audit logs.
:type avc_mode: Literal["fail", "warn", "ignore"]
:param avc_filter: Regular expression used to filter the AVC denials,
defaults to None
:type avc_filter: str | None, optional
"""
super().__init__(host)
self.avc_mode: Literal["fail", "warn", "ignore"] = avc_mode
self.avc_filter: str | None = avc_filter
self.artifacts: set[str] = {"/var/log/audit/audit.log"}
self._backup: str | None = None
self._auditd_running: bool = False
def get_artifacts_list(self, host: MultihostHost, artifacts_type: MultihostArtifactsType) -> set[str]:
"""
Dump backtrace and other information from generated core files for easy access.
:param host: Host where the artifacts are being collected.
:type host: MultihostHost
:param artifacts_type: Type of artifacts that are being collected.
:type artifacts_type: MultihostArtifactsType
:return: List of artifacts to collect.
:rtype: set[str]
"""
if self._corefiles is None:
self._corefiles = self.list_core_files()
if not self._corefiles:
return set()
# Parse PID and timestamp that we can use to get information for journal
for name in self._corefiles:
try:
pid, timestamp = self.parse_core_file_name(name)
except ValueError:
self.logger.warn(f"Invalid core file name: {name}")
continue
# Dump the information
self.host.conn.run(
rf"""
journalctl --output=verbose \
'COREDUMP_PID={pid}' \
'COREDUMP_TIMESTAMP={timestamp}' \
> '{self.path}/{name}.backtrace'
""",
log_level=ProcessLogLevel.Error,
)
return {self.path}
User-defined artifacts
The pytest-mh configuration file has a field artifacts in the host section
where it is possible to define a list of artifacts that should be automatically
downloaded from a host when a test is finished and before teardown is executed.
This list can also contain a wildcard.
- hostname: client.test
role: client
artifacts:
- /etc/myapp/myapp.conf
- /var/lib/myapp/db/*
- /var/log/myapp/*
Dynamic artifacts
Dynamic artifacts are not defined in the configuration file, but are defined in the code and therefore the list of artifacts does not have to be static but can be dynamically extended.
Dynamic artifacts can be defined in MultihostHost,
MultihostRole, MultihostUtility and
TopologyController by adding items to the artifacts
attribute of the class.
See also
The type of the artifacts attribute is slightly more complex for hosts
and topology controller since the artifacts can be collected on multiple
phases for these objects. Definition of the attribute can be found here:
New artifacts can also be produced when a test is finished, or the list of
artifacts can be set more dynamically based on your own conditions (e.g.
installation failed). To achieve this, it is possible to override
get_artifacts_list() method of each class. This method is used by pytest-mh
to obtain the list of artifacts to collect and it must return a set() of
artifacts.
See also
You can find definition of get_artifacts_list() here:
Warning
The default implementation of get_artifacts_list() simply returns
self.artifacts. It is not mandatory to reference this attribute in any
way in your implementation, but keep in mind that then this attribute will
not have any effect.
def get_artifacts_list(self, host: MultihostHost, artifacts_type: MultihostArtifactsType) -> set[str]:
"""
Return the list of artifacts to collect.
This just returns :attr:`artifacts`, but it is possible to override this
method in order to generate additional artifacts that were not created
by the test, or detect which artifacts were created and update the
artifacts list.
:param host: Host where the artifacts are being collected.
:type host: MultihostHost
:param artifacts_type: Type of artifacts that are being collected.
:type artifacts_type: MultihostArtifactsType
:return: List of artifacts to collect.
:rtype: set[str]
"""
return self.artifacts
The get_artifacts_list() method takes two arguments:
hostwhich is the host where the artifacts will be collected. This does not have much meaning for hosts, roles and utilities but it is used in the topology controller. Each topology consists of one or more hosts and artifacts are collected from each host.artifacts_typeidentifies when artifacts are being collected. See its definition:MultihostArtifactsTypeMultihostArtifactsType: TypeAlias = Literal[ "pytest_setup", "pytest_teardown", "topology_setup", "topology_teardown", "test" ] """ Multihost artifacts type. * ``pytest_setup``: collected after :meth:`MultihostHost.pytest_setup` * ``pytest_teardown``: collected after :meth:`MultihostHost.pytest_teardown` * ``topology_setup``: collected after :meth:`TopologyController.topology_setup` * ``topology_teardown``: collected after :meth:`TopologyController.topology_teardown` * ``test``: collected after each test run """
Diagram
%%{init: {'theme': 'neutral'}}%%
graph TD
s --> host_pytest_setup --> host_pytest_setup_artifacts --> topology
topology --> host_pytest_teardown -->host_pytest_teardown_artifacts --> e
s(["`**Start**`"])
e(["`**End**`"])
host_pytest_setup("`**Setup hosts**
MultihostHost.pytest_setup`")
host_pytest_setup_artifacts("`**Collect hosts artifacts**
type: pytest_setup`")
host_pytest_teardown("`**Teardown hosts**
MultihostHost.pytest_teardown`")
host_pytest_teardown_artifacts("`**Collect hosts artifacts**
type: pytest_teardown`")
subgraph topology ["`**Topology**`"]
topology_setup --> topology_setup_artifacts --> test
test --> topology_teardown --> topology_teardown_artifacts
topology_setup("`**Setup topology**
TopologyController.topology_setup`")
topology_setup_artifacts("`**Collect topology artifacts**
type: topology_setup`")
subgraph test ["`**Test run**`"]
direction TB
setup --> run(("`**Run test**`")) --> test_artifacts --> teardown
setup("`**Setup before test**`")
test_artifacts("`**Collect test artifacts**
type: test`")
teardown("`**Teardown after test**`")
end
topology_teardown("`**Teardown topology**
TopologyController.topology_teardown`")
topology_teardown_artifacts("`**Collect topology artifacts**
type: topology_teardown`")
end
classDef section fill:#fff,stroke-width:2px,stroke:#ccc
class topology,test section;
classDef setup fill:#44d585,stroke-width:2px,stroke:#33d17a,font-size:1px
class ue,hs,ts,rs,us setup;
class uex,ht,tt,rt,ut setup;
classDef artifacts fill:#ffbc00,stroke-width:0
class host_pytest_setup_artifacts,host_pytest_teardown_artifacts,topology_setup_artifacts,topology_teardown_artifacts,test_artifacts artifacts;
classDef test_node fill:#ff9,stroke-width:0
class run test_node;