Changing Test Status

Sometimes, it is required to run some additional checks when a test is finished and maybe even change the test result from success to fail, or change the test result category or how its result is displayed in the verbose output.

This is possible by invoking a pytest built-in hook pytest_report_teststatus(). This hook can be added to MultihostUtility, see: MultihostUtility.pytest_report_teststatus

See also

This feature is used in Auditd and Coredumpd in order to set the result category to a custom value and optionally also fail the test.

If an AVC denial occurred during test, it is moved to AVC DENIALS category. If a coredump occured, it is moved to COREDUMPS category.

Example source code

Auditd utility

Modifying the test result in Auditd

    def pytest_report_teststatus(
        self, report: pytest.CollectReport | pytest.TestReport, config: pytest.Config
    ) -> tuple[str, str, str | tuple[str, dict[str, bool]]] | None:
        """
        Report AVC denial error if found and matches requested filter.

        :param report: Pytest report
        :type report: pytest.CollectReport | pytest.TestReport
        :param config: Pytest config
        :type config: pytest.Config
        :return: Pytest test status
        :rtype: tuple[str, str, str | tuple[str, dict[str, bool]]] | None
        """
        if report.when != "call":
            return None

        if not self._auditd_running or self.avc_mode == "ignore" or report.outcome == "skipped":
            return None

        self.logger.info("Checking for AVC denials")

        result = self.host.conn.run(
            "ausearch --input-logs -m AVC,USER_AVC", raise_on_error=False, log_level=ProcessLogLevel.Silent
        )
        if result.rc:
            return None

        records = result.stdout
        if not records:
            return None

        # Ignore if no message matches the filter
        if self.avc_filter:
            match = re.search(self.avc_filter, records)
            if match is None:
                return None

        original_outcome = report.outcome

        # Fail the test if fail mode is selected
        if report.outcome == "passed" and self.avc_mode == "fail":
            report.outcome = "failed"

        # Count this test into "AVC DENIALS" category in the final summary,
        # mark it with "A"/"AVC DENIAL" in short/verbose listing.
        return ("AVC DENIALS", "A", f"{original_outcome.upper()}/AVC DENIAL")

Coredumpd utility

Modifying the test result in Coredumpd

    def pytest_report_teststatus(
        self, report: pytest.CollectReport | pytest.TestReport, config: pytest.Config
    ) -> tuple[str, str, str | tuple[str, dict[str, bool]]] | None:
        """
        Report core file found error if found and matches requested filter.

        :param report: Pytest report
        :type report: pytest.CollectReport | pytest.TestReport
        :param config: Pytest config
        :type config: pytest.Config
        :return: Pytest test status
        :rtype: tuple[str, str, str | tuple[str, dict[str, bool]]] | None
        """
        if report.when != "call":
            return None

        if self.mode == "ignore" or report.outcome == "skipped":
            return None

        self.logger.info("Checking for core files")

        if self._corefiles is None:
            self._corefiles = self.list_core_files()

        if not self._corefiles:
            return None

        # Ignore if no core files matches the filter
        if self.filter:
            match = re.search(self.filter, "\n".join(self._corefiles))
            if match is None:
                return None

        original_outcome = report.outcome

        # Fail the test if fail mode is selected
        if report.outcome == "passed" and self.mode == "fail":
            report.outcome = "failed"

        # Count this test into "COREDUMPS" category in the final summary,
        # mark it with "C"/"COREDUMP" in short/verbose listing.
        return ("COREDUMPS", "C", f"{original_outcome.upper()}/COREDUMP")