Changing dir in test makes pytest-html fail

[adrien-berchet]

I use a fixture like the following:

@pytest.fixture()
def tmp_working_dir(tmp_path, monkeypatch):
    """Change working directory before a test and change it back when the test is finished."""
    monkeypatch.chdir(tmp_path)
    yield tmp_path

It was working well with pytest-html<4 but with pytest-html>=4 I get the following error:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/_pytest/main.py", line 271, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/_pytest/main.py", line 325, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_callers.py", line 130, in _multicall
INTERNALERROR>     teardown[0].send(outcome)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_result.py", line 114, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_callers.py", line 77, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/_pytest/main.py", line 350, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_callers.py", line 152, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_result.py", line 114, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_callers.py", line 77, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/_pytest/runner.py", line 114, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/_pytest/runner.py", line 133, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/_pytest/runner.py", line 226, in call_and_report
INTERNALERROR>     hook.pytest_runtest_logreport(report=report)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_callers.py", line 113, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pluggy/_callers.py", line 77, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pytest_html/basereport.py", line 251, in pytest_runtest_logreport
INTERNALERROR>     self._generate_report()
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pytest_html/selfcontained_report.py", line 39, in _generate_report
INTERNALERROR>     super()._generate_report(self_contained=True)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pytest_html/basereport.py", line 65, in _generate_report
INTERNALERROR>     self._write_report(rendered_report)
INTERNALERROR>   File "/tmp/my-repo/.tox/py38/lib/python3.8/site-packages/pytest_html/basereport.py", line 134, in _write_report
INTERNALERROR>     with self._report_path.open("w", encoding="utf-8") as f:
INTERNALERROR>   File "/usr/local/lib/python3.8/pathlib.py", line 1222, in open
INTERNALERROR>     return io.open(self, mode, buffering, encoding, errors, newline,
INTERNALERROR>   File "/usr/local/lib/python3.8/pathlib.py", line 1078, in _opener
INTERNALERROR>     return self._accessor.open(self, flags, mode)
INTERNALERROR> FileNotFoundError: [Errno 2] No such file or directory: 'reports/pytest-py38.html'

I use the following command to call pytest in a tox job:

    pytest \
        --basetemp={envtmpdir} \
        --cov={[base]name} \
        --cov-branch \
        --cov-fail-under=64 \
        --no-cov-on-fail \
        --cov-report term-missing \
        --cov-report html:reports/coverage-{envname} \
        --cov-report xml:reports/coverage-{envname}.xml \
        --html reports/pytest-{envname}.html \
        --junit-xml=reports/pytest-{envname}.xml \
        --self-contained-html \
        {posargs}

[BeyondEvil]

Oh, that's interesting!

Can you provide a complete minimal example?

[adrien-berchet]

Hi @BeyondEvil
Here is a simple project that shows the issue: pytest-html-issue-732.zip
If you run the command tox run -e py310, you will see the failure.

Please note the following behaviors:

• if you set pytest-html<4 in setup.py the tests pass.
• if you change the order of the tests in tests/test_example.py, pytest-html will fail after the first test using the monkeypatch.chdir passes, so it seems that pytest-html is executed after the test passes but before the fixture teardowns are executed, while I think it should be executed after all the fixture teardowns.

[BeyondEvil]

Hi @BeyondEvil Here is a simple project that shows the issue: pytest-html-issue-732.zip If you run the command tox run -e py310, you will see the failure.

Please note the following behaviors:

• if you set pytest-html<4 in setup.py the tests pass.
• if you change the order of the tests in tests/test_example.py, pytest-html will fail after the first test using the monkeypatch.chdir passes, so it seems that pytest-html is executed after the test passes but before the fixture teardowns are executed, while I think it should be executed after all the fixture teardowns.

Thanks for the example!

As for your second bullet point. A pytest test (item) has three states: setup, call, teardown. Since we create an entry in the result table depending on where a tests fails - we can't wait until teardown has happened.

Regardless, I'm 99% sure that if we fix the root issue here (which likely has to do with relative vs. absolute paths) that won't be an issue.

[BeyondEvil]

Getting an error when running your example:

☸ deployment (ci) in ~/Downloads/pytest-html-issue-732 via 🐍 v3.9.10 (.venv) 
❯ tox run -e py39       
.pkg: _optional_hooks> python /Users/jimbrannlund/Downloads/pytest-html-issue-732/.venv/lib/python3.9/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_sdist> python /Users/jimbrannlund/Downloads/pytest-html-issue-732/.venv/lib/python3.9/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_wheel> python /Users/jimbrannlund/Downloads/pytest-html-issue-732/.venv/lib/python3.9/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: prepare_metadata_for_build_wheel> python /Users/jimbrannlund/Downloads/pytest-html-issue-732/.venv/lib/python3.9/site-packages/pyproject_api/_backend.py True setuptools.build_meta
py39: packaging backend failed (code=1), with LookupError: setuptools-scm was unable to detect version for /Users/jimbrannlund/Downloads/pytest-html-issue-732.

[BeyondEvil]

Also, I'd prefer to be able to repro this without tox, which I can't.

[adrien-berchet]

As for your second bullet point. A pytest test (item) has three states: setup, call, teardown. Since we create an entry in the result table depending on where a tests fails - we can't wait until teardown has happened.

Regardless, I'm 99% sure that if we fix the root issue here (which likely has to do with relative vs. absolute paths) that won't be an issue.

Indeed it should work with absolute paths and it's simpler.

Getting an error when running your example:

☸ deployment (ci) in ~/Downloads/pytest-html-issue-732 via 🐍 v3.9.10 (.venv) 
❯ tox run -e py39       
.pkg: _optional_hooks> python /Users/jimbrannlund/Downloads/pytest-html-issue-732/.venv/lib/python3.9/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_sdist> python /Users/jimbrannlund/Downloads/pytest-html-issue-732/.venv/lib/python3.9/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_wheel> python /Users/jimbrannlund/Downloads/pytest-html-issue-732/.venv/lib/python3.9/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: prepare_metadata_for_build_wheel> python /Users/jimbrannlund/Downloads/pytest-html-issue-732/.venv/lib/python3.9/site-packages/pyproject_api/_backend.py True setuptools.build_meta
py39: packaging backend failed (code=1), with LookupError: setuptools-scm was unable to detect version for /Users/jimbrannlund/Downloads/pytest-html-issue-732.

Ah yes I guess you have to initialize a git repo and commit everything to make setuptools-scm work, I forgot about this...

Also, I'd prefer to be able to repro this without tox, which I can't.

To run it without tox you can just create a virtualenv and do:

pip install ".[test]"
pytest --html reports/pytest-py39.html

[adrien-berchet]

Here is a new version to fix the version stuff, so it should be simpler to install it: pytest-html-issue-732.zip