Recently at Fenerum, we moved our test suite into Docker to improve consistency across environments. While this was the right architectural decision, it came with a performance hit - our test runtime increased from 50 minutes to 1 hour and 20 minutes.
The culprit? Python's coverage.py library. While excellent for its purpose, it adds significant overhead when running inside Docker. After some research, we found Slipcover, an alternative coverage library focused on performance.
Implementing Slipcover was straightforward, and the results were immediate - we got our runtime back down to 50 minutes. The performance gains come from Slipcover's more efficient approach to tracking code execution, particularly in containerized environments.
There's one caveat though: Slipcover doesn't support pytest-xdist, which means we can't parallelize our tests across multiple processes. We're currently trying to solve this - hopefully we will be able to contribute this to slipcover.
If you're running tests in Docker and struggling with coverage performance, Slipcover might be worth investigating. Just be prepared to make the parallelization tradeoff.