Context

We currently install the library in the site-packages of the tox environment, and then re-build the C extensions locally. Additionally, we install the library when we don't need to for some tasks.

Definition of done

Clean up the tox config to be more efficient.

Pitfalls

Ensure all affected tasks still run properly.