According to one of the PyArrow maintainers, if we do not pin the pyarrow version in our conda recipe, it will automatically build packages for all pyarrow versions supported by conda-forge, and we get automatic updates like this PR.

I just verified locally if we change recursive-exclude test * to recursive-include test * in MANIFEST.in then the test files end up in the sdist but not the wheel. If we then remove the pyarrow pin and run the tests in the conda recipe, this should just work.

In the meeting today we said that we would simply drop support for older versions on conda-forge instead of trying to support them if they start to fail.