-
Type:
Bug
-
Resolution: Cannot Reproduce
-
Priority:
Minor - P4
-
None
-
Affects Version/s: None
-
Component/s: pymongoarrow
-
None
Context
`find_polars_all` (and `find_arrow_all`) crash when `parallelism="processes"` is used inside
an async context (`asyncio.to_thread`). The crash occurs deep in Python's multiprocessing
resource tracker when setting up the process pool, due to an invalid file descriptor
being passed to `fork_exec`.
Error Details
ValueError: bad value(s) in fds_to_keep Stack trace (relevant frames): - `pymongoarrow/api.py:175` — `with multiprocessing.Pool(processes=4) as pool:` - `multiprocessing/util.py:524` — `_posixsubprocess.fork_exec(...)` with `passfds=(-1, 25)`
Steps to Reproduce
1. Call `find_polars_all(..., parallelism="processes")` from within `asyncio.to_thread`
Workaround
Using `promote_options="permissive"` (from related PR #390) resolves the crash.
The reporter plans to provide a minimal repro in ~2 weeks.
Additional Context
- Reported in: https://github.com/mongodb-labs/mongo-arrow/issues/391
- Related to: #390 (parallel arrow with differing schema)
- The `-1` fd appearing in `fds_to_keep` is likely the root cause — a bad fd leaking
into the multiprocessing context when spawning from an async thread.