[PyMongoArrow] parallelism="processes" crashes in fds_to_keep when called from asyncio.to_thread

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Cannot Reproduce
    • Priority: Minor - P4
    • None
    • Affects Version/s: None
    • Component/s: pymongoarrow
    • None
    • None
    • Python Drivers
    • Hide

      1. What would you like to communicate to the user about this feature?
      2. Would you like the user to see examples of the syntax and/or executable code and its output?
      3. Which versions of the driver/connector does this apply to?

      Show
      1. What would you like to communicate to the user about this feature? 2. Would you like the user to see examples of the syntax and/or executable code and its output? 3. Which versions of the driver/connector does this apply to?
    • None
    • None
    • None
    • None
    • None
    • None

      Context

      `find_polars_all` (and `find_arrow_all`) crash when `parallelism="processes"` is used inside
      an async context (`asyncio.to_thread`). The crash occurs deep in Python's multiprocessing
      resource tracker when setting up the process pool, due to an invalid file descriptor
      being passed to `fork_exec`.

      Error Details

        ValueError: bad value(s) in fds_to_keep
        Stack trace (relevant frames):
        - `pymongoarrow/api.py:175` — `with multiprocessing.Pool(processes=4) as pool:`
        - `multiprocessing/util.py:524` — `_posixsubprocess.fork_exec(...)` with `passfds=(-1, 25)`
      

      Steps to Reproduce

      1. Call `find_polars_all(..., parallelism="processes")` from within `asyncio.to_thread`

      Workaround

      Using `promote_options="permissive"` (from related PR #390) resolves the crash.
      The reporter plans to provide a minimal repro in ~2 weeks.

      Additional Context

      • Reported in: https://github.com/mongodb-labs/mongo-arrow/issues/391
      • Related to: #390 (parallel arrow with differing schema)
      • The `-1` fd appearing in `fds_to_keep` is likely the root cause — a bad fd leaking
        into the multiprocessing context when spawning from an async thread.

            Assignee:
            Iris Ho
            Reporter:
            Jib Adegunloye
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: