Uploaded image for project: 'Motor'
  1. Motor
  2. MOTOR-1087

motor_asyncio freezes when using multiprocessing

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 3.2, 3.1.2
    • Affects Version/s: None
    • Component/s: None
    • None

      I can use multiprocessing on mac Os successfully, but after publishing the code on the Linux production server the script blocks and waits forever. 

      the scripts work if I don't make any connection to `mongod` from the main task

       

      • sys: 3.11.1 (main, Dec 23 2022, 19:15:06) [GCC 9.4.0]
      • motor.version 3.1.1
      • pymongo.version, has_c 4.3.3, True
      • os and version: Ubuntu 20.04.4 LTS

      No exceptions are raised

       
      import asyncio
      import multiprocessing
      import motor.motor_asyncio
      
      class Blah:
          @staticmethod
          async def publish():
              connection = motor.motor_asyncio.AsyncIOMotorClient(host='localhost')['SNP']
              res = await connection.publish_logs.find_one()
              return res.get('_id')
      
      
      def worker_main():
          return asyncio.run(do_work())
      
      async def do_work():
          blah = await Blah.publish()
          return blah
      
      
      def run_tasks(tasks=None):
          workers = len(tasks)
          results = []
          with multiprocessing.Pool(processes=workers) as pool:
              for task in tasks:
                  async_result = pool.apply_async(worker_main)
                  results.append(async_result)
      
              pool.close()
              pool.join()
      
              # Print task results
              for result in results:
                  print(result.get())
      
      
      async def build_tasks():
          # if I comment these three lines it "works" on the multiprocess tasks
          connection = motor.motor_asyncio.AsyncIOMotorClient(host='localhost')['SNP']
          res = await connection.publish_logs.find_one()
          print(res.get('_id'))
      
          chunks = []
          for c in range(8):
              blah = []
              chunks.append(blah)
      
          return chunks
      
      
      if __name__ == '__main__':
      
          t = asyncio.run(build_tasks())
          run_tasks(t)
      

            Assignee:
            julius.park@mongodb.com Julius Park (Inactive)
            Reporter:
            testipazzi4@gmail.com Francesco Musso
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: