replace_one() sometimes fail to find match in python3.5.2

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Works as Designed
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Hi,

      I had some code that worked for python2, but when moving to python3 it worked intermittently. Python 3.5.2. MongoDB:
      > db.version()
      3.4.1

      Python snippet:

              client = pymongo.MongoClient(MONGO_URI)
              db = client[database]
      
              potential_task =  db.tasks.find_one({'processing':False})
              if None == potential_task:
                  print "Found no potentials tasks in database"
                  return None
              update = copy.copy(potential_task)
              update['processing'] = True
              update['jobid'] = os.environ['SLURM_JOBID']
              update['start'] = datetime.datetime.utcnow()
              result =  db.tasks.replace_one(potential_task, update)
              if 1 == result.matched_count and 1 == result.modified_count:
                  found_task = True
                  print('Found task', update)
                  return update
              client.close()
      

      In practice, db.tasks.replace_one() often could not find a match for potential_task even though I just read it from the database. Here's an example, after I added some debug output:

      Starting main loop
      Connecting...
      {'_id': ObjectId('5949aa5f729d186983e69e4c'), 'processing': False, 'payload': {'output_file': '/fh/scratch/delete30/peters_u/krcurtis/cidr_epic_work/chr11/epic_chr11_chunk_262.intermediate.nc', 'input_file': '/fh/scratch/delete30/peters_u/krcurtis/cidr_epic_work/chr11/epic_chr11_chunk_262.nc', 'platform': 'epic_omniexpressexome'}}
      {'_id': ObjectId('5949aa5f729d186983e69e4c'), 'processing': False, 'payload': {'output_file': '/fh/scratch/delete30/peters_u/krcurtis/cidr_epic_work/chr11/epic_chr11_chunk_262.intermediate.nc', 'input_file': '/fh/scratch/delete30/peters_u/krcurtis/cidr_epic_work/chr11/epic_chr11_chunk_262.nc', 'platform': 'epic_omniexpressexome'}}
      replace collision, matched=0 modified=0

      I discovered that I could I just use the '_id' field instead, and that worked.

              match_task = { '_id' : potential_task['_id'] }
              result =  db.tasks.replace_one(match_task, update)
      

      I heard from pycon 2017 that python dictionaries in 3.5 had some kind of randomized implementation, so thought trying to match on just one field, the _id field, might work better. Maybe this all goes away in python 3.6 when dictionaries were changed yet again.

      Hope this helps.

            Assignee:
            Shane Harvey
            Reporter:
            Keith Curtis
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: