Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-36737

Aggregating on a collection that is rebuilt using $out will sometimes raise an exception

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Works as Designed
    • Icon: Major - P3 Major - P3
    • None
    • 4.0.0
    • Aggregation Framework
    • None
    • ALL
    • Hide

      Run this python code:

       

      from pymongo import MongoClient
      c=MongoClient('127.0.0.1:27017')
       
       
      def bigdict():
      	return {str(i):i**2 for i in range(10)}
       
       
       
      col1 = c['test']['source']
      col2 = c['test']['dest']
       
      col1.insert_many([bigdict() for x in range(100000)])
      col1.aggregate([
      		{
      	        "$out": col2.name
      	    }
      	    ])
      print("written to db")
       
      def write_to_2():
      	col1.aggregate([
      		{
      	        "$out": col2.name
      	    }
      	    ])
      	print("done writing")
       
      from threading import Thread
      Thread(target=write_to_2).start()
      while True:
      	list(col2.aggregate([
      		{
      			"$sort": {
      				"1": 1
      			}
      		},
      		{
      			"$project": {
      				"3": 1
      			}
      		},
      		{
      			"$limit": 100
      		}
      		], allowDiskUse=True))
      	print(".", end='')
      

       

      Show
      Run this python code:   from pymongo import MongoClient c=MongoClient( '127.0.0.1:27017' )     def bigdict(): return {str(i):i** 2 for i in range( 10 )}       col1 = c[ 'test' ][ 'source' ] col2 = c[ 'test' ][ 'dest' ]   col1.insert_many([bigdict() for x in range( 100000 )]) col1.aggregate([ { "$out" : col2.name } ]) print( "written to db" )   def write_to_2(): col1.aggregate([ { "$out" : col2.name } ]) print( "done writing" )   from threading import Thread Thread(target=write_to_2).start() while True: list(col2.aggregate([ { "$sort" : { "1" : 1 } }, { "$project" : { "3" : 1 } }, { "$limit" : 100 } ], allowDiskUse=True)) print( "." , end= '' )  

    Description

      Let us have two collections, COL1 and COL2.

      Let's assume COL1 has many documents.

      If we aggregate COL1 and end it with an $out operation to COL2,

      while in parallel we aggregate (over and over again for the purpose of this demonstration) over COL2, at the moment that COL2 is rebuilt (the moment that the aggregation over COL1 with the $out has ended) - the aggregation over COL2 will fail.

       

      The exception raised in the code I have given below is:

      OperationFailure: Error in $cursor stage :: caused by :: all indexes on collection dropped

       

      I have tested this with a mongo 4.0.0 instance inside a docker on a Windows 10 machine.

      Attachments

        Activity

          People

            nick.brewer Nick Brewer
            Segal Mark [X]
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: