Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-36737

Aggregating on a collection that is rebuilt using $out will sometimes raise an exception

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Works as Designed
    • Affects Version/s: 4.0.0
    • Fix Version/s: None
    • Component/s: Aggregation Framework
    • Labels:
      None
    • Operating System:
      ALL
    • Steps To Reproduce:
      Hide

      Run this python code:

       

      from pymongo import MongoClient
      c=MongoClient('127.0.0.1:27017')
       
       
      def bigdict():
      	return {str(i):i**2 for i in range(10)}
       
       
       
      col1 = c['test']['source']
      col2 = c['test']['dest']
       
      col1.insert_many([bigdict() for x in range(100000)])
      col1.aggregate([
      		{
      	        "$out": col2.name
      	    }
      	    ])
      print("written to db")
       
      def write_to_2():
      	col1.aggregate([
      		{
      	        "$out": col2.name
      	    }
      	    ])
      	print("done writing")
       
      from threading import Thread
      Thread(target=write_to_2).start()
      while True:
      	list(col2.aggregate([
      		{
      			"$sort": {
      				"1": 1
      			}
      		},
      		{
      			"$project": {
      				"3": 1
      			}
      		},
      		{
      			"$limit": 100
      		}
      		], allowDiskUse=True))
      	print(".", end='')
      

       

      Show
      Run this python code:   from pymongo import MongoClient c=MongoClient( '127.0.0.1:27017' )     def bigdict(): return {str(i):i** 2 for i in range( 10 )}       col1 = c[ 'test' ][ 'source' ] col2 = c[ 'test' ][ 'dest' ]   col1.insert_many([bigdict() for x in range( 100000 )]) col1.aggregate([ { "$out" : col2.name } ]) print( "written to db" )   def write_to_2(): col1.aggregate([ { "$out" : col2.name } ]) print( "done writing" )   from threading import Thread Thread(target=write_to_2).start() while True: list(col2.aggregate([ { "$sort" : { "1" : 1 } }, { "$project" : { "3" : 1 } }, { "$limit" : 100 } ], allowDiskUse=True)) print( "." , end= '' )  

      Description

      Let us have two collections, COL1 and COL2.

      Let's assume COL1 has many documents.

      If we aggregate COL1 and end it with an $out operation to COL2,

      while in parallel we aggregate (over and over again for the purpose of this demonstration) over COL2, at the moment that COL2 is rebuilt (the moment that the aggregation over COL1 with the $out has ended) - the aggregation over COL2 will fail.

       

      The exception raised in the code I have given below is:

      OperationFailure: Error in $cursor stage :: caused by :: all indexes on collection dropped

       

      I have tested this with a mongo 4.0.0 instance inside a docker on a Windows 10 machine.

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: