Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-36737

Aggregating on a collection that is rebuilt using $out will sometimes raise an exception

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Works as Designed
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 4.0.0
    • Component/s: Aggregation Framework
    • None
    • ALL
    • Hide

      Run this python code:

       

      from pymongo import MongoClient
c=MongoClient('127.0.0.1:27017')


def bigdict():
	return {str(i):i**2 for i in range(10)}



col1 = c['test']['source']
col2 = c['test']['dest']

col1.insert_many([bigdict() for x in range(100000)])
col1.aggregate([
		{
	        "$out": col2.name
	    }
	    ])
print("written to db")

def write_to_2():
	col1.aggregate([
		{
	        "$out": col2.name
	    }
	    ])
	print("done writing")

from threading import Thread
Thread(target=write_to_2).start()
while True:
	list(col2.aggregate([
		{
			"$sort": {
				"1": 1
			}
		},
		{
			"$project": {
				"3": 1
			}
		},
		{
			"$limit": 100
		}
		], allowDiskUse=True))
	print(".", end='')

       

      Show
      Run this python code:   from pymongo import MongoClient c=MongoClient( '127.0.0.1:27017' ) def bigdict(): return {str(i):i**2 for i in range(10)} col1 = c[ 'test' ][ 'source' ] col2 = c[ 'test' ][ 'dest' ] col1.insert_many([bigdict() for x in range(100000)]) col1.aggregate([ { "$out" : col2.name } ]) print( "written to db" ) def write_to_2(): col1.aggregate([ { "$out" : col2.name } ]) print( "done writing" ) from threading import Thread Thread (target=write_to_2).start() while True: list(col2.aggregate([ { "$sort" : { "1" : 1 } }, { "$project" : { "3" : 1 } }, { "$limit" : 100 } ], allowDiskUse=True)) print( "." , end='')  

      Let us have two collections, COL1 and COL2.

      Let's assume COL1 has many documents.

      If we aggregate COL1 and end it with an $out operation to COL2,

      while in parallel we aggregate (over and over again for the purpose of this demonstration) over COL2, at the moment that COL2 is rebuilt (the moment that the aggregation over COL1 with the $out has ended) - the aggregation over COL2 will fail.

       

      The exception raised in the code I have given below is:

      OperationFailure: Error in $cursor stage :: caused by :: all indexes on collection dropped

       

      I have tested this with a mongo 4.0.0 instance inside a docker on a Windows 10 machine.

            Assignee:
            nick.brewer Nick Brewer
            Reporter:
            Segal Mark [X]
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: