Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17371

findAndModify can update a document that doesn't match the predicate if there are concurrent updates

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Priority: Icon: Critical - P2 Critical - P2
    • None
    • Affects Version/s: 3.0.0-rc9
    • Component/s: Querying
    • None
    • ALL
    • Hide 
      using MongoDB.Bson;
using MongoDB.Driver;
using MongoDB.Driver.Builders;
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

namespace ConsoleApplication1
{
    class Program
    {

        public static ConcurrentDictionary<ObjectId, DateTime> myDict2 = new ConcurrentDictionary<ObjectId, DateTime>();

        static void Test()
        {
            while (true)
            {

                MongoDatabase database = DatabaseUtils.GetDatabase(...);
                var collection = database.GetCollection<SampleObject>("Test");
                collection.Insert(new SampleObject());

                var query = Query<SampleObject>.Where(r =>
                           enabled == false);

                var sortBy = SortBy<SampleObject>.Ascending(r => r._id);

                var update = Update<SampleObject>
                    .Set(r => enabled, true);

                var result = collection.FindAndModify(query, sortBy, update, true);
                var sampleObject = result.GetModifiedDocumentAs<SampleObject>();

                if (sampleObject == null)
                {
                    Console.Write(".");
                    continue;
                }

                if (myDict2.ContainsKey(sampleObject._id))
                {
                    Console.Title = "We hit a race condition";
                    Console.WriteLine("BOOM - This should never happen, but it happens");
                }
                else
                {
                    Console.Write("- ");
                    bool resultx = myDict2.TryAdd(sampleObject._id, DateTime.UtcNow);

                    if (!resultx)
                        Console.WriteLine("this should never happen either");
                }
            }
        }


        static void Main(string[] args)
        {
            MongoDatabase database = DatabaseUtils.GetDatabase(...);
            var collection = database.GetCollection<SampleObject>("Test");
            for (int i = 0; i < 1000; i++)
                collection.Insert(new SampleObject());

            for (int i = 0; i < 20; i++)
               new Thread(() => Test()).Start();

            Console.ReadLine();
        }
    }
}
      Show
      using MongoDB.Bson; using MongoDB.Driver; using MongoDB.Driver.Builders; using System ; using System .Collections.Concurrent; using System .Collections.Generic; using System .Linq; using System .Text; using System .Threading; using System .Threading.Tasks; namespace ConsoleApplication1 { class Program { public static ConcurrentDictionary<ObjectId, DateTime> myDict2 = new ConcurrentDictionary<ObjectId, DateTime>(); static void Test() { while ( true ) { MongoDatabase database = DatabaseUtils.GetDatabase(...); var collection = database.GetCollection<SampleObject>( "Test" ); collection.Insert( new SampleObject()); var query = Query<SampleObject>.Where(r => enabled == false ); var sortBy = SortBy<SampleObject>.Ascending(r => r._id); var update = Update<SampleObject> .Set(r => enabled, true ); var result = collection.FindAndModify(query, sortBy, update, true ); var sampleObject = result.GetModifiedDocumentAs<SampleObject>(); if (sampleObject == null ) { Console.Write( "." ); continue ; } if (myDict2.ContainsKey(sampleObject._id)) { Console.Title = "We hit a race condition" ; Console.WriteLine( "BOOM - This should never happen, but it happens" ); } else { Console.Write( "- " ); bool resultx = myDict2.TryAdd(sampleObject._id, DateTime.UtcNow); if (!resultx) Console.WriteLine( " this should never happen either" ); } } } static void Main( string [] args) { MongoDatabase database = DatabaseUtils.GetDatabase(...); var collection = database.GetCollection<SampleObject>( "Test" ); for ( int i = 0; i < 1000; i++) collection.Insert( new SampleObject()); for ( int i = 0; i < 20; i++) new Thread(() => Test()).Start(); Console.ReadLine(); } } }

      After a coupe of days lost troubleshooting a race condition on our end, we determined MongoDB r3.0.0-rc6/rc9 has issues with concurrent findAndModify writes. Specifically we have business logic that acquires a lease (think: lock) on a given object for a few minutes, for processing. The find and modify sets the lease datetime in the present and the query makes sure it's at least three minutes in the past.

      We found MongoDB 3.0 acquires the same object and modifies it twice, within a matter of milliseconds, having a TOCTOU (time-of-check time-of-use) condition. We are able to reproduce consistently on a threaded environment with several queries going out at the same time.

      We initially suspected the following issue was the culptrit, but it doesn't seem like it was fixed 100%:

      SERVER-17132
      https://github.com/mongodb/mongo/commit/1c3f32bedac9a9277b7088f83848fe577526b6da

      Unfortunately we still see findAndModify returning objects that were modified by a previous findAndModify statement and no longer fullfil the query predicate due to a concurrent update.

            Assignee:
            david.storch@mongodb.com David Storch
            Reporter:
            aprado@salesforce.com Angelo Prado
            Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated:
              Resolved: