[SERVER-17371] findAndModify can update a document that doesn't match the predicate if there are concurrent updates Created: 25/Feb/15  Updated: 26/Feb/15  Resolved: 26/Feb/15

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: 3.0.0-rc9
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Angelo Prado Assignee: David Storch
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-17132 Index contains extra entries Closed
Operating System: ALL
Steps To Reproduce:

using MongoDB.Bson;
using MongoDB.Driver;
using MongoDB.Driver.Builders;
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
 
namespace ConsoleApplication1
{
    class Program
    {
 
        public static ConcurrentDictionary<ObjectId, DateTime> myDict2 = new ConcurrentDictionary<ObjectId, DateTime>();
 
        static void Test()
        {
            while (true)
            {
 
                MongoDatabase database = DatabaseUtils.GetDatabase(...);
                var collection = database.GetCollection<SampleObject>("Test");
                collection.Insert(new SampleObject());
 
                var query = Query<SampleObject>.Where(r =>
                           enabled == false);
 
                var sortBy = SortBy<SampleObject>.Ascending(r => r._id);
 
                var update = Update<SampleObject>
                    .Set(r => enabled, true);
 
                var result = collection.FindAndModify(query, sortBy, update, true);
                var sampleObject = result.GetModifiedDocumentAs<SampleObject>();
 
                if (sampleObject == null)
                {
                    Console.Write(".");
                    continue;
                }
 
                if (myDict2.ContainsKey(sampleObject._id))
                {
                    Console.Title = "We hit a race condition";
                    Console.WriteLine("BOOM - This should never happen, but it happens");
                }
                else
                {
                    Console.Write("- ");
                    bool resultx = myDict2.TryAdd(sampleObject._id, DateTime.UtcNow);
 
                    if (!resultx)
                        Console.WriteLine("this should never happen either");
                }
            }
        }
 
 
        static void Main(string[] args)
        {
            MongoDatabase database = DatabaseUtils.GetDatabase(...);
            var collection = database.GetCollection<SampleObject>("Test");
            for (int i = 0; i < 1000; i++)
                collection.Insert(new SampleObject());
 
            for (int i = 0; i < 20; i++)
               new Thread(() => Test()).Start();
 
            Console.ReadLine();
        }
    }
}

Participants:

 Description   

After a coupe of days lost troubleshooting a race condition on our end, we determined MongoDB r3.0.0-rc6/rc9 has issues with concurrent findAndModify writes. Specifically we have business logic that acquires a lease (think: lock) on a given object for a few minutes, for processing. The find and modify sets the lease datetime in the present and the query makes sure it's at least three minutes in the past.

We found MongoDB 3.0 acquires the same object and modifies it twice, within a matter of milliseconds, having a TOCTOU (time-of-check time-of-use) condition. We are able to reproduce consistently on a threaded environment with several queries going out at the same time.

We initially suspected the following issue was the culptrit, but it doesn't seem like it was fixed 100%:

SERVER-17132
https://github.com/mongodb/mongo/commit/1c3f32bedac9a9277b7088f83848fe577526b6da

Unfortunately we still see findAndModify returning objects that were modified by a previous findAndModify statement and no longer fullfil the query predicate due to a concurrent update.



 Comments   
Comment by David Storch [ 26/Feb/15 ]

Thanks for your help aprado@salesforce.com. Given that neither you nor I can reproduce this issue on the latest release candidate, I am going to close this issue as Cannot Reproduce. If you see similar symptoms again, feel free to re-open.

Comment by Angelo Prado [ 26/Feb/15 ]

David, I was able to reproduce on rc9 last week, that's why I thought it was not related to SERVER-17132.

Oddly, after deploying rc10 today... the script does no longer hit the race condition. I cannot longer reproduce with the latest build.

Comment by David Storch [ 25/Feb/15 ]

Hi aprado@salesforce.com,

We are having some trouble reproducing this with the provided script. We reproduced immediately on rc6 and rc7, but not rc8, rc9, or rc10. The evidence on our end now suggests that the issue was fixed for rc8 under SERVER-17132 in 1c3f32bedac9.

Could you provide a bit more information in order to help us track this down?

  • Can you confirm that the script attached in "Repro Steps" will reproduce the issue against both 3.0.0-rc6 and 3.0.0-rc9?
  • Are you running the script against an existing collection or does it start off with an empty collection? If the collection exists, can you please provide the output of getIndexes?

Best,
Dave

Comment by David Storch [ 25/Feb/15 ]

Thanks aprado@salesforce.com. We have identified an MMAPv1-only issue that is likely to be the root cause.

Comment by Angelo Prado [ 25/Feb/15 ]

MMAP

  • angelo
Comment by David Storch [ 25/Feb/15 ]

Hi aprado@salesforce.com,

Thanks for reporting this issue. Could you tell me which storage engine your server is configured with? That is, are you seeing this problem against the default MMAP v1 storage engine or against WiredTiger?

Best,
Dave

Generated at Thu Feb 08 03:44:11 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.