[SERVER-4907] sparse indexes break replication of array based operators Created: 07/Feb/12  Updated: 11/Jul/16  Resolved: 16/Feb/12

Status: Closed
Project: Core Server
Component/s: Index Maintenance, Replication, Write Ops
Affects Version/s: None
Fix Version/s: 2.0.7, 2.1.1

Type: Bug Priority: Major - P3
Reporter: Aaron Staple Assignee: Aaron Staple
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
related to SERVER-8225 slow replication and long write locks... Closed
is related to SERVER-4781 replica set initial sync failure when... Closed
is related to SERVER-12071 updates from oplog must use _id or do... Closed
Operating System: ALL
Participants:

 Description   

When an array based operator is applied to a field that is absent, the replicated operation checks that the field matches null. If there is a sparse index on the field, the operation may not be replicated properly.

Test

 
function doTest( sparse ) {
    rt = new ReplTest( "repl14tests" );
    
    m = rt.start( true );
    mc = m.getDB( 'd' )[ 'c' ];
    
    mc.ensureIndex( {a:1}, {sparse:sparse} );
    mc.save( {} );
    
    s = rt.start( false );
    sc = s.getDB( 'd' )[ 'c' ];
    
    assert.soon( function() { return sc.count() > 0; } );
    
    mc.update( {}, {$push:{a:1}} );
    
    assert.soon( function() { return sc.count( {a:1} ) > 0; } );
    
    rt.stop();
}
 
doTest( false );
doTest( true );
 



 Comments   
Comment by auto [ 04/Jul/12 ]

Author:

{u'date': u'2012-07-04T16:39:39-07:00', u'email': u'aaron@10gen.com', u'name': u'Aaron'}

Message: SERVER-4907 Backport query plan hinting when oplog entries are applied.
Branch: v2.0
https://github.com/mongodb/mongo/commit/b72aa7471cc8a6dd95c19a3246acb19a65a548eb

Comment by Aaron Staple [ 14/Feb/12 ]

@Neil - SERVER-3918 has been specced out and is scheduled for 2.1.2.

Comment by Neil Sanchala [ 14/Feb/12 ]

glad you were able to make a patch. that said, i think this bug really does indicate how broken the behavior in https://jira.mongodb.org/browse/SERVER-3918 is. i can add this comment on that bug if it would be more helpful, but don't want to spark another round of flames since it's unclear that would be helpful to anyone involved.

the sparse index behavior is unexpected, and can trip knowledgeable people up quite easily. if it trips up the authors of mongo, it's going to trip up the users of mongo.

Comment by auto [ 13/Feb/12 ]

Author:

{u'login': u'astaple', u'name': u'Aaron', u'email': u'aaron@10gen.com'}

Message: SERVER-4907 hint _id or else $natural when calling updateObjects for oplog entries
Branch: master
https://github.com/mongodb/mongo/commit/e7504b9679ebf7101ff55b0cc2cf9c4967673381

Comment by Aaron Staple [ 08/Feb/12 ]

Thanks for the suggestions. There may be other ways of implementing this as well - if we decide to go the route you've suggested we can add this relationship.

Comment by Neil Sanchala [ 08/Feb/12 ]

sigh, i don't see a way to add that relationship. can you?

Comment by Neil Sanchala [ 08/Feb/12 ]

oh, re: question 2, https://jira.mongodb.org/browse/SERVER-3918 is the correct bug; i see that it's not marked resolved, so nevermind about that question. i'll add a relationship between these two bugs.

Comment by Neil Sanchala [ 08/Feb/12 ]

two followup questions:

1. shouldn the oplog entry say {a: {$exists: false}} instead of

{a: null}

?
2. it seems to me that sparse indexes should never be used for null or exists: false checks. https://jira.mongodb.org/browse/SERVER-4578 seems to be for that issue, but is marked as resolved. was that only fixed in 2.0.2?

thanks!

Comment by Aaron Staple [ 08/Feb/12 ]

The update is applied via the standard update code path (the same as is used for updates received from a client). There is no special handling of the _id index in this case. If the query portion of the update consists of {_id:1,a:null} and there is a sparse index on a, the a index will match no results and the _id index will match one result. The a index may be picked because it is more selective for this query.

Comment by Neil Sanchala [ 08/Feb/12 ]

given that the oplog contains the _id of the updated records, why would the sparse index be used at all?

Generated at Thu Feb 08 03:07:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.