[SERVER-25168] Foreground index build blocks all R/W on ALL database on a sharded cluster with secondaryPreferred read preference Created: 20/Jul/16 Updated: 22/Jul/16 Resolved: 21/Jul/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Index Maintenance, Replication |
| Affects Version/s: | 3.0.12 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Alessio Checcucci | Assignee: | Kelsey Schubert |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
One of the users of our MongoDB 3.0.12 6-shards sharded cluster has issued a foreground index build on its own database (pride_archive_ms), after the operation completed on the primary member of each replica set (shard), it got replicated to the secondaries as expected. I was expecting the database where the index build still is in progress to be blocked for read and writes, but actually read/writes to ALL databases are blocked when using secondary or secondaryPreferred read preference. Basically:
MongoDB docs claim that: Any operation that requires a read or write lock on all databases (e.g. listDatabases) will wait for the foreground index build to complete. But I don't see how it matches the case. In fact, simple find() on distinct databases from the one the index is being built on can't carry on. It seems like:
Could you please comment if this is the expected behaviour? Our system is suffering of availability problems because of this. Thanks a lot |
| Comments |
| Comment by Kelsey Schubert [ 22/Jul/16 ] |
|
Hi alessio.checcucci@gmail.com, You can find additional details about concurrency and locking in MongoDB at https://docs.mongodb.com/manual/faq/concurrency/. Specifically, the behavior you have observed is documented here. In MongoDB 3.4, we will support a new eligibility requirement for server selection based on the staleness of the secondaries, There is currently an open ticket to configure whether a default index build is foreground or background - please feel free to vote for Kind regards, |
| Comment by Alessio Checcucci [ 22/Jul/16 ] |
|
Hi Thomas, This triggers the second part of my question, given that the secondaries are not able to serve any query during the build which spans many hours, shouldn't the Mongo router detect this condition and serve from the primary when secondaryPreferred is specified as read preference (this is what we recommend the customers to use)? If the main aim of the secondary read-lock is prevent serving stale data, the primary would be the perfect candidate to answer any query and would prevent availability issues. Last, but not least, is there any way to alter the default index build method to background? I know the answer is "currently not", but I wanted to double check. Thanks a lot |
| Comment by Kelsey Schubert [ 21/Jul/16 ] |
|
Hi alessio.checcucci@gmail.com, Thank you for reporting this behavior. It is expected that operations such as a foreground index build will block reads on a secondary. Replication holds a lock when applying batches of oplog entries to ensure no readers can consume inconsistent data. We have open tickets, which would improve this behavior, Kind regards, |
| Comment by Alessio Checcucci [ 21/Jul/16 ] |
|
I have reproduced (using MongoDB 3.2.8) the behaviour I reported in a very simple manner:
Test setup:
Tests after the index build on the secondaries was triggered:
Could you please shed some light whether this is the behaviour is should expect? I supposed to:
Thanks in advance for any help you can provide |