-
Type:
Bug
-
Resolution: Done
-
Priority:
Major - P3
-
Affects Version/s: 2.6.4
-
Component/s: Replication
-
Labels:None
-
Fully Compatible
-
ALL
ISSUE SUMMARY
On a replica set, when a secondary is building multiple background indices on a given collection and that collection is dropped, the index builds may not be interrupted properly and the secondary may fail with an assertion error as follows:
2014-08-26T11:18:13.384-0400 [repl writer worker 1] Assertion: 17348:cannot dropAllIndexes when index builds in progress
In some rare cases, this issue may crash secondary nodes.
USER IMPACT
The affected collection and indices may not be successfully dropped on secondary nodes, so if one of these nodes later becomes primary, the collection and indices may still be present.
If a secondary node crashes, it needs to be restarted. If the secondary has become stale, it needs to be resynchronized by removing its data and performing an initial sync.
WORKAROUNDS
The workaround is to avoid the situation of multiple background index builds concurrent with collection drops:
- by making the indexes foreground builds, or
- by synchronizing collection drop with index build completion on secondaries through the use of write concern on the background index builds
AFFECTED VERSIONS
MongoDB 2.6 production releases up to 2.6.4 are affected by this issue.
FIX VERSION
The fix is included in the 2.6.5 production release.
RESOLUTION DETAILS
After an index build yields and regains its lock, check for interrupt before recovering our cursor.
Original description
After the bugfix in 2.6.4 related to this bug https://jira.mongodb.org/browse/SERVER-14494 , we update to 2.6.4, but we still reproducing the error:
2014-08-25T05:02:21.430+0200 [repl index builder 35135] build index on: 542-110-6347.keyword_adgroup properties: { v: 1, key: { adgroup: 1 }, name: "adgroup_1", ns: "542-110-6347.keyword_adgroup", background: true }
2014-08-25T05:02:21.430+0200 [repl index builder 35135] building index in background
2014-08-25T05:02:21.537+0200 [repl writer worker 12] CMD: drop 542-110-6347.keyword_adgroup
2014-08-25T05:02:21.537+0200 [repl writer worker 12] halting index build: { adgroup: 1 }
2014-08-25T05:02:21.537+0200 [repl writer worker 12] halting index build: { campaign: 1 }
2014-08-25T05:02:21.537+0200 [repl writer worker 12] SEVERE: Invalid access at address: 0x1c00736e3a
2014-08-25T05:02:21.542+0200 [repl writer worker 12] SEVERE: Got signal: 11 (Segmentation fault).
Backtrace:0x11e6111 0x11e54ee 0x11e55df 0x7f5367a15340 0x8d9077 0xb898a5 0xa2eaa9 0xa31380 0xa2889a 0xa2a651 0xa2bea6 0xe50dd3 0xeb9819 0xeb9d60 0x117b6ee 0x122aaa9 0x7f5367a0d182 0x7f5366d1230d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11e6111]
/usr/bin/mongod() [0x11e54ee]
/usr/bin/mongod() [0x11e55df]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f5367a15340]
/usr/bin/mongod(_ZN5mongo12IndexCatalog23killMatchingIndexBuildsERKNS0_17IndexKillCriteriaE+0x227) [0x8d9077]
/usr/bin/mongod(_ZN5mongo12IndexBuilder23killMatchingIndexBuildsEPNS_10CollectionERKNS_12IndexCatalog17IndexKillCriteriaE+0x15) [0xb898a5]
/usr/bin/mongod(_ZN5mongo7CmdDrop15stopIndexBuildsEPNS_8DatabaseERKNS_7BSONObjE+0xe9) [0xa2eaa9]
/usr/bin/mongod(_ZN5mongo7CmdDrop3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x1c0) [0xa31380]
/usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x3a) [0xa2889a]
/usr/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x19b1) [0xa2a651]
/usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6c6) [0xa2bea6]
/usr/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0x973) [0xe50dd3]
/usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x749) [0xeb9819]
/usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x50) [0xeb9d60]
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x19e) [0x117b6ee]
/usr/bin/mongod() [0x122aaa9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f5367a0d182]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f5366d1230d]
2014-08-25T08:34:21.426+0200 ***** SERVER RESTARTED *****
2014-08-25T08:34:21.429+0200 [initandlisten] MongoDB starting : pid=1422 port=27017 dbpath=/var/lib/mongodb 64-bit host=lohap2
2014-08-25T08:34:21.429+0200 [initandlisten] db version v2.6.4
- is related to
-
-
- Closed
-
- related to
-
-
- Closed
-