Loading...

XML

Word

Printable

JSON

Type: Bug
Resolution: Done
Priority: Major - P3
Fix Version/s: 2.6.4, 2.7.4
Affects Version/s: 2.6.2
Component/s: Index Maintenance, Replication
Labels:
None

Backwards Compatibility:
Fully Compatible
Operating System:
ALL
Backport Completed:

2.6.4

Issue Status as of Jul 18, 2014

ISSUE SUMMARY
If during a background index build in a collection on a secondary node this collection gets dropped on the primary, the secondary may crash with a segmentation fault once it replicates the collection drop.

USER IMPACT
Only background index builds are affected. Secondary failures affect replica set quorum and can potentially lead to unavailability of the replica set.

WORKAROUNDS
Use foreground index builds, or ensure that the secondaries have finished building the index before dropping the specific collection in which the index is being built.

AFFECTED VERSIONS
Production release versions from 2.6.0 to 2.6.3 are affected by this bug.

FIX VERSION
The fix is included in the 2.6.4 production release.

RESOLUTION DETAILS
The patch now ensures that IndexBuildBlock::fail() can never throw an exception.

Original description

We have a repeated server crash after two days, while performing same operation (drop).

This instance was running as a secondary in a replicaset.

2014-07-08T08:54:19.676+0200 [repl index builder 17755] build index on: 233-340-6496.keyword_adgroup properties: { v: 1, key: { campaign: 1 }, name: "campaign_1", ns: "233-340-6496.keyword_adgroup", background: true }
2014-07-08T08:54:19.676+0200 [repl index builder 17755]          building index in background
2014-07-08T08:54:19.678+0200 [repl writer worker 8] CMD: drop 233-340-6496.keyword_adgroup
2014-07-08T08:54:19.678+0200 [repl writer worker 8] halting index build: { adgroup: 1 }
2014-07-08T08:54:19.678+0200 [repl writer worker 8] SEVERE: Invalid access at address: 0x1c00736e3a
2014-07-08T08:54:19.684+0200 [repl writer worker 8] SEVERE: Got signal: 11 (Segmentation fault).
Backtrace:0x11dae01 0x11da1de 0x11da2cf 0x7f7f08bb7340 0x8d5ec7 0xb856b2 0xa29ba9 0xa2a8d0 0xa2296a 0xa24958 0xa26279 0xe48ab5 0xeb27b9 0xeb2d00 0x117089e 0x121f819 0x7f7f08baf182 0x7f7f07eb430d 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11dae01]
 /usr/bin/mongod() [0x11da1de]
 /usr/bin/mongod() [0x11da2cf]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f7f08bb7340]
 /usr/bin/mongod(_ZN5mongo12IndexCatalog23killMatchingIndexBuildsERKNS0_17IndexKillCriteriaE+0x217) [0x8d5ec7]
 /usr/bin/mongod(_ZN5mongo12IndexBuilder23killMatchingIndexBuildsEPNS_10CollectionERKNS_12IndexCatalog17IndexKillCriteriaE+0x12) [0xb856b2]
 /usr/bin/mongod(_ZN5mongo7CmdDrop15stopIndexBuildsEPNS_8DatabaseERKNS_7BSONObjE+0xe9) [0xa29ba9]
 /usr/bin/mongod(_ZN5mongo7CmdDrop3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x1c0) [0xa2a8d0]
 /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x3a) [0xa2296a]
 /usr/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x1618) [0xa24958]
 /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6c9) [0xa26279]
 /usr/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0x8d5) [0xe48ab5]
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x749) [0xeb27b9]
 /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x50) [0xeb2d00]
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x19e) [0x117089e]
 /usr/bin/mongod() [0x121f819]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f7f08baf182]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f7f07eb430d]

Another crash, day before, same operation.

2014-07-07T09:59:36.387+0200 [conn405111] end connection 192.168.0.1:51685 (32 connections now open)
2014-07-07T09:59:36.388+0200 [initandlisten] connection accepted from 192.168.0.1:51968 #405172 (33 connections now open)
2014-07-07T09:59:37.589+0200 [conn405168] end connection 172.17.0.13:37430 (32 connections now open)
2014-07-07T09:59:38.533+0200 [repl index builder 11740] build index on: 558-613-4859.keyword_adgroup properties: { v: 1, key: { campaign: 1 }, name: "campaign_1", ns: "558-613-4859.keyword_adgroup", background: true }
2014-07-07T09:59:38.533+0200 [repl index builder 11740] building index in background
2014-07-07T09:59:38.535+0200 [repl writer worker 8] CMD: drop 558-613-4859.keyword_adgroup
2014-07-07T09:59:38.535+0200 [repl writer worker 8] halting index build: { campaign: 1 }
2014-07-07T09:59:38.535+0200 [repl writer worker 8] halting index build: { adgroup: 1 }
2014-07-07T09:59:38.535+0200 [repl writer worker 8] SEVERE: Invalid access at address: 0x1c00736e3a
2014-07-07T09:59:38.541+0200 [repl writer worker 8] SEVERE: Got signal: 11 (Segmentation fault).
Backtrace:0x11dae01 0x11da1de 0x11da2cf 0x7fbb88e7c340 0x8d5ec7 0xb856b2 0xa29ba9 0xa2a8d0 0xa2296a 0xa24958 0xa26279 0xe48ab5 0xeb27b9 0xeb2d00 0x117089e 0x121f819 0x7fbb88e74182 0x7fbb8817930d 
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11dae01]
/usr/bin/mongod() [0x11da1de]
/usr/bin/mongod() [0x11da2cf]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fbb88e7c340]
/usr/bin/mongod(_ZN5mongo12IndexCatalog23killMatchingIndexBuildsERKNS0_17IndexKillCriteriaE+0x217) [0x8d5ec7]
/usr/bin/mongod(_ZN5mongo12IndexBuilder23killMatchingIndexBuildsEPNS_10CollectionERKNS_12IndexCatalog17IndexKillCriteriaE+0x12) [0xb856b2]
/usr/bin/mongod(_ZN5mongo7CmdDrop15stopIndexBuildsEPNS_8DatabaseERKNS_7BSONObjE+0xe9) [0xa29ba9]
/usr/bin/mongod(_ZN5mongo7CmdDrop3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x1c0) [0xa2a8d0]
/usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x3a) [0xa2296a]
/usr/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x1618) [0xa24958]
/usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6c9) [0xa26279]
/usr/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0x8d5) [0xe48ab5]
/usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x749) [0xeb27b9]
/usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x50) [0xeb2d00]
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x19e) [0x117089e]
/usr/bin/mongod() [0x121f819]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7fbb88e74182]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fbb8817930d]

Mongo 2.6.2 is runnig on top of Ubuntu 14.04 x86_64 (10gen mongo packages)

mongod.conf:

dbpath=/var/lib/mongodb
logpath=/var/log/mongodb/mongod.log
logappend=true
smallfiles = true
oplogSize=20480
replSet=rs0

is related to

SERVER-15871 Segmentation fault in secondary rebuilding index in background

Closed

SERVER-14505 cannot dropAllIndexes when index builds in progress assertion failure

Closed

SERVER-15038 Multiple background index builds may not interrupt cleanly for commands, on secondaries

Closed

related to

SERVER-17927 Dropping collection during active background index build on secondary crashed the server

Closed

Assignee:: Eric Milkie

Reporter:: Bernardo Cabezas Serra

Participants:: Bernardo Cabezas Serra, Edu Herraiz, Eric Milkie, Giorgio Sironi, Githook User, Ramon Fernandez Marina

Votes:: 0 Vote for this issue

Watchers:: 11 Start watching this issue

Created:: Jul 08 2014 07:53:22 AM UTC

Updated:: Apr 08 2015 08:10:17 AM UTC

Resolved:: Jul 14 2014 02:44:24 PM UTC

Details

Description

Original description

Attachments

Issue Links

Activity

People

Dates