[SERVER-16678] MongoDB crash due to assert Created: 29/Dec/14  Updated: 22/Jan/15  Resolved: 22/Jan/15

Status: Closed
Project: Core Server
Component/s: Index Maintenance, Stability
Affects Version/s: 2.6.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Dharshan Rangegowda Assignee: Ramon Fernandez Marina
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File mongod.log    
Operating System: ALL
Participants:

 Description   

I have a 2.6.4 2+1 replica set . Once of my servers crashed with the following assert

2014-12-28T04:09:01.075+0000 [rsBackgroundSync] replset setting syncSourceFeedback to 10.0.2.188:27017
2014-12-28T04:09:01.098+0000 [repl writer worker 1] CMD: drop cove.product_active
2014-12-28T04:09:01.098+0000 [repl writer worker 1] ERROR: About to fassert -  numIndexesTotal(): 0 numSystemIndexesEntries: 1 _entries.size(): 0 indexNamesToDrop: 1 haveIdIndex: 0
2014-12-28T04:09:01.098+0000 [repl writer worker 1] cove Fatal Assertion 17328
2014-12-28T04:09:01.105+0000 [repl writer worker 1] cove 0xf53ad6 0xf01462 0xeea693 0x8c5c55 0x8b3ab6 0x9b242e 0x9ab804 0x9acfec 0x9ade6e 0xcbaf45 0xcfa45f 0xcfa090 0xef827a 0xf8c4cc 0x7f2da12c1f18 0x7f2d9ff86b9d 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x26) [0xf53ad6]
 /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x1a2) [0xf01462]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xeea693]
 /usr/bin/mongod(_ZN5mongo12IndexCatalog14dropAllIndexesEb+0xb95) [0x8c5c55]
 /usr/bin/mongod(_ZN5mongo8Database14dropCollectionERKNS_10StringDataE+0x326) [0x8b3ab6]
 /usr/bin/mongod(_ZN5mongo7CmdDrop3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x38e) [0x9b242e]
 /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x34) [0x9ab804]
 /usr/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xcdc) [0x9acfec]
 /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x29e) [0x9ade6e]
 /usr/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0xe65) [0xcbaf45]
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x2bf) [0xcfa45f]
 /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x50) [0xcfa090]
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x43a) [0xef827a]
 /usr/bin/mongod() [0xf8c4cc]
 /lib64/libpthread.so.0(+0x7f18) [0x7f2da12c1f18]
 /lib64/libc.so.6(clone+0x6d) [0x7f2d9ff86b9d]
2014-12-28T04:09:01.105+0000 [repl writer worker 1] 
 
***aborting after fassert() failure
 
 
2014-12-28T04:09:01.111+0000 [repl writer worker 1] SEVERE: Got signal: 6 (Aborted).
Backtrace:0xf53ad6 0xf538b0 0x7f2d9fed7c60 0x7f2d9fed7be9 0x7f2d9fed8fe8 0xeea72c 0x8c5c55 0x8b3ab6 0x9b242e 0x9ab804 0x9acfec 0x9ade6e 0xcbaf45 0xcfa45f 0xcfa090 0xef827a 0xf8c4cc 0x7f2da12c1f18 0x7f2d9ff86b9d 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x26) [0xf53ad6]
 /usr/bin/mongod() [0xf538b0]
 /lib64/libc.so.6(+0x33c60) [0x7f2d9fed7c60]
 /lib64/libc.so.6(gsignal+0x39) [0x7f2d9fed7be9]
 /lib64/libc.so.6(abort+0x148) [0x7f2d9fed8fe8]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x13c) [0xeea72c]
 /usr/bin/mongod(_ZN5mongo12IndexCatalog14dropAllIndexesEb+0xb95) [0x8c5c55]
 /usr/bin/mongod(_ZN5mongo8Database14dropCollectionERKNS_10StringDataE+0x326) [0x8b3ab6]
 /usr/bin/mongod(_ZN5mongo7CmdDrop3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x38e) [0x9b242e]
 /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x34) [0x9ab804]
 /usr/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xcdc) [0x9acfec]
 /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x29e) [0x9ade6e]
 /usr/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0xe65) [0xcbaf45]
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x2bf) [0xcfa45f]
 /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x50) [0xcfa090]
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x43a) [0xef827a]
 /usr/bin/mongod() [0xf8c4cc]
 /lib64/libpthread.so.0(+0x7f18) [0x7f2da12c1f18]
 /lib64/libc.so.6(clone+0x6d) [0x7f2d9ff86b9d]

The full logs are attached. I also receive the following errors messages on the shell

2014-12-26T21:59:26.482+0000 [initandlisten] WARNING: the collection 'cove.tmp.mr.product_11239_inc' lacks a unique index on _id. This index is needed for replication to function properly
2014-12-26T21:59:26.482+0000 [initandlisten] 	 To fix this, you need to create a unique index on _id. See http://dochub.mongodb.org/core/build-replica-set-indexes
2014-12-26T21:59:26.488+0000 [initandlisten] WARNING: the collection 'cove.tmp.mr.product_11238_inc' lacks a unique index on _id. This index is needed for replication to function properly
2014-12-26T21:59:26.488+0000 [initandlisten] 	 To fix this, you need to create a unique index on _id. See http://dochub.mongodb.org/core/build-replica-set-indexes
2014-12-26T21:59:26.489+0000 [initandlisten] WARNING: the collection 'cove.tmp.mr.product_11237_inc' lacks a unique index on _id. This index is needed for replication to function properly
2014-12-26T21:59:26.489+0000 [initandlisten] 	 To fix this, you need to create a unique index on _id. See http://dochub.mongodb.org/core/build-replica-set-indexes
2014-12-26T21:59:26.489+0000 [initandlisten] WARNING: the collection 'cove.tmp.mr.product_11240_inc' lacks a unique index on _id. This index is needed for replication to function properly
2014-12-26T21:59:26.489+0000 [initandlisten] 	 To fix this, you need to create a unique index on _id. See http://dochub.mongodb.org/core/build-replica-set-indexes
2014-12-26T21:59:26.494+0000 [initandlisten] WARNING: the collection 'cove.tmp.mr.product_11236_inc' lacks a unique index on _id. This index is needed for replication to function properly
2014-12-26T21:59:26.494+0000 [initandlisten] 	 To fix this, you need to create a unique index on _id. See http://dochub.mongodb.org/core/build-replica-set-indexes

Looks like the two of them are related. Can you suggest how I can proceed?



 Comments   
Comment by Ramon Fernandez Marina [ 22/Jan/15 ]

dharshanr@scalegrid.net, I'm closing this issue after a period of radio silence. If this is still an issue for you feel free to re-open it and provide the information requested above.

Regards,
Ramón.

Comment by Ramon Fernandez Marina [ 07/Jan/15 ]

The same assertion appeared in SERVER-14999, where an index was accidentally created in the system.indexes collection using 2.4, and then the version was upgraded to 2.6. What does

db.system.indexes.getIndexSpecs()

say? Note that if an index was created in system.indexes you may need to use the second workaround from SERVER-14999 (the first one posted is no good).

Generated at Thu Feb 08 03:41:53 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.