[SERVER-18990] terminate() :SEVERE: Fatal DBException in logOp(): 0 assertion src/mongo/db/storage/extent_manager.cpp:109 Created: 16/Jun/15  Updated: 18/Jun/15  Resolved: 17/Jun/15

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 2.6.8
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: xiaoli wang Assignee: Sam Kleinman (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

Conditionally Randomly Reoccur

Participants:

 Description   

2015-06-17T02:46:06.895+0800 [conn6] uh oh: 404207220
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xffec79]
 mongod(_ZN5mongo10logContextEPKc+0x1f5) [0xfa7ea5]
 mongod(_ZN5mongo12verifyFailedEPKcS1_j+0x136) [0xf90bd6]
 mongod(_ZNK5mongo13ExtentManager12_getOpenFileEi+0xcc) [0xdb211c]
 mongod(_ZNK5mongo13ExtentManager9recordForERKNS_7DiskLocE+0x1d) [0xdb32cd]
 mongod(_ZNK5mongo16NamespaceDetails11inCapExtentERKNS_7DiskLocE+0x26) [0xdd9f76]
 mongod(_ZN5mongo16NamespaceDetails10__capAllocEi+0x16b) [0xdda34b]
 mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPNS_10CollectionERKNS_10StringDataEi+0x1c1) [0xddb151]
 mongod(_ZN5mongo16NamespaceDetails6_allocEPNS_10CollectionERKNS_10StringDataEi+0x1f) [0xde045f]
 mongod(_ZN5mongo16NamespaceDetails5allocEPNS_10CollectionERKNS_10StringDataEi+0x58) [0xde04d8]
 mongod(_ZN5mongo19CappedRecordStoreV111allocRecordEii+0x32) [0xde99d2]
 mongod(_ZN5mongo17RecordStoreV1Base12insertRecordEPKNS_9DocWriterEi+0x59) [0xde9709]
 mongod(_ZN5mongo10Collection14insertDocumentEPKNS_9DocWriterEb+0x6d) [0x8dfeed]
 mongod() [0xd3cfea]
 mongod(_ZN5mongo5logOpEPKcS1_RKNS_7BSONObjEPS2_PbbPS3_+0x5d) [0xd3b0bd]
 mongod(_ZN5mongo6updateERKNS_13UpdateRequestEPNS_7OpDebugEPNS_12UpdateDriverEPNS_14CanonicalQueryE+0x14ae) [0xbb2aae]
 mongod(_ZN5mongo14UpdateExecutor7executeEv+0xb9) [0xbba809]
 mongod() [0x9e1c9f]
 mongod(_ZN5mongo18WriteBatchExecutor10execUpdateERKNS_12BatchItemRefEPNS_7BSONObjEPPNS_16WriteErrorDetailE+0x7d) [0x9e217d]
 mongod(_ZN5mongo18WriteBatchExecutor11bulkExecuteERKNS_21BatchedCommandRequestEPSt6vectorIPNS_19BatchedUpsertDetailESaIS6_EEPS4_IPNS_16WriteErrorDetailESaISB_EE+0x2da) [0x9e25ba]
2015-06-17T02:46:06.906+0800 [conn6] SEVERE: Fatal DBException in logOp(): 0 assertion src/mongo/db/storage/extent_manager.cpp:109
2015-06-17T02:46:06.908+0800 [initandlisten] connection accepted from 172.28.0.3:42108 #25 (21 connections now open)
2015-06-17T02:46:06.909+0800 [initandlisten] connection accepted from 172.28.0.3:42109 #26 (22 connections now open)
2015-06-17T02:46:06.910+0800 [initandlisten] connection accepted from 172.28.0.3:42110 #27 (23 connections now open)
2015-06-17T02:46:06.911+0800 [initandlisten] connection accepted from 172.28.0.3:42111 #28 (24 connections now open)
2015-06-17T02:46:06.912+0800 [initandlisten] connection accepted from 172.28.0.3:42112 #29 (25 connections now open)
2015-06-17T02:46:06.914+0800 [conn6] SEVERE: terminate() called, printing stack (if implemented for platform):
0xffec79 0xffe470 0x2ae83183b9a6 0x2ae83183b9d3 0xd3b1fb 0xbb2aae 0xbba809 0x9e1c9f 0x9e217d 0x9e25ba 0x9e2ca6 0x9e6244 0x9f6600 0x9f7e64 0x9f93df 0xc85dd2 0xb20d48 0xb293a0 0x7dbc33 0xfb8720
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xffec79]
 mongod() [0xffe470]
 /usr/lib64/libstdc++.so.6(+0x639a6) [0x2ae83183b9a6]
 /usr/lib64/libstdc++.so.6(+0x639d3) [0x2ae83183b9d3]
 mongod(_ZN5mongo5logOpEPKcS1_RKNS_7BSONObjEPS2_PbbPS3_+0x19b) [0xd3b1fb]
 mongod(_ZN5mongo6updateERKNS_13UpdateRequestEPNS_7OpDebugEPNS_12UpdateDriverEPNS_14CanonicalQueryE+0x14ae) [0xbb2aae]
 mongod(_ZN5mongo14UpdateExecutor7executeEv+0xb9) [0xbba809]
 mongod() [0x9e1c9f]
 mongod(_ZN5mongo18WriteBatchExecutor10execUpdateERKNS_12BatchItemRefEPNS_7BSONObjEPPNS_16WriteErrorDetailE+0x7d) [0x9e217d]
 mongod(_ZN5mongo18WriteBatchExecutor11bulkExecuteERKNS_21BatchedCommandRequestEPSt6vectorIPNS_19BatchedUpsertDetailESaIS6_EEPS4_IPNS_16WriteErrorDetailESaISB_EE+0x2da) [0x9e25ba]
 mongod(_ZN5mongo18WriteBatchExecutor12executeBatchERKNS_21BatchedCommandRequestEPNS_22BatchedCommandResponseE+0x5f6) [0x9e2ca6]
 mongod(_ZN5mongo8WriteCmd3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x3e4) [0x9e6244]
 mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x30) [0x9f6600]
 mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xb24) [0x9f7e64]
 mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x63f) [0x9f93df]
 mongod(_ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x1122) [0xc85dd2]
 mongod() [0xb20d48]
 mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x3c0) [0xb293a0]
 mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xf3) [0x7dbc33]
 mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x5f0) [0xfb8720]



 Comments   
Comment by xiaoli wang [ 18/Jun/15 ]

Hi Sam,

Thank you so much for the speedy reply.
But I cannot use "resync this member" of the set to recover this first instance.
Because of the other 2 instances is not master or secondary. One is "STARTUP2", another is "RECOVERING". Their oplog is not normal too.

And I cannot validate this first instance oplog collection, because it cannot be startup.

Then, how can I do to recovery my replica set of 3 nodes , and reserve my data in database.

Regards,
esala

Comment by Sam Kleinman (Inactive) [ 16/Jun/15 ]

Sorry, it looks like you've encountered some kind of corruption for in the oplog collection for this instance. The most likely cause of these kinds of errors are failures in storage and networking systems.

In cases like this, the solution is to resync this member of the set to ensure that it has a valid data set. Additionally:

  • Ensure that you're running the latest version of your release series, in this case 2.6.10.
  • For diagnostic purposes, you may run validate docs on the oplog collection.

Sorry again that you've encountered this issue, and I hope this helps you resolve this error. I'm going to go ahead and close this issue, but feel free to address additional questions to one of our support channels.

Regards,
sam

Generated at Thu Feb 08 03:49:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.