[SERVER-8604] replica set member down, Got signal: 7 (Bus error). Created: 18/Feb/13  Updated: 03/Feb/16  Resolved: 13/Mar/13

Status: Closed
Project: Core Server
Component/s: Performance
Affects Version/s: 2.2.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: michael cao Assignee: Andre de Frere
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux is13084905-0050 2.6.18-194.el5 #1 SMP Tue Mar 16 21:52:39 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux


Operating System: ALL
Participants:

 Description   

Mon Feb 4 14:47:04 [conn26] Setting temporary authorization to: { local:

{ __system: 2 }

}
Mon Feb 4 14:47:04 [conn26] command: { replSetHeartbeat: "rs1", v: 13, pv: 1, checkEmpty: false, from: "192.168.161.134:27010", $auth: { local:

{ __system: 2 }

} }
Mon Feb 4 14:47:04 [conn26] command admin.$cmd command: { replSetHeartbeat: "rs1", v: 13, pv: 1, checkEmpty: false, from: "192.168.161.134:27010", $auth: { local:

{ __system: 2 }

} } ntoreturn:1 keyUpdates:0 reslen:155 0ms
Mon Feb 4 14:47:05 Invalid access at address: 0x2ab52bd98084 from thread: rsSync

Mon Feb 4 14:47:05 Got signal: 7 (Bus error).

Mon Feb 4 14:47:05 [conn3] runQuery called admin.$cmd

{ ismaster: 1 }

Mon Feb 4 14:47:05 [conn3] run command admin.$cmd

{ ismaster: 1 }

Mon Feb 4 14:47:05 [conn3] command admin.$cmd command:

{ ismaster: 1 }

ntoreturn:1 keyUpdates:0 reslen:283 0ms
Mon Feb 4 14:47:05 [rsHealthPoll] Sending command

{ replSetHeartbeat: "rs1", v: 13, pv: 1, checkEmpty: false, from: "192.168.161.134:27012" }

to 192.168.161.134:27010 with $auth: { local:

{ __system: 2 }

}
Mon Feb 4 14:47:05 BackgroundJob starting: ConnectBG
Mon Feb 4 14:47:05 [rsHealthPoll] Sending command

{ replSetHeartbeat: "rs1", v: 13, pv: 1, checkEmpty: false, from: "192.168.161.134:27012" }

to 192.168.161.134:27011 with $auth: { local:

{ __system: 2 }

}
Mon Feb 4 14:47:05 [conn27] runQuery called admin.$cmd { replSetHeartbeat: "rs1", v: 13, pv: 1, checkEmpty: false, from: "192.168.161.134:27011", $auth: { local:

{ __system: 2 }

} }
Mon Feb 4 14:47:05 [conn27] run command admin.$cmd { replSetHeartbeat: "rs1", v: 13, pv: 1, checkEmpty: false, from: "192.168.161.134:27011", $auth: { local:

{ __system: 2 }

} }
Mon Feb 4 14:47:05 [conn27] Setting temporary authorization to: { local:

{ __system: 2 }

}
Mon Feb 4 14:47:05 [conn27] command: { replSetHeartbeat: "rs1", v: 13, pv: 1, checkEmpty: false, from: "192.168.161.134:27011", $auth: { local:

{ __system: 2 }

} }
Mon Feb 4 14:47:05 [conn27] command admin.$cmd command: { replSetHeartbeat: "rs1", v: 13, pv: 1, checkEmpty: false, from: "192.168.161.134:27011", $auth: { local:

{ __system: 2 }

} } ntoreturn:1 keyUpdates:0 reslen:155 0ms
Mon Feb 4 14:47:05 Backtrace:
0xaffd31 0x558bb9 0x559142 0x3e34e0eb10 0x8101db 0x85c948 0x86cd2e 0x86df23 0x67bfaa 0x5944f6 0x6735ba 0x67619b 0x678f23 0x9857c9 0x986b60 0x988e55 0x98a027 0x9a5518 0x9a55ba 0x9a5878
./mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31]
./mongod(_ZN5mongo10abruptQuitEi+0x399) [0x558bb9]
./mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x262) [0x559142]
/lib64/libpthread.so.0 [0x3e34e0eb10]
./mongod(_ZN5mongo16NamespaceDetails5allocEPKciRNS_7DiskLocE+0x1ab) [0x8101db]
./mongod(_ZN5mongo26allocateSpaceForANewRecordEPKcPNS_16NamespaceDetailsEib+0x48) [0x85c948]
./mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibbPb+0x106e) [0x86cd2e]
./mongod(_ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEb+0x43) [0x86df23]
./mongod(_ZN5mongo6Cloner3FunclERNS_27DBClientCursorBatchIteratorE+0x54a) [0x67bfaa]
./mongod(_ZN5mongo18DBClientConnection5queryEN5boost8functionIFvRNS_27DBClientCursorBatchIteratorEEEERKSsNS_5QueryEPKNS_7BSONObjEi+0x2b6) [0x5944f6]
./mongod(_ZN5mongo6Cloner4copyEPKcS2_bbbbbbNS_5QueryE+0x28a) [0x6735ba]
./mongod(_ZN5mongo6Cloner2goEPKcRKNS_12CloneOptionsERSt3setISsSt4lessISsESaISsEERSsPi+0x85b) [0x67619b]
./mongod(_ZN5mongo9cloneFromERKSsRKNS_12CloneOptionsERSsPiPSt3setISsSt4lessISsESaISsEE+0x43) [0x678f23]
./mongod [0x9857c9]
./mongod(_ZN5mongo11ReplSetImpl24_syncDoInitialSync_cloneEPKcRKSt4listISsSaISsEEb+0x1d0) [0x986b60]
./mongod(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0x845) [0x988e55]
./mongod(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x37) [0x98a027]
./mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x68) [0x9a5518]
./mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2a) [0x9a55ba]
./mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0x9a5878]



 Comments   
Comment by Ramon Fernandez Marina [ 03/Feb/16 ]

vionemc, please note that MongoDB 2.2 is no longer supported. If you're having issues with a recent version of MongoDB please open a new ticket.

Thanks,
Ramón.

Comment by Aminah Nuraini [ 19/Jan/16 ]

Currently, there is an unofficial recovery method written in this article https://www.compose.io/articles/shipwrecked-a-mongodb-data-recovery-tale/

This is the script https://github.com/MongoHQ/purplebeard

Can you please make the official one?

Comment by Aminah Nuraini [ 19/Jan/16 ]

I also got this error. It was caused by hardware failure. There is a huge probability it will be hard to recover the data.
Some of the data files can't be backup-ed to other storage.

Comment by Andre de Frere [ 13/Mar/13 ]

Hi,

I'm setting this ticket to resolved as we have not heard back from you in a while. Please do not hesitate to reopen the case if you have anything further to share, or if you would like to discuss this ticket further.

Regards,
André

Comment by Andre de Frere [ 03/Mar/13 ]

Hi,

Just checking in with you on this issue. Have you had a chance to check the filesystem for corruption, or have you found any related errors in dmesg or system logs?

Comment by Andre de Frere [ 20/Feb/13 ]

A "Got signal: 7 (Bus error)." message can appear (along with the "Invalid access at address" message) when there is IO problems or filesystem corruption. Could you please check you dmesg output for errors, especially to do with IO. Have you run an fsck on the volume that the dbpath is on?

Generated at Thu Feb 08 03:17:53 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.