[SERVER-13541] Freeze using mongo + mongodump + fsyncLock Created: 10/Apr/14  Updated: 10/Dec/14  Resolved: 29/Apr/14

Status: Closed
Project: Core Server
Component/s: Tools
Affects Version/s: 2.4.10
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Camille Reverdy Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: dump, fsync, mongo, mongodump, replicaset
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-1423 reads often aren't possible while in ... Closed
Operating System: ALL
Steps To Reproduce:

1- Set up a replica set with some data
2- Connect to a secondary
3- fsyncLock the secondary mongod
4- mongodump one database
Gets stuck to Thu Apr 10 10:08:45.324 DATABASE: XXXX to dump/XXXX

5- open another mongo on the same server
6- try to get running ops ; it's frozen

7- try to stop mongodb process – won't stop until if force using kill -9

Participants:

 Description   

While trying to dump data from a secondary server in a replica set, I get issues using fsyncLock command + mongodump; please see Steps to reproduce.

In the Log file, I have strange messages :

1-

Thu Apr 10 09:58:13.817 [conn21] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 55729, after repl: 55729, at end: 55729 }

2-

Thu Apr 10 09:48:59.613 [conn4605] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 540135, after repl: 540135, at end: 540135 }
Thu Apr 10 09:48:59.615 [conn4605] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:33 reslen:3712 547721ms
Thu Apr 10 09:48:59.615 [conn4605] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.0.0.5:45108] 

3-

ubuntu@ip-10-0-0-6:~$ mongo admin --host 10.0.0.6 --port 27017 --eval "printjson(db.fsyncUnlock())"
MongoDB shell version: 2.4.10
connecting to: 10.0.0.6:27017/admin
^CThu Apr 10 09:50:34.352 Assertion: 13111:field not found, expected type 2
0x750821 0x71982b 0x719d6c 0x5f3aa6 0x5dc364 0x7f12186feff0 0x7f12194c5065 0x73fb53 0x73fb69 0x743f55 0x73c71c 0x73e68b 0x73eb44 0x64cf7f 0x67fb17 0x63deba 0x657ca1 0x70ddb7 0x6ee5dc 0x835ff2 
 mongo(_ZN5mongo15printStackTraceERSo+0x21) [0x750821]
 mongo(_ZN5mongo11msgassertedEiPKc+0x9b) [0x71982b]
 mongo() [0x719d6c]
 mongo(_ZNK5mongo11shell_utils18ConnectionRegistry30killOperationsOnAllConnectionsEb+0x1526) [0x5f3aa6]
 mongo(_Z10quitNicelyi+0x24) [0x5dc364]
 /lib/x86_64-linux-gnu/libc.so.6(+0x36ff0) [0x7f12186feff0]
 /lib/x86_64-linux-gnu/libpthread.so.0(recv+0x75) [0x7f12194c5065]
 mongo(_ZN5mongo6Socket5_recvEPci+0x13) [0x73fb53]
 mongo(_ZN5mongo6Socket11unsafe_recvEPci+0x9) [0x73fb69]
 mongo(_ZN5mongo6Socket4recvEPci+0x75) [0x743f55]
 mongo(_ZN5mongo13MessagingPort4recvERNS_7MessageE+0x8c) [0x73c71c]
 mongo(_ZN5mongo13MessagingPort4recvERKNS_7MessageERS1_+0x1b) [0x73e68b]
 mongo(_ZN5mongo13MessagingPort4callERNS_7MessageES2_+0x34) [0x73eb44]
 mongo(_ZN5mongo18DBClientConnection4callERNS_7MessageES2_bPSs+0x4f) [0x64cf7f]
 mongo(_ZN5mongo14DBClientCursor4initEv+0xb7) [0x67fb17]
 mongo(_ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0xea) [0x63deba]
 mongo(_ZN5mongo18DBClientConnection5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0xa1) [0x657ca1]
 mongo(_ZN5mongo9mongoFindEPNS_7V8ScopeERKN2v89ArgumentsE+0x387) [0x70ddb7]
 mongo(_ZN5mongo7V8Scope10v8CallbackERKN2v89ArgumentsE+0xdc) [0x6ee5dc]
 mongo() [0x835ff2]
Thu Apr 10 09:50:34.361 Error: field not found, expected type 2 at src/mongo/shell/query.js:78

Am I doing anything wrong ?



 Comments   
Comment by Asya Kamsky [ 11/Apr/14 ]

I think you are seeing https://jira.mongodb.org/browse/SERVER-1423 - does that seem about right?

Technically you don't need to fsyncLock to use mongodump - fsyncLock is needed if you are doing a file copy and want to make sure that they are flushed and not changing during the file copy. Mongodump does a query, so I'm not sure whether you technically need to use it.

I believe your issue will be resolved regardless when SERVER-1423 is fixed.

Comment by Camille Reverdy [ 10/Apr/14 ]

It seems that my dump is starting before the lock has been applied, and this transition state is not stable

Comment by Camille Reverdy [ 10/Apr/14 ]

Sorry for the formatting error, and it seems that i can't edit to correct

Generated at Thu Feb 08 03:32:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.