Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-13541

Freeze using mongo + mongodump + fsyncLock

    XMLWordPrintable

    Details

    • Operating System:
      ALL
    • Steps To Reproduce:
      Hide

      1- Set up a replica set with some data
      2- Connect to a secondary
      3- fsyncLock the secondary mongod
      4- mongodump one database
      Gets stuck to Thu Apr 10 10:08:45.324 DATABASE: XXXX to dump/XXXX

      5- open another mongo on the same server
      6- try to get running ops ; it's frozen

      7- try to stop mongodb process – won't stop until if force using kill -9

      Show
      1- Set up a replica set with some data 2- Connect to a secondary 3- fsyncLock the secondary mongod 4- mongodump one database Gets stuck to Thu Apr 10 10:08:45.324 DATABASE: XXXX to dump/XXXX 5- open another mongo on the same server 6- try to get running ops ; it's frozen 7- try to stop mongodb process – won't stop until if force using kill -9

      Description

      While trying to dump data from a secondary server in a replica set, I get issues using fsyncLock command + mongodump; please see Steps to reproduce.

      In the Log file, I have strange messages :

      1-

      Thu Apr 10 09:58:13.817 [conn21] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 55729, after repl: 55729, at end: 55729 }

      2-

      Thu Apr 10 09:48:59.613 [conn4605] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after cursors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, after opcountersRepl: 0, after recordStats: 540135, after repl: 540135, at end: 540135 }
      Thu Apr 10 09:48:59.615 [conn4605] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:33 reslen:3712 547721ms
      Thu Apr 10 09:48:59.615 [conn4605] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.0.0.5:45108] 

      3-

      ubuntu@ip-10-0-0-6:~$ mongo admin --host 10.0.0.6 --port 27017 --eval "printjson(db.fsyncUnlock())"
      MongoDB shell version: 2.4.10
      connecting to: 10.0.0.6:27017/admin
      ^CThu Apr 10 09:50:34.352 Assertion: 13111:field not found, expected type 2
      0x750821 0x71982b 0x719d6c 0x5f3aa6 0x5dc364 0x7f12186feff0 0x7f12194c5065 0x73fb53 0x73fb69 0x743f55 0x73c71c 0x73e68b 0x73eb44 0x64cf7f 0x67fb17 0x63deba 0x657ca1 0x70ddb7 0x6ee5dc 0x835ff2 
       mongo(_ZN5mongo15printStackTraceERSo+0x21) [0x750821]
       mongo(_ZN5mongo11msgassertedEiPKc+0x9b) [0x71982b]
       mongo() [0x719d6c]
       mongo(_ZNK5mongo11shell_utils18ConnectionRegistry30killOperationsOnAllConnectionsEb+0x1526) [0x5f3aa6]
       mongo(_Z10quitNicelyi+0x24) [0x5dc364]
       /lib/x86_64-linux-gnu/libc.so.6(+0x36ff0) [0x7f12186feff0]
       /lib/x86_64-linux-gnu/libpthread.so.0(recv+0x75) [0x7f12194c5065]
       mongo(_ZN5mongo6Socket5_recvEPci+0x13) [0x73fb53]
       mongo(_ZN5mongo6Socket11unsafe_recvEPci+0x9) [0x73fb69]
       mongo(_ZN5mongo6Socket4recvEPci+0x75) [0x743f55]
       mongo(_ZN5mongo13MessagingPort4recvERNS_7MessageE+0x8c) [0x73c71c]
       mongo(_ZN5mongo13MessagingPort4recvERKNS_7MessageERS1_+0x1b) [0x73e68b]
       mongo(_ZN5mongo13MessagingPort4callERNS_7MessageES2_+0x34) [0x73eb44]
       mongo(_ZN5mongo18DBClientConnection4callERNS_7MessageES2_bPSs+0x4f) [0x64cf7f]
       mongo(_ZN5mongo14DBClientCursor4initEv+0xb7) [0x67fb17]
       mongo(_ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0xea) [0x63deba]
       mongo(_ZN5mongo18DBClientConnection5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0xa1) [0x657ca1]
       mongo(_ZN5mongo9mongoFindEPNS_7V8ScopeERKN2v89ArgumentsE+0x387) [0x70ddb7]
       mongo(_ZN5mongo7V8Scope10v8CallbackERKN2v89ArgumentsE+0xdc) [0x6ee5dc]
       mongo() [0x835ff2]
      Thu Apr 10 09:50:34.361 Error: field not found, expected type 2 at src/mongo/shell/query.js:78

      Am I doing anything wrong ?

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              camille@triplefun.com Camille Reverdy
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: