Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-9947

Shell is stuck when primary node stopped (Ctrl-z)

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Won't Do
    • Priority: Icon: Minor - P4 Minor - P4
    • None
    • Affects Version/s: 2.4.4
    • Component/s: Shell
    • Environment:
      Ubuntu 12.04 and Dell XPS13 laptop
    • Linux
    • Hide

      1) Setup a config server, a mongos and 3 replicaset members.

      2) Run the primary node in a shell and in foreground (without --fork).

      3) Run attached rstest.sh to create an insert load via mongos to the primary.

      4) Stop the primary node with Ctrl-Z.

      5) Wait forever

      6) Notice that rstest.sh is stuck and not inserting anything anymore, even if a new primary is available and known to mongos.

      7) press Ctrl-C in the rstest.sh shell. (twice)

      hingo@hingo-sputnik:~/hacking/rstests$ ./rstest.sh
      Mon Jun 17 10:43:18 EEST 2013
      MongoDB shell version: 2.4.4
      connecting to: hingo-sputnik:27000/rstest
      bye
      Mon Jun 17 10:43:19 EEST 2013
      *****************************************************
      Mon Jun 17 10:43:20 EEST 2013
      MongoDB shell version: 2.4.4
      connecting to: hingo-sputnik:27000/rstest

      --> Notice many minutes pass by:

      ^CMon Jun 17 10:46:03.942 Assertion: 13111:field not found, expected type 2
      0x7495d1 0x71189b 0x711ddc 0x5f3c76 0x5dc554 0x7f514e48d4a0 0x7f514f23f14a 0x738aa3 0x738ab9 0x73cd05 0x73566c 0x7375db 0x737a94 0x64ce0f 0x67ab37 0x63dd4a 0x657b11 0x705efc 0x6e8a70 0x82ef62
      mongo(_ZN5mongo15printStackTraceERSo+0x21) [0x7495d1]
      mongo(_ZN5mongo11msgassertedEiPKc+0x9b) [0x71189b]
      mongo() [0x711ddc]
      mongo(_ZNK5mongo11shell_utils18ConnectionRegistry30killOperationsOnAllConnectionsEb+0x1526) [0x5f3c76]
      mongo(_Z10quitNicelyi+0x24) [0x5dc554]
      /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f514e48d4a0]
      /lib/x86_64-linux-gnu/libpthread.so.0(recv+0x6a) [0x7f514f23f14a]
      mongo(_ZN5mongo6Socket5_recvEPci+0x13) [0x738aa3]
      mongo(_ZN5mongo6Socket11unsafe_recvEPci+0x9) [0x738ab9]
      mongo(_ZN5mongo6Socket4recvEPci+0x75) [0x73cd05]
      mongo(_ZN5mongo13MessagingPort4recvERNS_7MessageE+0x8c) [0x73566c]
      mongo(ZN5mongo13MessagingPort4recvERKNS_7MessageERS1+0x1b) [0x7375db]
      mongo(ZN5mongo13MessagingPort4callERNS_7MessageES2+0x34) [0x737a94]
      mongo(_ZN5mongo18DBClientConnection4callERNS_7MessageES2_bPSs+0x4f) [0x64ce0f]
      mongo(_ZN5mongo14DBClientCursor4initEv+0xb7) [0x67ab37]
      mongo(_ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0xea) [0x63dd4a]
      mongo(_ZN5mongo18DBClientConnection5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0xa1) [0x657b11]
      mongo(_ZN5mongo9mongoFindEPNS_7V8ScopeERKN2v89ArgumentsE+0x42c) [0x705efc]
      mongo(_ZN5mongo7V8Scope10v8CallbackERKN2v89ArgumentsE+0xb0) [0x6e8a70]
      mongo() [0x82ef62]
      Mon Jun 17 10:46:03.958 JavaScript execution failed: Error: field not found, expected type 2 at src/mongo/shell/query.js:L78

      ^CMon Jun 17 10:46:14.853 Assertion: 13111:field not found, expected type 2
      0x7495d1 0x71189b 0x711ddc 0x5f3c76 0x5dc554 0x7f514e48d4a0 0x7f514f23bd82 0x6ff469 0x792b79 0x7f514f237e9a 0x7f514e54accd
      mongo(_ZN5mongo15printStackTraceERSo+0x21) [0x7495d1]
      mongo(_ZN5mongo11msgassertedEiPKc+0x9b) [0x71189b]
      mongo() [0x711ddc]
      mongo(_ZNK5mongo11shell_utils18ConnectionRegistry30killOperationsOnAllConnectionsEb+0x1526) [0x5f3c76]
      mongo(_Z10quitNicelyi+0x24) [0x5dc554]
      /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f514e48d4a0]
      /lib/x86_64-linux-gnu/libpthread.so.0(pthread_cond_wait+0xc2) [0x7f514f23bd82]
      mongo(_ZN5mongo15DeadlineMonitorINS_7V8ScopeEE21deadlineMonitorThreadEv+0x3b9) [0x6ff469]
      mongo() [0x792b79]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f514f237e9a]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f514e54accd]
      Mon Jun 17 10:46:14.858 terminate() called in shell, printing stack:
      0x7495d1 0x5dc36e 0x7f514eddd846 0x7f514eddd873 0x7f514eddd96e 0x711941 0x711ddc 0x5f3c76 0x5dc554 0x7f514e48d4a0 0x7f514f23bd82 0x6ff469 0x792b79 0x7f514f237e9a 0x7f514e54accd
      mongo(_ZN5mongo15printStackTraceERSo+0x21) [0x7495d1]
      mongo(_Z11myterminatev+0x3e) [0x5dc36e]
      /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb5846) [0x7f514eddd846]
      /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb5873) [0x7f514eddd873]
      /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb596e) [0x7f514eddd96e]
      mongo(_ZN5mongo11msgassertedEiPKc+0x141) [0x711941]
      mongo() [0x711ddc]
      mongo(_ZNK5mongo11shell_utils18ConnectionRegistry30killOperationsOnAllConnectionsEb+0x1526) [0x5f3c76]
      mongo(_Z10quitNicelyi+0x24) [0x5dc554]
      /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f514e48d4a0]
      /lib/x86_64-linux-gnu/libpthread.so.0(pthread_cond_wait+0xc2) [0x7f514f23bd82]
      mongo(_ZN5mongo15DeadlineMonitorINS_7V8ScopeEE21deadlineMonitorThreadEv+0x3b9) [0x6ff469]
      mongo() [0x792b79]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f514f237e9a]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f514e54accd]
      Mon Jun 17 10:46:14 EEST 2013
      *****************************************************
      Mon Jun 17 10:46:15 EEST 2013
      MongoDB shell version: 2.4.4
      connecting to: hingo-sputnik:27000/rstest
      bye
      Mon Jun 17 10:46:15 EEST 2013
      *****************************************************
      Mon Jun 17 10:46:16 EEST 2013
      MongoDB shell version: 2.4.4
      connecting to: hingo-sputnik:27000/rstest
      bye
      Mon Jun 17 10:46:17 EEST 2013

      Show
      1) Setup a config server, a mongos and 3 replicaset members. 2) Run the primary node in a shell and in foreground (without --fork). 3) Run attached rstest.sh to create an insert load via mongos to the primary. 4) Stop the primary node with Ctrl-Z. 5) Wait forever 6) Notice that rstest.sh is stuck and not inserting anything anymore, even if a new primary is available and known to mongos. 7) press Ctrl-C in the rstest.sh shell. (twice) hingo@hingo-sputnik:~/hacking/rstests$ ./rstest.sh Mon Jun 17 10:43:18 EEST 2013 MongoDB shell version: 2.4.4 connecting to: hingo-sputnik:27000/rstest bye Mon Jun 17 10:43:19 EEST 2013 ***************************************************** Mon Jun 17 10:43:20 EEST 2013 MongoDB shell version: 2.4.4 connecting to: hingo-sputnik:27000/rstest --> Notice many minutes pass by: ^CMon Jun 17 10:46:03.942 Assertion: 13111:field not found, expected type 2 0x7495d1 0x71189b 0x711ddc 0x5f3c76 0x5dc554 0x7f514e48d4a0 0x7f514f23f14a 0x738aa3 0x738ab9 0x73cd05 0x73566c 0x7375db 0x737a94 0x64ce0f 0x67ab37 0x63dd4a 0x657b11 0x705efc 0x6e8a70 0x82ef62 mongo(_ZN5mongo15printStackTraceERSo+0x21) [0x7495d1] mongo(_ZN5mongo11msgassertedEiPKc+0x9b) [0x71189b] mongo() [0x711ddc] mongo(_ZNK5mongo11shell_utils18ConnectionRegistry30killOperationsOnAllConnectionsEb+0x1526) [0x5f3c76] mongo(_Z10quitNicelyi+0x24) [0x5dc554] /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f514e48d4a0] /lib/x86_64-linux-gnu/libpthread.so.0(recv+0x6a) [0x7f514f23f14a] mongo(_ZN5mongo6Socket5_recvEPci+0x13) [0x738aa3] mongo(_ZN5mongo6Socket11unsafe_recvEPci+0x9) [0x738ab9] mongo(_ZN5mongo6Socket4recvEPci+0x75) [0x73cd05] mongo(_ZN5mongo13MessagingPort4recvERNS_7MessageE+0x8c) [0x73566c] mongo( ZN5mongo13MessagingPort4recvERKNS_7MessageERS1 +0x1b) [0x7375db] mongo( ZN5mongo13MessagingPort4callERNS_7MessageES2 +0x34) [0x737a94] mongo(_ZN5mongo18DBClientConnection4callERNS_7MessageES2_bPSs+0x4f) [0x64ce0f] mongo(_ZN5mongo14DBClientCursor4initEv+0xb7) [0x67ab37] mongo(_ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0xea) [0x63dd4a] mongo(_ZN5mongo18DBClientConnection5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0xa1) [0x657b11] mongo(_ZN5mongo9mongoFindEPNS_7V8ScopeERKN2v89ArgumentsE+0x42c) [0x705efc] mongo(_ZN5mongo7V8Scope10v8CallbackERKN2v89ArgumentsE+0xb0) [0x6e8a70] mongo() [0x82ef62] Mon Jun 17 10:46:03.958 JavaScript execution failed: Error: field not found, expected type 2 at src/mongo/shell/query.js:L78 ^CMon Jun 17 10:46:14.853 Assertion: 13111:field not found, expected type 2 0x7495d1 0x71189b 0x711ddc 0x5f3c76 0x5dc554 0x7f514e48d4a0 0x7f514f23bd82 0x6ff469 0x792b79 0x7f514f237e9a 0x7f514e54accd mongo(_ZN5mongo15printStackTraceERSo+0x21) [0x7495d1] mongo(_ZN5mongo11msgassertedEiPKc+0x9b) [0x71189b] mongo() [0x711ddc] mongo(_ZNK5mongo11shell_utils18ConnectionRegistry30killOperationsOnAllConnectionsEb+0x1526) [0x5f3c76] mongo(_Z10quitNicelyi+0x24) [0x5dc554] /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f514e48d4a0] /lib/x86_64-linux-gnu/libpthread.so.0(pthread_cond_wait+0xc2) [0x7f514f23bd82] mongo(_ZN5mongo15DeadlineMonitorINS_7V8ScopeEE21deadlineMonitorThreadEv+0x3b9) [0x6ff469] mongo() [0x792b79] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f514f237e9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f514e54accd] Mon Jun 17 10:46:14.858 terminate() called in shell, printing stack: 0x7495d1 0x5dc36e 0x7f514eddd846 0x7f514eddd873 0x7f514eddd96e 0x711941 0x711ddc 0x5f3c76 0x5dc554 0x7f514e48d4a0 0x7f514f23bd82 0x6ff469 0x792b79 0x7f514f237e9a 0x7f514e54accd mongo(_ZN5mongo15printStackTraceERSo+0x21) [0x7495d1] mongo(_Z11myterminatev+0x3e) [0x5dc36e] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb5846) [0x7f514eddd846] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb5873) [0x7f514eddd873] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb596e) [0x7f514eddd96e] mongo(_ZN5mongo11msgassertedEiPKc+0x141) [0x711941] mongo() [0x711ddc] mongo(_ZNK5mongo11shell_utils18ConnectionRegistry30killOperationsOnAllConnectionsEb+0x1526) [0x5f3c76] mongo(_Z10quitNicelyi+0x24) [0x5dc554] /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f514e48d4a0] /lib/x86_64-linux-gnu/libpthread.so.0(pthread_cond_wait+0xc2) [0x7f514f23bd82] mongo(_ZN5mongo15DeadlineMonitorINS_7V8ScopeEE21deadlineMonitorThreadEv+0x3b9) [0x6ff469] mongo() [0x792b79] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f514f237e9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f514e54accd] Mon Jun 17 10:46:14 EEST 2013 ***************************************************** Mon Jun 17 10:46:15 EEST 2013 MongoDB shell version: 2.4.4 connecting to: hingo-sputnik:27000/rstest bye Mon Jun 17 10:46:15 EEST 2013 ***************************************************** Mon Jun 17 10:46:16 EEST 2013 MongoDB shell version: 2.4.4 connecting to: hingo-sputnik:27000/rstest bye Mon Jun 17 10:46:17 EEST 2013

      I didn't see a separate project for the shell, so continuing to send this to core server. Please reroute as needed.

      I believe this is the shell getting stuck, but could potentially be mongos too.

      Bug: When primary node stops (Ctrl-Z) during a query, client (the shell) will wait forever.

      Proposed fix: Client should timeout and error out.

        1. rstest.sh
          0.2 kB
          Henrik Ingo

            Assignee:
            backlog-server-platform DO NOT USE - Backlog - Platform Team
            Reporter:
            henrik.ingo@mongodb.com Henrik Ingo (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: