Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-10607

strange behavior after deletes, may not be causal, shard locked up

    XMLWordPrintableJSON

Details

    • Icon: Question Question
    • Resolution: Cannot Reproduce
    • Icon: Major - P3 Major - P3
    • None
    • None
    • Shell
    • None
    • ubuntu 10.04.4 LTS, mongodb-10gen 2.2.1,

    Description

      Getting errors when doing a query. Earlier the primary was locked up when a user was doing deletes, typical of what he has done before. This time server locked up, had to restart the shard. Came back no issues. Now when I go to another mongos and do show dbs,it hangs. Ctrl C out and I get this error repeatedly. Fun part is that it is not consistent, I close the session and come back, another show dbs returns immediately with the expected results.

      mongos> show dbs
      ^CThu Aug 22 15:28:02 Assertion: 13111:field not found, expected type 2
      0x6073f1 0x5d1aa9 0x5d1c2c 0x4b7fc7 0x49f947 0x7fdd4bdfbba0 0x7fdd4cb12d3a 0x5f87d3 0x5f87e9 0x5fbdd4 0x5f4f5c 0x5f5a0b 0x5f5ec4 0x4f6695 0x5203dd 0x4eab4a 0x5010b4 0x5bd2ef 0x6aa5ac 0x69c025 
       mongo(_ZN5mongo15printStackTraceERSo+0x21) [0x6073f1]
       mongo(_ZN5mongo11msgassertedEiPKc+0x99) [0x5d1aa9]
       mongo() [0x5d1c2c]
       mongo(_ZNK5mongo11shell_utils18ConnectionRegistry30killOperationsOnAllConnectionsEb+0xf17) [0x4b7fc7]
       mongo(_Z10quitNicelyi+0x27) [0x49f947]
       /lib/libc.so.6(+0x33ba0) [0x7fdd4bdfbba0]
       /lib/libpthread.so.0(recv+0x6a) [0x7fdd4cb12d3a]
       mongo(_ZN5mongo6Socket5_recvEPci+0x13) [0x5f87d3]
       mongo(_ZN5mongo6Socket11unsafe_recvEPci+0x9) [0x5f87e9]
       mongo(_ZN5mongo6Socket4recvEPci+0xc4) [0x5fbdd4]
       mongo(_ZN5mongo13MessagingPort4recvERNS_7MessageE+0x8c) [0x5f4f5c]
       mongo(_ZN5mongo13MessagingPort4recvERKNS_7MessageERS1_+0x1b) [0x5f5a0b]
       mongo(_ZN5mongo13MessagingPort4callERNS_7MessageES2_+0x34) [0x5f5ec4]
       mongo(_ZN5mongo18DBClientConnection4callERNS_7MessageES2_bPSs+0x55) [0x4f6695]
       mongo(_ZN5mongo14DBClientCursor4initEv+0xbd) [0x5203dd]
       mongo(_ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0xea) [0x4eab4a]
       mongo(_ZN5mongo18DBClientConnection5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0xa4) [0x5010b4]
       mongo(_ZN5mongo10mongo_findEP9JSContextP8JSObjectjPlS4_+0x32f) [0x5bd2ef]
       mongo(js_Invoke+0x40c) [0x6aa5ac]
       mongo(js_Interpret+0x1305) [0x69c025]
      Thu Aug 22 15:28:02 Error: field not found, expected type 2 src/mongo/shell/collection.js:155
      Thu Aug 22 15:29:08 ERROR: MessagingPort::call() wrong id got:2 expect:4
        toSend op: 2004
        response msgid:167783245
        response len:  2987
        response op:  1
        remote: 127.0.0.1:27017
      Thu Aug 22 15:29:08   Assertion failure false src/mongo/util/net/message_port.cpp 241
      0x6073f1 0x5d21bd 0x5f5c6c 0x5f5ec4 0x4f6695 0x5203dd 0x4eab4a 0x5010b4 0x5bd2ef 0x6aa5ac 0x69c025 0x6aa13f 0x668e46 0x668ec2 0x66a2ce 0x5cbcc4 0x4a1226 0x4a41f6 0x4a67e6 0x7fdd4bde6c4d 
       mongo(_ZN5mongo15printStackTraceERSo+0x21) [0x6073f1]
       mongo(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0x5d21bd]
       mongo(_ZN5mongo13MessagingPort4recvERKNS_7MessageERS1_+0x27c) [0x5f5c6c]
       mongo(_ZN5mongo13MessagingPort4callERNS_7MessageES2_+0x34) [0x5f5ec4]
       mongo(_ZN5mongo18DBClientConnection4callERNS_7MessageES2_bPSs+0x55) [0x4f6695]
       mongo(_ZN5mongo14DBClientCursor4initEv+0xbd) [0x5203dd]
       mongo(_ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0xea) [0x4eab4a]
       mongo(_ZN5mongo18DBClientConnection5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0xa4) [0x5010b4]
       mongo(_ZN5mongo10mongo_findEP9JSContextP8JSObjectjPlS4_+0x32f) [0x5bd2ef]
       mongo(js_Invoke+0x40c) [0x6aa5ac]
       mongo(js_Interpret+0x1305) [0x69c025]
       mongo(js_Execute+0x36f) [0x6aa13f]
       mongo(JS_EvaluateUCScriptForPrincipals+0x66) [0x668e46]
       mongo(JS_EvaluateUCScript+0x22) [0x668ec2]
       mongo(JS_EvaluateScript+0x6e) [0x66a2ce]
       mongo(_ZN5mongo7SMScope4execERKNS_10StringDataERKSsbbbi+0x144) [0x5cbcc4]
       mongo(_Z10execPromptRN5mongo5ScopeEPKcRSs+0x146) [0x4a1226]
       mongo(_Z5_mainiPPc+0x1ea6) [0x4a41f6]
       mongo(main+0x26) [0x4a67e6]
       /lib/libc.so.6(__libc_start_main+0xfd) [0x7fdd4bde6c4d]
      > 
      Thu Aug 22 15:31:43 ERROR: MessagingPort::call() wrong id got:4 expect:5
        toSend op: 2004
        response msgid:167783246
        response len:  131
        response op:  1
        remote: 127.0.0.1:27017
      Thu Aug 22 15:31:43   Assertion failure false src/mongo/util/net/message_port.cpp 241
      0x6073f1 0x5d21bd 0x5f5c6c 0x5f5ec4 0x4f6695 0x5203dd 0x4eab4a 0x5010b4 0x5bd2ef 0x6aa5ac 0x69c025 0x6aa13f 0x668e46 0x668ec2 0x66a2ce 0x5cbcc4 0x4a1226 0x4a41f6 0x4a67e6 0x7fdd4bde6c4d 
       mongo(_ZN5mongo15printStackTraceERSo+0x21) [0x6073f1]
       mongo(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0x5d21bd]
       mongo(_ZN5mongo13MessagingPort4recvERKNS_7MessageERS1_+0x27c) [0x5f5c6c]
       mongo(_ZN5mongo13MessagingPort4callERNS_7MessageES2_+0x34) [0x5f5ec4]
       mongo(_ZN5mongo18DBClientConnection4callERNS_7MessageES2_bPSs+0x55) [0x4f6695]
       mongo(_ZN5mongo14DBClientCursor4initEv+0xbd) [0x5203dd]
       mongo(_ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0xea) [0x4eab4a]
       mongo(_ZN5mongo18DBClientConnection5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0xa4) [0x5010b4]
       mongo(_ZN5mongo10mongo_findEP9JSContextP8JSObjectjPlS4_+0x32f) [0x5bd2ef]
       mongo(js_Invoke+0x40c) [0x6aa5ac]
       mongo(js_Interpret+0x1305) [0x69c025]
       mongo(js_Execute+0x36f) [0x6aa13f]
       mongo(JS_EvaluateUCScriptForPrincipals+0x66) [0x668e46]
       mongo(JS_EvaluateUCScript+0x22) [0x668ec2]
       mongo(JS_EvaluateScript+0x6e) [0x66a2ce]
       mongo(_ZN5mongo7SMScope4execERKNS_10StringDataERKSsbbbi+0x144) [0x5cbcc4]
       mongo(_Z10execPromptRN5mongo5ScopeEPKcRSs+0x146) [0x4a1226]
       mongo(_Z5_mainiPPc+0x1ea6) [0x4a41f6]
       mongo(main+0x26) [0x4a67e6]
       /lib/libc.so.6(__libc_start_main+0xfd) [0x7fdd4bde6c4d]
      > 
      Thu Aug 22 15:31:43 ERROR: MessagingPort::call() wrong id got:5 expect:6
        toSend op: 2004
        response msgid:167784659
        response len:  131
        response op:  1
        remote: 127.0.0.1:27017
      Thu Aug 22 15:31:43   Assertion failure false src/mongo/util/net/message_port.cpp 241

      Attachments

        Activity

          People

            Unassigned Unassigned
            jerry.carpenter@reputation.com Jerry Carpenter
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: