Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-9014

Mongod and mongos crash induced by many concurrent invocations of the getnonce command.

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 2.4.2, 2.5.0
    • Affects Version/s: 2.4.0
    • Component/s: Security
    • None
    • Environment:
      OS X - 10.8.2
    • ALL

      The SecureRandom object held by the singleton instance of CmdGetNonce is accessed in an unsychronized manner by every thread processing a getnonce command on behalf of a connection. SecureRandom is not internally synchronized. It may only be used by one thread at a time. The result is that under sufficient offered load of getnonce commands, eventually two will access the SecureRandom object concurrently, with undefined resultant behavior. One of the resultant behaviors is a segfault, on some systems.

      (Original description below)

      I tested this on 2.2.2 and it did not crash mongod.

      Script is attached. Let me know if you need help building (go app).

      In a nutshell, this script is opening and closing connections rapidly in a lot of different threads (goroutines).

      Mon Mar 18 11:28:50.628 [initandlisten] connection accepted from 127.0.0.1:60649 #1428 (100 connections now open)
      Mon Mar 18 11:28:50.637 Invalid access at address: 0x101fffff8 from thread: conn1417
      Mon Mar 18 11:28:50.637
      Mon Mar 18 11:28:50.637 Invalid access at address: 0x101fffff8 from thread: conn1418
      Mon Mar 18 11:28:50.637 Mon Mar 18 11:28:50.637 Invalid access at address: 0x101fffff8 from thread: conn1419
      Mon Mar 18 11:28:50.637 Mon Mar 18 11:28:50.637
      Invalid access at address: 0x101fffff8 from thread: conn1345
      Got signal: 11 (Segmentation fault: 11).
      
      Invalid access at address: 0x101fffff8 from thread: conn1421
      Invalid access at address: 0x101fffff8 from thread: conn1422
      Mon Mar 18 11:28:50.637
      
      Mon Mar 18 11:28:50.637
      
      Got signal: 11 (Segmentation fault: 11).
      Mon Mar 18 11:28:50.637 Got signal: 11 (Segmentation fault: 11).
      Mon Mar 18 11:28:50.637
      Mon Mar 18 11:28:50.637 Got signal: 11 (Segmentation fault: 11).
      
      Got signal: 11 (Segmentation fault: 11).
      Got signal: 11 (Segmentation fault: 11).
      

      This is on OSX.

      I have a standalone go script that caused this (attached).

      Mon Mar 18 11:28:50.638 Mon Mar 18 11:28:50.638 Invalid access at address: 0x101fffff8 from thread: conn1423
      Invalid access at address: 0x101fffff8 from thread: conn1424
      Mon Mar 18 11:28:50.638 Mon Mar 18 11:28:50.638
      Mon Mar 18 11:28:50.638
      Invalid access at address: 0x101fffff8 from thread: conn1425
      Invalid access at address: 0x101fffff8 from thread: conn1426
      Mon Mar 18 11:28:50.638 Mon Mar 18 11:28:50.638 Invalid access at address: 0x101fffff8 from thread: conn1427
      Mon Mar 18 11:28:50.638
      
      Invalid access at address: 0x101fffff8 from thread: conn1428
      Got signal: 11 (Segmentation fault: 11).
      
      Got signal: 11 (Segmentation fault: 11).
      
      Mon Mar 18 11:28:50.638 Mon Mar 18 11:28:50.638
      Mon Mar 18 11:28:50.638
      Got signal: 11 (Segmentation fault: 11).
      Got signal: 11 (Segmentation fault: 11).
      Mon Mar 18 11:28:50.638 Got signal: 11 (Segmentation fault: 11).
      
      
      Got signal: 11 (Segmentation fault: 11).
      
      
      Mon Mar 18 11:28:50.662 Backtrace:
      0x1006dbb8b 0x10000185b 0x100001f11 0x7fff8b6698ea 0x7fff00000002 0x7ea947259716e7ee
       0   mongod                              0x00000001006dbb8b _ZN5mongo15printStackTraceERSo + 43
       1   mongod                              0x000000010000185b _ZN5mongo10abruptQuitEi + 987
       2   mongod                              0x0000000100001f11 _ZN5mongo24abruptQuitWithAddrSignalEiP9__siginfoPv + 673
       3   libsystem_c.dylib                   0x00007fff8b6698ea _sigtramp + 26
       4   ???                                 0x00007fff00000002 0x0 + 140733193388034
       5   ???                                 0x7ea947259716e7ee 0x0 + 9126904346617243630
      Mon Mar 18 11:28:50.662
      Mon Mar 18 11:28:50.662 Backtrace:
      0x1006dbb8b 0x10000185b 0x100001f11 0x7fff8b6698ea 0x7fff00000002 0x7ea947259716e7ee
       0   mongod                              0x00000001006dbb8b _ZN5mongo15printStackTraceERSo + 43
       1   mongod                              0x000000010000185b _ZN5mongo10abruptQuitEi + 987
       2   mongod                              0x0000000100001f11 _ZN5mongo24abruptQuitWithAddrSignalEiP9__siginfoPv + 673
       3   libsystem_c.dylib                   0x00007fff8b6698ea _sigtramp + 26
       4   ???                                 0x00007fff00000002 0x0 + 140733193388034
       5   ???                                 0x7ea947259716e7ee 0x0 + 9126904346617243630
      Mon Mar 18 11:28:50.662 Mon Mar 18 11:28:50.662 Backtrace:
      0x1006dbb8b 0x10000185b 0x100001f11 0x7fff8b6698ea 0x7fff00000002 0x7ea947259716e7ee
       0   mongod                              0x00000001006dbb8b _ZN5mongo15printStackTraceERSo + 43
       1   mongod                              0x000000010000185b _ZN5mongo10abruptQuitEi + 987
       2   mongod                              0x0000000100001f11 _ZN5mongo24abruptQuitWithAddrSignalEiP9__siginfoPv + 673
      

        1. go-memory-test.tar.gz
          419 kB
        2. test.go
          2 kB

            Assignee:
            schwerin@mongodb.com Andy Schwerin
            Reporter:
            rn@deftlabs.com Ryan Nitz
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: