Mongod and mongos crash induced by many concurrent invocations of the getnonce command.

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Done
    • Priority: Major - P3
    • 2.4.2, 2.5.0
    • Affects Version/s: 2.4.0
    • Component/s: Security
    • None
    • Environment:
      OS X - 10.8.2
    • ALL
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None

      The SecureRandom object held by the singleton instance of CmdGetNonce is accessed in an unsychronized manner by every thread processing a getnonce command on behalf of a connection. SecureRandom is not internally synchronized. It may only be used by one thread at a time. The result is that under sufficient offered load of getnonce commands, eventually two will access the SecureRandom object concurrently, with undefined resultant behavior. One of the resultant behaviors is a segfault, on some systems.

      (Original description below)

      I tested this on 2.2.2 and it did not crash mongod.

      Script is attached. Let me know if you need help building (go app).

      In a nutshell, this script is opening and closing connections rapidly in a lot of different threads (goroutines).

      Mon Mar 18 11:28:50.628 [initandlisten] connection accepted from 127.0.0.1:60649 #1428 (100 connections now open)
Mon Mar 18 11:28:50.637 Invalid access at address: 0x101fffff8 from thread: conn1417
Mon Mar 18 11:28:50.637
Mon Mar 18 11:28:50.637 Invalid access at address: 0x101fffff8 from thread: conn1418
Mon Mar 18 11:28:50.637 Mon Mar 18 11:28:50.637 Invalid access at address: 0x101fffff8 from thread: conn1419
Mon Mar 18 11:28:50.637 Mon Mar 18 11:28:50.637
Invalid access at address: 0x101fffff8 from thread: conn1345
Got signal: 11 (Segmentation fault: 11).

Invalid access at address: 0x101fffff8 from thread: conn1421
Invalid access at address: 0x101fffff8 from thread: conn1422
Mon Mar 18 11:28:50.637

Mon Mar 18 11:28:50.637

Got signal: 11 (Segmentation fault: 11).
Mon Mar 18 11:28:50.637 Got signal: 11 (Segmentation fault: 11).
Mon Mar 18 11:28:50.637
Mon Mar 18 11:28:50.637 Got signal: 11 (Segmentation fault: 11).

Got signal: 11 (Segmentation fault: 11).
Got signal: 11 (Segmentation fault: 11).

      This is on OSX.

      I have a standalone go script that caused this (attached).

      Mon Mar 18 11:28:50.638 Mon Mar 18 11:28:50.638 Invalid access at address: 0x101fffff8 from thread: conn1423
Invalid access at address: 0x101fffff8 from thread: conn1424
Mon Mar 18 11:28:50.638 Mon Mar 18 11:28:50.638
Mon Mar 18 11:28:50.638
Invalid access at address: 0x101fffff8 from thread: conn1425
Invalid access at address: 0x101fffff8 from thread: conn1426
Mon Mar 18 11:28:50.638 Mon Mar 18 11:28:50.638 Invalid access at address: 0x101fffff8 from thread: conn1427
Mon Mar 18 11:28:50.638

Invalid access at address: 0x101fffff8 from thread: conn1428
Got signal: 11 (Segmentation fault: 11).

Got signal: 11 (Segmentation fault: 11).

Mon Mar 18 11:28:50.638 Mon Mar 18 11:28:50.638
Mon Mar 18 11:28:50.638
Got signal: 11 (Segmentation fault: 11).
Got signal: 11 (Segmentation fault: 11).
Mon Mar 18 11:28:50.638 Got signal: 11 (Segmentation fault: 11).


Got signal: 11 (Segmentation fault: 11).


Mon Mar 18 11:28:50.662 Backtrace:
0x1006dbb8b 0x10000185b 0x100001f11 0x7fff8b6698ea 0x7fff00000002 0x7ea947259716e7ee
 0   mongod                              0x00000001006dbb8b _ZN5mongo15printStackTraceERSo + 43
 1   mongod                              0x000000010000185b _ZN5mongo10abruptQuitEi + 987
 2   mongod                              0x0000000100001f11 _ZN5mongo24abruptQuitWithAddrSignalEiP9__siginfoPv + 673
 3   libsystem_c.dylib                   0x00007fff8b6698ea _sigtramp + 26
 4   ???                                 0x00007fff00000002 0x0 + 140733193388034
 5   ???                                 0x7ea947259716e7ee 0x0 + 9126904346617243630
Mon Mar 18 11:28:50.662
Mon Mar 18 11:28:50.662 Backtrace:
0x1006dbb8b 0x10000185b 0x100001f11 0x7fff8b6698ea 0x7fff00000002 0x7ea947259716e7ee
 0   mongod                              0x00000001006dbb8b _ZN5mongo15printStackTraceERSo + 43
 1   mongod                              0x000000010000185b _ZN5mongo10abruptQuitEi + 987
 2   mongod                              0x0000000100001f11 _ZN5mongo24abruptQuitWithAddrSignalEiP9__siginfoPv + 673
 3   libsystem_c.dylib                   0x00007fff8b6698ea _sigtramp + 26
 4   ???                                 0x00007fff00000002 0x0 + 140733193388034
 5   ???                                 0x7ea947259716e7ee 0x0 + 9126904346617243630
Mon Mar 18 11:28:50.662 Mon Mar 18 11:28:50.662 Backtrace:
0x1006dbb8b 0x10000185b 0x100001f11 0x7fff8b6698ea 0x7fff00000002 0x7ea947259716e7ee
 0   mongod                              0x00000001006dbb8b _ZN5mongo15printStackTraceERSo + 43
 1   mongod                              0x000000010000185b _ZN5mongo10abruptQuitEi + 987
 2   mongod                              0x0000000100001f11 _ZN5mongo24abruptQuitWithAddrSignalEiP9__siginfoPv + 673

        1. go-memory-test.tar.gz
          419 kB
        2. test.go
          2 kB

            Assignee:
            Andy Schwerin
            Reporter:
            Ryan Nitz
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: