Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-9014

Mongod and mongos crash induced by many concurrent invocations of the getnonce command.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major - P3
    • Resolution: Fixed
    • 2.4.0
    • 2.4.2, 2.5.0
    • Security
    • None
    • OS X - 10.8.2
    • ALL

    Description

      The SecureRandom object held by the singleton instance of CmdGetNonce is accessed in an unsychronized manner by every thread processing a getnonce command on behalf of a connection. SecureRandom is not internally synchronized. It may only be used by one thread at a time. The result is that under sufficient offered load of getnonce commands, eventually two will access the SecureRandom object concurrently, with undefined resultant behavior. One of the resultant behaviors is a segfault, on some systems.

      (Original description below)

      I tested this on 2.2.2 and it did not crash mongod.

      Script is attached. Let me know if you need help building (go app).

      In a nutshell, this script is opening and closing connections rapidly in a lot of different threads (goroutines).

      Mon Mar 18 11:28:50.628 [initandlisten] connection accepted from 127.0.0.1:60649 #1428 (100 connections now open)
      		 
      Mon Mar 18 11:28:50.637 Invalid access at address: 0x101fffff8 from thread: conn1417
      		 
      Mon Mar 18 11:28:50.637
      		 
      Mon Mar 18 11:28:50.637 Invalid access at address: 0x101fffff8 from thread: conn1418
      		 
      Mon Mar 18 11:28:50.637 Mon Mar 18 11:28:50.637 Invalid access at address: 0x101fffff8 from thread: conn1419
      		 
      Mon Mar 18 11:28:50.637 Mon Mar 18 11:28:50.637
      		 
      Invalid access at address: 0x101fffff8 from thread: conn1345
      		 
      Got signal: 11 (Segmentation fault: 11).
      		 
       
      		 
      Invalid access at address: 0x101fffff8 from thread: conn1421
      		 
      Invalid access at address: 0x101fffff8 from thread: conn1422
      		 
      Mon Mar 18 11:28:50.637
      		 
       
      		 
      Mon Mar 18 11:28:50.637
      		 
       
      		 
      Got signal: 11 (Segmentation fault: 11).
      		 
      Mon Mar 18 11:28:50.637 Got signal: 11 (Segmentation fault: 11).
      		 
      Mon Mar 18 11:28:50.637
      		 
      Mon Mar 18 11:28:50.637 Got signal: 11 (Segmentation fault: 11).
      		 
       
      		 
      Got signal: 11 (Segmentation fault: 11).
      		 
      Got signal: 11 (Segmentation fault: 11).

      This is on OSX.

      I have a standalone go script that caused this (attached).

      Mon Mar 18 11:28:50.638 Mon Mar 18 11:28:50.638 Invalid access at address: 0x101fffff8 from thread: conn1423
      		 
      Invalid access at address: 0x101fffff8 from thread: conn1424
      		 
      Mon Mar 18 11:28:50.638 Mon Mar 18 11:28:50.638
      		 
      Mon Mar 18 11:28:50.638
      		 
      Invalid access at address: 0x101fffff8 from thread: conn1425
      		 
      Invalid access at address: 0x101fffff8 from thread: conn1426
      		 
      Mon Mar 18 11:28:50.638 Mon Mar 18 11:28:50.638 Invalid access at address: 0x101fffff8 from thread: conn1427
      		 
      Mon Mar 18 11:28:50.638
      		 
       
      		 
      Invalid access at address: 0x101fffff8 from thread: conn1428
      		 
      Got signal: 11 (Segmentation fault: 11).
      		 
       
      		 
      Got signal: 11 (Segmentation fault: 11).
      		 
       
      		 
      Mon Mar 18 11:28:50.638 Mon Mar 18 11:28:50.638
      		 
      Mon Mar 18 11:28:50.638
      		 
      Got signal: 11 (Segmentation fault: 11).
      		 
      Got signal: 11 (Segmentation fault: 11).
      		 
      Mon Mar 18 11:28:50.638 Got signal: 11 (Segmentation fault: 11).
      		 
       
      		 
       
      		 
      Got signal: 11 (Segmentation fault: 11).
      		 
       
      		 
       
      		 
      Mon Mar 18 11:28:50.662 Backtrace:
      		 
      0x1006dbb8b 0x10000185b 0x100001f11 0x7fff8b6698ea 0x7fff00000002 0x7ea947259716e7ee
      		 
       0   mongod                              0x00000001006dbb8b _ZN5mongo15printStackTraceERSo + 43
      		 
       1   mongod                              0x000000010000185b _ZN5mongo10abruptQuitEi + 987
      		 
       2   mongod                              0x0000000100001f11 _ZN5mongo24abruptQuitWithAddrSignalEiP9__siginfoPv + 673
      		 
       3   libsystem_c.dylib                   0x00007fff8b6698ea _sigtramp + 26
      		 
       4   ???                                 0x00007fff00000002 0x0 + 140733193388034
      		 
       5   ???                                 0x7ea947259716e7ee 0x0 + 9126904346617243630
      		 
      Mon Mar 18 11:28:50.662
      		 
      Mon Mar 18 11:28:50.662 Backtrace:
      		 
      0x1006dbb8b 0x10000185b 0x100001f11 0x7fff8b6698ea 0x7fff00000002 0x7ea947259716e7ee
      		 
       0   mongod                              0x00000001006dbb8b _ZN5mongo15printStackTraceERSo + 43
      		 
       1   mongod                              0x000000010000185b _ZN5mongo10abruptQuitEi + 987
      		 
       2   mongod                              0x0000000100001f11 _ZN5mongo24abruptQuitWithAddrSignalEiP9__siginfoPv + 673
      		 
       3   libsystem_c.dylib                   0x00007fff8b6698ea _sigtramp + 26
      		 
       4   ???                                 0x00007fff00000002 0x0 + 140733193388034
      		 
       5   ???                                 0x7ea947259716e7ee 0x0 + 9126904346617243630
      		 
      Mon Mar 18 11:28:50.662 Mon Mar 18 11:28:50.662 Backtrace:
      		 
      0x1006dbb8b 0x10000185b 0x100001f11 0x7fff8b6698ea 0x7fff00000002 0x7ea947259716e7ee
      		 
       0   mongod                              0x00000001006dbb8b _ZN5mongo15printStackTraceERSo + 43
      		 
       1   mongod                              0x000000010000185b _ZN5mongo10abruptQuitEi + 987
      		 
       2   mongod                              0x0000000100001f11 _ZN5mongo24abruptQuitWithAddrSignalEiP9__siginfoPv + 673

      Attachments

        1. go-memory-test.tar.gz
          419 kB
        2. test.go
          2 kB

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SERVER-10331 Mongodb SEGFAULTS with a null pointer

          • Blocker - P1 - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. SERVER-9329 Invalid access at address - Segmentation fault

          • Major - P3 - Major loss of function.
          • Closed
          related to

          Bug - A problem which impairs or prevents the functions of the product. SERVER-9362 Mongod crashes at first getnonce call if process started without stdout and stderr file descriptors initialized.

          • Minor - P4 - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Activity

            People

              schwerin@mongodb.com Andy Schwerin
              rgnitz Ryan Nitz
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: