Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-9964

mongod crashes when client disconnects (OpenIndiana 157a8)

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical - P2
    • Resolution: Works as Designed
    • 2.4.4
    • None
    • Networking
    • Opensolaris x86-64 (openindiana 157a8) on both own build and smartos build (pthreads,
      gcc/g++ 4.6), boost 1.49 from 2.4.4 source-bundle.
    • Solaris
    • Hide

      start mongod (LANG=C, LC_ALL=C required, otherwise it crashes in localeImpl),
      then start mongo (local connection), type exit --> mongod crashes with
      segfault.

      Show
      start mongod (LANG=C, LC_ALL=C required, otherwise it crashes in localeImpl), then start mongo (local connection), type exit --> mongod crashes with segfault.

    Description

      When the client disconnects, mongod crashes with a segfault in
      mongo::ipToAddrs . The thread sees that the connection has closed
      [conn1] Socket recv() conn closed? 127.0.0.1:37593 (ret is zero in recv)
      but then tries to do something on the closed connection:

      env LC_ALL=C LANG=C dbx ./mongod
      Reading mongod
      Reading ld.so.1
      Reading libsocket.so.1
      Reading libresolv.so.2
      Reading libnsl.so.1
      Reading libdl.so.1
      Reading librt.so.1
      Reading libstdc++.so.6.0.16
      Reading libm.so.2
      Reading libgcc_s.so.1
      Reading libpthread.so.1
      Reading libc.so.1
      (dbx) run -vvvvv
      Running: mongod -vvvvv 
      (process id 6037)
      Wed Jun 19 11:09:01.301 BackgroundJob starting: DataFileSync
      Wed Jun 19 11:09:01.301 versionArrayTest passed
      Wed Jun 19 11:09:01.301 shardKeyTest passed
      Wed Jun 19 11:09:01.302 isInRangeTest passed
      Wed Jun 19 11:09:01.302 shardObjTest passed
      Wed Jun 19 11:09:01.302 Matcher::matches() { abcd: 3.1, abcdef: "123456789" }
      Wed Jun 19 11:09:01.302 Matcher::matches() { abcd: 3.1, abcdef: "123456789" }
      Wed Jun 19 11:09:01.302 Matcher::matches() { abcd: 3.1, abcdef: "123456789" }
      Wed Jun 19 11:09:01.302 Matcher::matches() { abcdef: "z23456789" }
      Wed Jun 19 11:09:01.302 Matcher::matches() { abcd: 3.1, abcdef: "123456789" }
      Wed Jun 19 11:09:01.302 Matcher::matches() { abcdef: "z23456789" }
      Wed Jun 19 11:09:01.305 [initandlisten] MongoDB starting : pid=6037 port=27017 dbpath=/data/db/ 64-bit host=imksunug1
      Wed Jun 19 11:09:01.305 [initandlisten] 
      Wed Jun 19 11:09:01.305 [initandlisten] ** NOTE: your operating system version does not support the method that MongoDB
      Wed Jun 19 11:09:01.305 [initandlisten] **       uses to detect impending page faults.
      Wed Jun 19 11:09:01.305 [initandlisten] **       This may result in slower performance for certain use cases
      Wed Jun 19 11:09:01.305 [initandlisten] 
      Wed Jun 19 11:09:01.305 [initandlisten] db version v2.4.4
      Wed Jun 19 11:09:01.305 [initandlisten] git version: nogitversion
      Wed Jun 19 11:09:01.305 [initandlisten] build info: SunOS imksunug1 5.11 oi_151a8 i86pc BOOST_LIB_VERSION=1_49
      Wed Jun 19 11:09:01.305 [initandlisten] allocator: system
      Wed Jun 19 11:09:01.305 [initandlisten] options: { vvvvv: true }
      Wed Jun 19 11:09:01.307 [initandlisten] journal dir=/data/db/journal
      Wed Jun 19 11:09:01.307 [initandlisten] recover begin
      Wed Jun 19 11:09:01.307 [initandlisten] recover lsn: 0
      Wed Jun 19 11:09:01.307 [initandlisten] recover /data/db/journal/j._0
      Wed Jun 19 11:09:01.307 [initandlisten] mmf open /data/db/local.0
      Wed Jun 19 11:09:01.307 [initandlisten] mmf finishOpening 0xffffdd7ffb000000 /data/db/local.0 len:67108864
      Wed Jun 19 11:09:01.308 [initandlisten] mmf open /data/db/local.ns
      Wed Jun 19 11:09:01.308 [initandlisten] mmf finishOpening 0xffffdd7ff5c00000 /data/db/local.ns len:16777216
      Wed Jun 19 11:09:01.333 [initandlisten] mmf close /data/db/local.0
      Wed Jun 19 11:09:01.334 [initandlisten] _groupCommit 
      Wed Jun 19 11:09:01.334 [initandlisten] journal REMAPPRIVATEVIEW
      Wed Jun 19 11:09:01.334 [initandlisten] journal REMAPPRIVATEVIEW done startedAt: 0 n:2 0ms
      Wed Jun 19 11:09:01.334 [initandlisten] groupCommit end
      Wed Jun 19 11:09:01.335 [initandlisten] mmf close /data/db/local.ns
      Wed Jun 19 11:09:01.335 [initandlisten] _groupCommit 
      Wed Jun 19 11:09:01.335 [initandlisten] journal REMAPPRIVATEVIEW
      Wed Jun 19 11:09:01.335 [initandlisten] journal REMAPPRIVATEVIEW done startedAt: 0 n:1 0ms
      Wed Jun 19 11:09:01.335 [initandlisten] groupCommit end
      Wed Jun 19 11:09:01.335 [initandlisten] recover cleaning up
      Wed Jun 19 11:09:01.335 [initandlisten] removeJournalFiles
      Wed Jun 19 11:09:01.335 [initandlisten] removeJournalFiles end
      Wed Jun 19 11:09:01.335 [initandlisten] recover done
      Wed Jun 19 11:09:02.608 [initandlisten] opening db:  local
      Wed Jun 19 11:09:02.609 [initandlisten] mmf open /data/db/local.ns
      Wed Jun 19 11:09:02.609 [initandlisten] mmf finishOpening 0xffffdd7ffde00000 /data/db/local.ns len:16777216
      Wed Jun 19 11:09:02.609 [initandlisten] mmf open /data/db/local.0
      Wed Jun 19 11:09:02.609 [initandlisten] mmf finishOpening 0xffffdd7ff8a00000 /data/db/local.0 len:67108864
      Wed Jun 19 11:09:02.609 [initandlisten] mmf close 
      Wed Jun 19 11:09:02.612 [initandlisten] enter repairDatabases (to check pdfile version #)
      Wed Jun 19 11:09:02.612 [initandlisten]         local
      Wed Jun 19 11:09:02.613 [initandlisten] mmf close /data/db/local.0
      Wed Jun 19 11:09:02.613 [initandlisten] _groupCommit 
      Wed Jun 19 11:09:02.613 [initandlisten] journal REMAPPRIVATEVIEW
      Wed Jun 19 11:09:02.613 [initandlisten] journal REMAPPRIVATEVIEW done startedAt: 0 n:1 0ms
      Wed Jun 19 11:09:02.613 [initandlisten] groupCommit end
      Wed Jun 19 11:09:02.614 [initandlisten] mmf close /data/db/local.ns
      Wed Jun 19 11:09:02.614 [initandlisten] _groupCommit 
      Wed Jun 19 11:09:02.614 [initandlisten] journal REMAPPRIVATEVIEW
      Wed Jun 19 11:09:02.615 [initandlisten] journal REMAPPRIVATEVIEW done startedAt: 1 n:1 0ms
      Wed Jun 19 11:09:02.615 [initandlisten] groupCommit end
      Wed Jun 19 11:09:02.616 [initandlisten] done repairDatabases
      Wed Jun 19 11:09:02.617 BackgroundJob starting: snapshot
      Wed Jun 19 11:09:02.619 BackgroundJob starting: PeriodicTask::Runner
      Wed Jun 19 11:09:02.619 BackgroundJob starting: ClientCursorMonitor
      Wed Jun 19 11:09:02.620 BackgroundJob starting: TTLMonitor
      Wed Jun 19 11:09:02.620 [initandlisten] runQuery called local.$cmd { create: "startup_log", size: 10485760, capped: true }
      Wed Jun 19 11:09:02.621 [initandlisten] run command local.$cmd { create: "startup_log", size: 10485760, capped: true }
      Wed Jun 19 11:09:02.621 [initandlisten] opening db:  local
      Wed Jun 19 11:09:02.621 [initandlisten] mmf open /data/db/local.ns
      Wed Jun 19 11:09:02.621 [initandlisten] mmf finishOpening 0xffffdd7ffd600000 /data/db/local.ns len:16777216
      Wed Jun 19 11:09:02.621 [initandlisten] mmf open /data/db/local.0
      Wed Jun 19 11:09:02.621 [initandlisten] mmf finishOpening 0xffffdd7ff8200000 /data/db/local.0 len:67108864
      Wed Jun 19 11:09:02.621 [initandlisten] mmf close 
      Wed Jun 19 11:09:02.622 [initandlisten] create collection local.startup_log { create: "startup_log", size: 10485760, capped: true }
      Wed Jun 19 11:09:02.622 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0  reslen:75 0ms
      Wed Jun 19 11:09:02.622 [initandlisten] info PageFaultRetryableSection will not yield, already locked upon reaching
      Wed Jun 19 11:09:02.622 [initandlisten] insert local.startup_log ninserted:1 keyUpdates:0  0ms
      Wed Jun 19 11:09:02.623 [initandlisten] fd limit hard:65536 soft:1024 max conn: 819
      Wed Jun 19 11:09:02.623 [websvr] fd limit hard:65536 soft:1024 max conn: 819
      Wed Jun 19 11:09:02.623 [initandlisten] waiting for connections on port 27017
      Wed Jun 19 11:09:02.623 [websvr] admin web console waiting for connections on port 28017
      Wed Jun 19 11:09:02.716 [journal] lsn set 0
      Wed Jun 19 11:09:02.716 [journal] journal WRITETODATAFILES 1
      Wed Jun 19 11:09:02.716 [journal] journal WRITETODATAFILES 2
      Wed Jun 19 11:09:02.716 [journal] journal WRITETODATAFILES 0.37ms
      Wed Jun 19 11:09:03.637 [journal] _groupCommit 
      Wed Jun 19 11:09:03.637 [journal] _groupCommit upgrade
      Wed Jun 19 11:09:03.637 [journal] journal REMAPPRIVATEVIEW
      Wed Jun 19 11:09:03.637 [journal] journal REMAPPRIVATEVIEW done startedAt: 0 n:1 0ms
      Wed Jun 19 11:09:03.637 [journal] groupCommit end
      ...
      Wed Jun 19 11:09:13.868 [journal] _groupCommit 
      Wed Jun 19 11:09:13.868 [journal] _groupCommit upgrade
      Wed Jun 19 11:09:13.868 [journal] journal REMAPPRIVATEVIEW
      Wed Jun 19 11:09:13.868 [journal] journal REMAPPRIVATEVIEW done startedAt: 0 n:1 0ms
      Wed Jun 19 11:09:13.868 [journal] groupCommit end
      Wed Jun 19 11:09:14.338 [initandlisten] connection accepted from 127.0.0.1:37593 #1 (1 connection now open)
      Wed Jun 19 11:09:14.339 [conn1] runQuery called admin.$cmd { whatsmyuri: 1 }
      Wed Jun 19 11:09:14.339 [conn1] run command admin.$cmd { whatsmyuri: 1 }
      Wed Jun 19 11:09:14.340 [conn1] command admin.$cmd command: { whatsmyuri: 1 } ntoreturn:1 keyUpdates:0  reslen:62 0ms
      Wed Jun 19 11:09:14.345 [conn1] runQuery called admin.$cmd { getLog: "startupWarnings" }
      Wed Jun 19 11:09:14.345 [conn1] run command admin.$cmd { getLog: "startupWarnings" }
      Wed Jun 19 11:09:14.345 [conn1] command: { getLog: "startupWarnings" }
      Wed Jun 19 11:09:14.345 [conn1] command admin.$cmd command: { getLog: "startupWarnings" } ntoreturn:1 keyUpdates:0  reslen:503 0ms
      Wed Jun 19 11:09:14.349 [conn1] runQuery called admin.$cmd { replSetGetStatus: 1.0, forShell: 1.0 }
      Wed Jun 19 11:09:14.349 [conn1] run command admin.$cmd { replSetGetStatus: 1.0, forShell: 1.0 }
      Wed Jun 19 11:09:14.349 [conn1] command: { replSetGetStatus: 1.0, forShell: 1.0 }
      Wed Jun 19 11:09:14.349 [conn1] command admin.$cmd command: { replSetGetStatus: 1.0, forShell: 1.0 } ntoreturn:1 keyUpdates:0  reslen:76 0ms
      Wed Jun 19 11:09:14.895 [journal] _groupCommit 
      Wed Jun 19 11:09:14.895 [journal] _groupCommit upgrade
      Wed Jun 19 11:09:14.895 [journal] journal REMAPPRIVATEVIEW
      Wed Jun 19 11:09:14.895 [journal] journal REMAPPRIVATEVIEW done startedAt: 1 n:1 0ms
      Wed Jun 19 11:09:14.895 [journal] groupCommit end
      ...
      Wed Jun 19 11:09:20.011 [journal] _groupCommit 
      Wed Jun 19 11:09:20.011 [journal] _groupCommit upgrade
      Wed Jun 19 11:09:20.011 [journal] journal REMAPPRIVATEVIEW
      Wed Jun 19 11:09:20.011 [journal] journal REMAPPRIVATEVIEW done startedAt: 0 n:1 0ms
      Wed Jun 19 11:09:20.011 [journal] groupCommit end
      Wed Jun 19 11:09:20.085 [conn1] Socket recv() conn closed? 127.0.0.1:37593
      t@12 (l@12) signal SEGV (no mapping at the fault address) in mongo::ipToAddrs at 0xdd897d
      0x0000000000dd897d: ipToAddrs+0x00ed:   addb     %al,(%rax)
      (dbx) where
      current thread: t@12
      =>[1] mongo::ipToAddrs(0x1, 0x1, 0x474e5543432b2b00, 0x332cc30, 0xffffdd7ff81fe6c8, 0x332cc30), at 0xdd897d 
        ---- hidden frames, use 'where -h' to see them all ----
        [3] __cxa_throw(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xffffdd7fffb43ad9 
        [4] mongo::Socket::_handleRecvError(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xde3069 
        [5] mongo::Socket::recv(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xde3531 
        [6] mongo::MessagingPort::recv(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xddab9e 
        [7] mongo::PortMessageServer::handleIncomingMsg(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xddd6e4 
        [8] thread_proxy(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xe3e0be 
        [9] _thrp_setup(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xffffdd7fff8b7184 
        [10] _lwp_start(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xffffdd7fff8b7450 
      (dbx) where -h
      current thread: t@12
      =>[1] mongo::ipToAddrs(0x1, 0x1, 0x474e5543432b2b00, 0x332cc30, 0xffffdd7ff81fe6c8, 0x332cc30), at 0xdd897d 
        [2] _Unwind_RaiseException(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xffffdd7fff8bef05 
        [3] __cxa_throw(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xffffdd7fffb43ad9 
        [4] mongo::Socket::_handleRecvError(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xde3069 
        [5] mongo::Socket::recv(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xde3531 
        [6] mongo::MessagingPort::recv(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xddab9e 
        [7] mongo::PortMessageServer::handleIncomingMsg(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xddd6e4 
        [8] thread_proxy(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xe3e0be 
        [9] _thrp_setup(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xffffdd7fff8b7184 
        [10] _lwp_start(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xffffdd7fff8b7450 
      (dbx) 

      Attachments

        Issue Links

          Activity

            People

              tad Tad Marshall
              brightstar Udo Grabowski
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: