Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17319

invariant at shutdown rc9, rc10, rc11 with wiredTiger

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Critical - P2 Critical - P2
    • 3.0.1
    • Affects Version/s: 3.0.0-rc8
    • Component/s: Storage, WiredTiger
    • Labels:
    • Fully Compatible
    • ALL
    • Hide

      Sysbench workload, same as SERVER-17261

      Show
      Sysbench workload, same as SERVER-17261

      Using the 2/12 nightly build, pre-rc9 79492d9cc1885d74b31b5fe24194dbc227096d6e, found an invariant failure on Amazon Linux in EC2 (C3-8XL) upon shutdown.

      Cache was full and checkpoints outstanding at time of shutdown (kill -15) but mixed read/write sysbench execute workload had completed.

      2015-02-18T18:49:15.895+0000 I NETWORK  [conn56] end connection 127.0.0.1:43053 (1 connection now open)
      2015-02-18T18:49:26.135+0000 I NETWORK  [conn10] end connection 127.0.0.1:43004 (0 connections now open)
      2015-02-18T18:49:44.740+0000 I CONTROL  [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
      2015-02-18T18:49:44.753+0000 I CONTROL  [signalProcessingThread] now exiting
      2015-02-18T18:49:44.753+0000 I NETWORK  [signalProcessingThread] shutdown: going to close listening sockets...
      2015-02-18T18:49:44.753+0000 I NETWORK  [signalProcessingThread] closing listening socket: 6
      2015-02-18T18:49:44.753+0000 I NETWORK  [signalProcessingThread] closing listening socket: 7
      2015-02-18T18:49:44.756+0000 I NETWORK  [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
      2015-02-18T18:49:44.761+0000 I NETWORK  [signalProcessingThread] shutdown: going to flush diaglog...
      2015-02-18T18:49:44.761+0000 I NETWORK  [signalProcessingThread] shutdown: going to close sockets...
      2015-02-18T18:49:44.766+0000 I STORAGE  [signalProcessingThread] WiredTigerKVEngine shutting down
      2015-02-18T18:51:28.501+0000 E STORAGE  [signalProcessingThread] WiredTiger (0) [1424285488:501446][30154:0x7f2de5bbf700], log-server: Connection has open file handles: sizeStorer.wt
      2015-02-18T18:51:28.502+0000 E STORAGE  WiredTiger (0) [1424285488:502494][30154:0x7f2de53be700], eviction-server: cache server: exiting with 2352566 pages in memory and 2352564 pages evicted
      2015-02-18T18:51:28.502+0000 E STORAGE  WiredTiger (0) [1424285488:502567][30154:0x7f2de53be700], eviction-server: cache server: exiting with 7026 bytes in memory
      2015-02-18T18:51:28.502+0000 E STORAGE  WiredTiger (0) [1424285488:502582][30154:0x7f2de53be700], eviction-server: cache server: exiting with 6777 bytes dirty and 1 pages dirty
      2015-02-18T18:51:28.511+0000 I -        [signalProcessingThread] Invariant failure: _conn->close(_conn, config) resulted in status UnknownError 16: Device or resource busy at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 206
      2015-02-18T18:51:28.596+0000 I CONTROL  [signalProcessingThread]
       0xf41659 0xeeb791 0xed176a 0xd56707 0xa69d1a 0xa99c09 0xf40506 0xf8ee04 0x7f2ded0b7f18 0x7f2dec1c9b2d
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"B41659"},{"b":"400000","o":"AEB791"},{"b":"400000","o":"AD176A"},{"b":"400000","o":"956707"},{"b":"400000","o":"669D1A"},{"b":"400000","o":"699C09"},{"b":"400000","o":"B40506"},{"b":"400000","o":"B8EE04"},{"b":"7F2DED0B0000","o":"7F18"},{"b":"7F2DEC0E7000","o":"E2B2D"}],"processInfo":{ "mongodbVersion" : "3.0.0-rc9-pre-", "gitVersion" : "79492d9cc1885d74b31b5fe24194dbc227096d6e", "uname" : { "sysname" : "Linux", "release" : "3.14.20-20.44.amzn1.x86_64", "version" : "#1 SMP Mon Oct 6 22:52:46 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFCA0FE000", "elfType" : 3 }, { "b" : "7F2DED0B0000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7F2DECEA8000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7F2DECCA4000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7F2DEC9A0000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F2DEC6A2000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7F2DEC48C000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F2DEC0E7000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7F2DED2CC000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf41659]
       mongod(_ZN5mongo10logContextEPKc+0xE1) [0xeeb791]
       mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0xDA) [0xed176a]
       mongod(_ZN5mongo18WiredTigerKVEngine13cleanShutdownEv+0x107) [0xd56707]
       mongod(_ZN5mongo23GlobalEnvironmentMongoD34shutdownGlobalStorageEngineCleanlyEv+0x1A) [0xa69d1a]
       mongod(_ZN5mongo11exitCleanlyENS_8ExitCodeE+0x219) [0xa99c09]
       mongod(+0xB40506) [0xf40506]
       mongod(+0xB8EE04) [0xf8ee04]
       libpthread.so.0(+0x7F18) [0x7f2ded0b7f18]
       libc.so.6(clone+0x6D) [0x7f2dec1c9b2d]
      -----  END BACKTRACE  -----
      2015-02-18T18:51:28.596+0000 I -        [signalProcessingThread]
      
      ***aborting after invariant() failure
      

        1. invariant.html
          728 kB
        2. invariant.png
          invariant.png
          110 kB
        3. iostat.log
          69 kB
        4. mongodb-sysbench.log
          20.50 MB
        5. ss.log
          1.08 MB

            Assignee:
            quentin.conner Quentin Conner
            Reporter:
            quentin.conner Quentin Conner
            Votes:
            1 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated:
              Resolved: