Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17319

invariant at shutdown rc9, rc10, rc11 with wiredTiger

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical - P2
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-rc8
    • Fix Version/s: 3.0.1
    • Component/s: Storage, WiredTiger
    • Labels:
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Steps To Reproduce:
      Hide

      Sysbench workload, same as SERVER-17261

      Show
      Sysbench workload, same as SERVER-17261

      Description

      Using the 2/12 nightly build, pre-rc9 79492d9cc1885d74b31b5fe24194dbc227096d6e, found an invariant failure on Amazon Linux in EC2 (C3-8XL) upon shutdown.

      Cache was full and checkpoints outstanding at time of shutdown (kill -15) but mixed read/write sysbench execute workload had completed.

      2015-02-18T18:49:15.895+0000 I NETWORK  [conn56] end connection 127.0.0.1:43053 (1 connection now open)
      2015-02-18T18:49:26.135+0000 I NETWORK  [conn10] end connection 127.0.0.1:43004 (0 connections now open)
      2015-02-18T18:49:44.740+0000 I CONTROL  [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
      2015-02-18T18:49:44.753+0000 I CONTROL  [signalProcessingThread] now exiting
      2015-02-18T18:49:44.753+0000 I NETWORK  [signalProcessingThread] shutdown: going to close listening sockets...
      2015-02-18T18:49:44.753+0000 I NETWORK  [signalProcessingThread] closing listening socket: 6
      2015-02-18T18:49:44.753+0000 I NETWORK  [signalProcessingThread] closing listening socket: 7
      2015-02-18T18:49:44.756+0000 I NETWORK  [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
      2015-02-18T18:49:44.761+0000 I NETWORK  [signalProcessingThread] shutdown: going to flush diaglog...
      2015-02-18T18:49:44.761+0000 I NETWORK  [signalProcessingThread] shutdown: going to close sockets...
      2015-02-18T18:49:44.766+0000 I STORAGE  [signalProcessingThread] WiredTigerKVEngine shutting down
      2015-02-18T18:51:28.501+0000 E STORAGE  [signalProcessingThread] WiredTiger (0) [1424285488:501446][30154:0x7f2de5bbf700], log-server: Connection has open file handles: sizeStorer.wt
      2015-02-18T18:51:28.502+0000 E STORAGE  WiredTiger (0) [1424285488:502494][30154:0x7f2de53be700], eviction-server: cache server: exiting with 2352566 pages in memory and 2352564 pages evicted
      2015-02-18T18:51:28.502+0000 E STORAGE  WiredTiger (0) [1424285488:502567][30154:0x7f2de53be700], eviction-server: cache server: exiting with 7026 bytes in memory
      2015-02-18T18:51:28.502+0000 E STORAGE  WiredTiger (0) [1424285488:502582][30154:0x7f2de53be700], eviction-server: cache server: exiting with 6777 bytes dirty and 1 pages dirty
      2015-02-18T18:51:28.511+0000 I -        [signalProcessingThread] Invariant failure: _conn->close(_conn, config) resulted in status UnknownError 16: Device or resource busy at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 206
      2015-02-18T18:51:28.596+0000 I CONTROL  [signalProcessingThread]
       0xf41659 0xeeb791 0xed176a 0xd56707 0xa69d1a 0xa99c09 0xf40506 0xf8ee04 0x7f2ded0b7f18 0x7f2dec1c9b2d
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"B41659"},{"b":"400000","o":"AEB791"},{"b":"400000","o":"AD176A"},{"b":"400000","o":"956707"},{"b":"400000","o":"669D1A"},{"b":"400000","o":"699C09"},{"b":"400000","o":"B40506"},{"b":"400000","o":"B8EE04"},{"b":"7F2DED0B0000","o":"7F18"},{"b":"7F2DEC0E7000","o":"E2B2D"}],"processInfo":{ "mongodbVersion" : "3.0.0-rc9-pre-", "gitVersion" : "79492d9cc1885d74b31b5fe24194dbc227096d6e", "uname" : { "sysname" : "Linux", "release" : "3.14.20-20.44.amzn1.x86_64", "version" : "#1 SMP Mon Oct 6 22:52:46 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFCA0FE000", "elfType" : 3 }, { "b" : "7F2DED0B0000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7F2DECEA8000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7F2DECCA4000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7F2DEC9A0000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F2DEC6A2000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7F2DEC48C000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F2DEC0E7000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7F2DED2CC000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf41659]
       mongod(_ZN5mongo10logContextEPKc+0xE1) [0xeeb791]
       mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0xDA) [0xed176a]
       mongod(_ZN5mongo18WiredTigerKVEngine13cleanShutdownEv+0x107) [0xd56707]
       mongod(_ZN5mongo23GlobalEnvironmentMongoD34shutdownGlobalStorageEngineCleanlyEv+0x1A) [0xa69d1a]
       mongod(_ZN5mongo11exitCleanlyENS_8ExitCodeE+0x219) [0xa99c09]
       mongod(+0xB40506) [0xf40506]
       mongod(+0xB8EE04) [0xf8ee04]
       libpthread.so.0(+0x7F18) [0x7f2ded0b7f18]
       libc.so.6(clone+0x6D) [0x7f2dec1c9b2d]
      -----  END BACKTRACE  -----
      2015-02-18T18:51:28.596+0000 I -        [signalProcessingThread]
       
      ***aborting after invariant() failure

      1. invariant.html
        728 kB
        Quentin Conner
      2. iostat.log
        69 kB
        Quentin Conner
      3. mongodb-sysbench.log
        20.50 MB
        Quentin Conner
      4. ss.log
        1.08 MB
        Quentin Conner
      1. invariant.png
        110 kB

        Issue Links

          Activity

          Hide
          quentin.conner Quentin Conner (Inactive) added a comment - - edited

          I have been using the following javascript to "fill up" the wired tiger cache (watch top to see when RSS gets to about half the physical DRAM). After this mongo script finishes, send signal 2 or signal 15 to the mongod process. A few trials may be needed.

          'use strict';
           
          load('jstests/core/parallelTester.js');
           
          function insert(myCollection, desired, randSeed) {
              var batchSize = 1000
           
              print('bulk loading ' + desired + ' documents to ' + myCollection + ' with batch size of ' + batchSize)
              var count = 0
           
              var mydb = db.getSiblingDB('sbtest')
           
              var seed = randSeed
           
              while (count < desired) {
                  var bulk = mydb.getCollection(myCollection).initializeUnorderedBulkOp();
                  for (var j=0; j < batchSize; count++, j++) {
                      var k = Math.round(random()*10000000)
                      var c = sysbenchString()
                      var pad = '' + Math.round(random()*100000000000)
                      for (var y = 0; y < 4; y++)
                          pad += '-' + Math.round(random()*100000000000)
                      var d = { _id: count, k: k, c: c, pad: pad }
                      //print(tojson(d))
                      bulk.insert(d)
                  }
                  bulk.execute();
              }
           
              // create indexes
              print('creating index')
              mydb.getCollection(myCollection).ensureIndex({k: 1})
           
              function sysbenchString() {
                  var s = '' + Math.round(random()*100000000000)
                  for (var x = 0; x < 8; x++)
                      s += '-' + Math.round(random()*100000000000)
                  return s
              }
           
              // PRNG borrowed from http://stackoverflow.com/questions/521295/javascript-random-seeds
              function random() {
                  var x = Math.sin(seed++) * 10000
                  return x - Math.floor(x)
              }
           
          }
           
           
          function simulate_sysbench_load(num_collections, num_docs_per_collection) {
              // data load phase
              print("creating data for", num_collections, " collections")
           
              // drop old collections
              print('dropping old collections:')
              var mydb = db.getSiblingDB('sbtest')
              for (var offset=1; offset <= num_collections; offset++) {
                  var s = 'sbtest' + offset
                  print(s)
                  mydb.getCollection(s).drop()
              }
           
              var threads = []
           
              print('\nloading new collections:')
              var t = Math.round(Date.now() / num_collections)
              for (var offset=0; offset < num_collections; offset++) {
                  var n = offset + 1
                  var s = 'sbtest' + n
                  threads[offset] = new ScopedThread(insert, s, num_docs_per_collection, (offset + 1) * t)
                  threads[offset].start()
              }
           
              print("all threads started")
              threads.forEach(function(t) {
                  t.join();
              });
              threads = []
          }
           
          // load data
          var collection_count = 16
          var docs_per_collection = 20000000
          simulate_sysbench_load(collection_count, docs_per_collection)

          Show
          quentin.conner Quentin Conner (Inactive) added a comment - - edited I have been using the following javascript to "fill up" the wired tiger cache (watch top to see when RSS gets to about half the physical DRAM). After this mongo script finishes, send signal 2 or signal 15 to the mongod process. A few trials may be needed. 'use strict';   load('jstests/core/parallelTester.js');   function insert(myCollection, desired, randSeed) { var batchSize = 1000   print('bulk loading ' + desired + ' documents to ' + myCollection + ' with batch size of ' + batchSize) var count = 0   var mydb = db.getSiblingDB('sbtest')   var seed = randSeed   while (count < desired) { var bulk = mydb.getCollection(myCollection).initializeUnorderedBulkOp(); for (var j=0; j < batchSize; count++, j++) { var k = Math.round(random()*10000000) var c = sysbenchString() var pad = '' + Math.round(random()*100000000000) for (var y = 0; y < 4; y++) pad += '-' + Math.round(random()*100000000000) var d = { _id: count, k: k, c: c, pad: pad } //print(tojson(d)) bulk.insert(d) } bulk.execute(); }   // create indexes print('creating index') mydb.getCollection(myCollection).ensureIndex({k: 1})   function sysbenchString() { var s = '' + Math.round(random()*100000000000) for (var x = 0; x < 8; x++) s += '-' + Math.round(random()*100000000000) return s }   // PRNG borrowed from http://stackoverflow.com/questions/521295/javascript-random-seeds function random() { var x = Math.sin(seed++) * 10000 return x - Math.floor(x) }   }     function simulate_sysbench_load(num_collections, num_docs_per_collection) { // data load phase print("creating data for", num_collections, " collections")   // drop old collections print('dropping old collections:') var mydb = db.getSiblingDB('sbtest') for (var offset=1; offset <= num_collections; offset++) { var s = 'sbtest' + offset print(s) mydb.getCollection(s).drop() }   var threads = []   print('\nloading new collections:') var t = Math.round(Date.now() / num_collections) for (var offset=0; offset < num_collections; offset++) { var n = offset + 1 var s = 'sbtest' + n threads[offset] = new ScopedThread(insert, s, num_docs_per_collection, (offset + 1) * t) threads[offset].start() }   print("all threads started") threads.forEach(function(t) { t.join(); }); threads = [] }   // load data var collection_count = 16 var docs_per_collection = 20000000 simulate_sysbench_load(collection_count, docs_per_collection)
          Hide
          nickj Nick Judson added a comment -

          Yes, just ^C. IIRC data was still being written (bulk insert/update via c# beta 2 driver) when the db was shutdown. I'll see if I can repro with another test.

          Show
          nickj Nick Judson added a comment - Yes, just ^C. IIRC data was still being written (bulk insert/update via c# beta 2 driver) when the db was shutdown. I'll see if I can repro with another test.
          Hide
          oleg@evergage.com Oleg Rekutin added a comment -

          Saw same while shutting down a replica on rc9 after being up for a while and having a constant insert workload:

          2015-03-03T18:08:39.142+0000 I -        [signalProcessingThread] Invariant failure: _conn->close(_conn, config) resulted in status UnknownError 16: Device or resource busy at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 206
          2015-03-03T18:08:39.268+0000 I CONTROL  [signalProcessingThread] 
           0xf5ae39 0xefb651 0xee25aa 0xd71ed7 0xa96bea 0xac6789 0xf5a026 0xfa7894 0x7fb114d65f18 0x7fb113822b2d
          ----- BEGIN BACKTRACE -----
          {"backtrace":[{"b":"400000","o":"B5AE39"},{"b":"400000","o":"AFB651"},{"b":"400000","o":"AE25AA"},{"b":"400000","o":"971ED7"},{"b":"400000","o":"696BEA"},{"b":"400000","o":"6C6789"},{"b":"400000","o":"B5A026"},{"b":"400000","o":"BA7894"},{"b":"7FB114D5E000","o":"7F18"},{"b":"7FB113740000","o":"E2B2D"}],"processInfo":{ "mongodbVersion" : "3.0.0-rc9", "gitVersion" : "e6577bc37a2edba81b99146934cf7bad00c6e1b2", "uname" : { "sysname" : "Linux", "release" : "3.4.73-64.112.amzn1.x86_64", "version" : "#1 SMP Tue Dec 10 01:50:05 UTC 2013", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "0EC0EADFA8F4AD45610DC4515F6732A685D68742" }, { "b" : "7FFF06C10000", "elfType" : 3, "buildId" : "6A7E58423C94E707E8B4A45F098E78D557DD5A7D" }, { "b" : "7FB114D5E000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "A8E8015F84374807F6920DF95D16BBAAAC02AB41" }, { "b" : "7FB114AF1000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "A5C6FCE56D21542B9535D5FBCFE74EBA71D24613" }, { "b" : "7FB11470D000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "75565934551BECB21A027BE0A3AE28A62AEEB7EE" }, { "b" : "7FB114505000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "CC3700E5EACA3A520D1FD1BA3B469F2EFB1B8F11" }, { "b" : "7FB114301000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "3D5EB706F159BCCB9A3B3E5F528B512E2C500F90" }, { "b" : "7FB113FF8000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "66F1CF311C61879639BD3DC0034DEE0D6D042261" }, { "b" : "7FB113CFA000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "226E54A756FDC288551092B4FDE818C8D6D6FC04" }, { "b" : "7FB113AE5000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "E77BA674F63D5C56373C03316B5E74C5C781A0BC" }, { "b" : "7FB113740000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "D40AD0538A2FFAB96E5540EF73816C675CC0AF5E" }, { "b" : "7FB114F7A000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "37F70DE2ABEA184F24518EB5CC538FF0F54E081F" }, { "b" : "7FB1134FD000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "9DF61878D8918F25CC74AD01F417FDB051DFE3DA" }, { "b" : "7FB113218000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "6F1DB0F811D1B210520443442D4437BC43BF9A80" }, { "b" : "7FB113015000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "1A6E97644CC9149C2E1871C6AE1DB51975E78A41" }, { "b" : "7FB112DEA000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "F7DF34078FD7BFD684FE46D5F677EEDA1D9B9DC9" }, { "b" : "7FB112BD4000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "87B4EBF2183C8EA4AB657212203EFFE6340E2F4F" }, { "b" : "7FB1129C9000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "381960ACAB9C39461D58BDE7B272C4F61BB3582F" }, { "b" : "7FB1127C6000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "BF48CD5658DE95CE058C4B828E81C97E2AE19643" }, { "b" : "7FB1125AF000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "F01971089728188EFA4AA473D8B2DE22CA6DE963" }, { "b" : "7FB11238E000", "path" : "/usr/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "803D7EF21A989677D056E52BAEB9AB5B154FB9D9" } ] }}
           mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf5ae39]
           mongod(_ZN5mongo10logContextEPKc+0xE1) [0xefb651]
           mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0xDA) [0xee25aa]
           mongod(_ZN5mongo18WiredTigerKVEngine13cleanShutdownEv+0x107) [0xd71ed7]
           mongod(_ZN5mongo23GlobalEnvironmentMongoD34shutdownGlobalStorageEngineCleanlyEv+0x1A) [0xa96bea]
           mongod(_ZN5mongo11exitCleanlyENS_8ExitCodeE+0x219) [0xac6789]
           mongod(+0xB5A026) [0xf5a026]
           mongod(+0xBA7894) [0xfa7894]
           libpthread.so.0(+0x7F18) [0x7fb114d65f18]
           libc.so.6(clone+0x6D) [0x7fb113822b2d]
          -----  END BACKTRACE  -----
          2015-03-03T18:08:39.268+0000 I -        [signalProcessingThread] 
           
          ***aborting after invariant() failure

          Using rc9 Amazon build on Amazon Linux.3.4.73-64.112.amzn1.x86_64

          Show
          oleg@evergage.com Oleg Rekutin added a comment - Saw same while shutting down a replica on rc9 after being up for a while and having a constant insert workload: 2015-03-03T18:08:39.142+0000 I - [signalProcessingThread] Invariant failure: _conn->close(_conn, config) resulted in status UnknownError 16: Device or resource busy at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 206 2015-03-03T18:08:39.268+0000 I CONTROL [signalProcessingThread] 0xf5ae39 0xefb651 0xee25aa 0xd71ed7 0xa96bea 0xac6789 0xf5a026 0xfa7894 0x7fb114d65f18 0x7fb113822b2d ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"B5AE39"},{"b":"400000","o":"AFB651"},{"b":"400000","o":"AE25AA"},{"b":"400000","o":"971ED7"},{"b":"400000","o":"696BEA"},{"b":"400000","o":"6C6789"},{"b":"400000","o":"B5A026"},{"b":"400000","o":"BA7894"},{"b":"7FB114D5E000","o":"7F18"},{"b":"7FB113740000","o":"E2B2D"}],"processInfo":{ "mongodbVersion" : "3.0.0-rc9", "gitVersion" : "e6577bc37a2edba81b99146934cf7bad00c6e1b2", "uname" : { "sysname" : "Linux", "release" : "3.4.73-64.112.amzn1.x86_64", "version" : "#1 SMP Tue Dec 10 01:50:05 UTC 2013", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "0EC0EADFA8F4AD45610DC4515F6732A685D68742" }, { "b" : "7FFF06C10000", "elfType" : 3, "buildId" : "6A7E58423C94E707E8B4A45F098E78D557DD5A7D" }, { "b" : "7FB114D5E000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "A8E8015F84374807F6920DF95D16BBAAAC02AB41" }, { "b" : "7FB114AF1000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "A5C6FCE56D21542B9535D5FBCFE74EBA71D24613" }, { "b" : "7FB11470D000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "75565934551BECB21A027BE0A3AE28A62AEEB7EE" }, { "b" : "7FB114505000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "CC3700E5EACA3A520D1FD1BA3B469F2EFB1B8F11" }, { "b" : "7FB114301000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "3D5EB706F159BCCB9A3B3E5F528B512E2C500F90" }, { "b" : "7FB113FF8000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "66F1CF311C61879639BD3DC0034DEE0D6D042261" }, { "b" : "7FB113CFA000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "226E54A756FDC288551092B4FDE818C8D6D6FC04" }, { "b" : "7FB113AE5000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "E77BA674F63D5C56373C03316B5E74C5C781A0BC" }, { "b" : "7FB113740000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "D40AD0538A2FFAB96E5540EF73816C675CC0AF5E" }, { "b" : "7FB114F7A000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "37F70DE2ABEA184F24518EB5CC538FF0F54E081F" }, { "b" : "7FB1134FD000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "9DF61878D8918F25CC74AD01F417FDB051DFE3DA" }, { "b" : "7FB113218000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "6F1DB0F811D1B210520443442D4437BC43BF9A80" }, { "b" : "7FB113015000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "1A6E97644CC9149C2E1871C6AE1DB51975E78A41" }, { "b" : "7FB112DEA000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "F7DF34078FD7BFD684FE46D5F677EEDA1D9B9DC9" }, { "b" : "7FB112BD4000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "87B4EBF2183C8EA4AB657212203EFFE6340E2F4F" }, { "b" : "7FB1129C9000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "381960ACAB9C39461D58BDE7B272C4F61BB3582F" }, { "b" : "7FB1127C6000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "BF48CD5658DE95CE058C4B828E81C97E2AE19643" }, { "b" : "7FB1125AF000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "F01971089728188EFA4AA473D8B2DE22CA6DE963" }, { "b" : "7FB11238E000", "path" : "/usr/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "803D7EF21A989677D056E52BAEB9AB5B154FB9D9" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf5ae39] mongod(_ZN5mongo10logContextEPKc+0xE1) [0xefb651] mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0xDA) [0xee25aa] mongod(_ZN5mongo18WiredTigerKVEngine13cleanShutdownEv+0x107) [0xd71ed7] mongod(_ZN5mongo23GlobalEnvironmentMongoD34shutdownGlobalStorageEngineCleanlyEv+0x1A) [0xa96bea] mongod(_ZN5mongo11exitCleanlyENS_8ExitCodeE+0x219) [0xac6789] mongod(+0xB5A026) [0xf5a026] mongod(+0xBA7894) [0xfa7894] libpthread.so.0(+0x7F18) [0x7fb114d65f18] libc.so.6(clone+0x6D) [0x7fb113822b2d] ----- END BACKTRACE ----- 2015-03-03T18:08:39.268+0000 I - [signalProcessingThread]   ***aborting after invariant() failure Using rc9 Amazon build on Amazon Linux.3.4.73-64.112.amzn1.x86_64
          Hide
          quentin.conner Quentin Conner (Inactive) added a comment -

          Testing with 085ad3aa18938262a6664ccda6b5fad509c1f68e on the v3.0 branch is going well. No invariants so far with three trials completed.

          My plan is to conduct at least 12 trials to achieve a reasonable degree of confidence. Previously we would see the invariant in one out of three trials.

          Show
          quentin.conner Quentin Conner (Inactive) added a comment - Testing with 085ad3aa18938262a6664ccda6b5fad509c1f68e on the v3.0 branch is going well. No invariants so far with three trials completed. My plan is to conduct at least 12 trials to achieve a reasonable degree of confidence. Previously we would see the invariant in one out of three trials.
          Hide
          quentin.conner Quentin Conner (Inactive) added a comment -

          13 successful test trials using r3.0.1-rc0

          Show
          quentin.conner Quentin Conner (Inactive) added a comment - 13 successful test trials using r3.0.1-rc0

            People

            • Votes:
              1 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: