Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-18192

Crash running WiredTiger with "cache_resident=true"

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Minor - P4 Minor - P4
    • 3.0.3, 3.1.2
    • Affects Version/s: 3.0.2, 3.1.1
    • Component/s: WiredTiger
    • Labels:
      None
    • Environment:
      Linux mortar 3.16.0-33-generic #44~14.04.1-Ubuntu SMP Fri Mar 13 10:33:29 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
    • Fully Compatible
    • Linux

      Start mongod with:

      mongod --dbpath /home/dan/wt-data/ --storageEngine wiredTiger --wiredTigerCollectionConfigString 'cache_resident=true'
      

      Insert data. I've been using YCSB load, but this single threaded javascript loop will usually trigger the crash within a few seconds:

          db = db.getSisterDB("foo");
          col = db.getCollection("bar");
          var i = 0;
          while (i < 10000000 ) {
              var bulk = col.initializeUnorderedBulkOp();
              for (j = 0; j < 100; j++) {
                 bulk.insert({x:i, y: new Array(1000).join(".")});
              }
              var result = bulk.execute();
              i += j;
          }
          print(col.count() + " documents in " + dbName + "." + colName);
      

      Logs

      dan:(git)mongo[master]/$ rm -r /home/dan/wt-data/*; mkdir /home/dan/wt-data; mongod --dbpath /home/dan/wt-data/ --storageEngine wiredTiger --wiredTigerCollectionConfigString 'cache_resident=true'
      'mkdir: cannot create directory ‘/home/dan/wt-data’: File exists
      2015-04-23T10:57:54.118-0400 I STORAGE  Collection custom option: cache_resident=true
      2015-04-23T10:57:54.158-0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=62G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2015-04-23T10:57:54.217-0400 I CONTROL  [initandlisten] MongoDB starting : pid=168711 port=27017 dbpath=/home/dan/wt-data/ 64-bit host=mortar
      2015-04-23T10:57:54.217-0400 I CONTROL  [initandlisten]
      2015-04-23T10:57:54.217-0400 I CONTROL  [initandlisten] ** NOTE: This is a development version (3.1.2-pre-) of MongoDB.
      2015-04-23T10:57:54.217-0400 I CONTROL  [initandlisten] **       Not recommended for production.
      2015-04-23T10:57:54.217-0400 I CONTROL  [initandlisten]
      2015-04-23T10:57:54.219-0400 I CONTROL  [initandlisten]
      2015-04-23T10:57:54.219-0400 I CONTROL  [initandlisten] ** WARNING: You are running on a NUMA machine.
      2015-04-23T10:57:54.219-0400 I CONTROL  [initandlisten] **          We suggest launching mongod like this to avoid performance problems:
      2015-04-23T10:57:54.219-0400 I CONTROL  [initandlisten] **              numactl --interleave=all mongod [other options]
      2015-04-23T10:57:54.219-0400 I CONTROL  [initandlisten]
      2015-04-23T10:57:54.219-0400 I CONTROL  [initandlisten] db version v3.1.2-pre-
      2015-04-23T10:57:54.219-0400 I CONTROL  [initandlisten] git version: fd21f148c02f96995ca987ee08d874b06e8d752a
      2015-04-23T10:57:54.219-0400 I CONTROL  [initandlisten] build info: Linux mortar 3.16.0-33-generic #44~14.04.1-Ubuntu SMP Fri Mar 13 10:33:29 UTC 2015 x86_64 BOOST_LIB_VERSION=1_56
      2015-04-23T10:57:54.219-0400 I CONTROL  [initandlisten] allocator: tcmalloc
      2015-04-23T10:57:54.219-0400 I CONTROL  [initandlisten] options: { storage: { dbPath: "/home/dan/wt-data/", engine: "wiredTiger", wiredTiger: { collectionConfig: { configString: "cache_resident=true" } } } }
      2015-04-23T10:57:54.228-0400 I NETWORK  [initandlisten] waiting for connections on port 27017
      2015-04-23T10:57:54.561-0400 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:51812 #1 (1 connection now open)
      2015-04-23T10:57:56.642-0400 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:51813 #2 (2 connections now open)
      2015-04-23T10:58:18.048-0400 F -        Invalid access at address: 0
      2015-04-23T10:58:18.053-0400 F -        Got signal: 11 (Segmentation fault).
      
       0xf22b56 0xf22272 0xf225ae 0x7fa5fee78340 0x12e6afd 0x7fa5fee70182 0x7fa5feb9d47d
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"B22B56"},{"b":"400000","o":"B22272"},{"b":"400000","o":"B225AE"},{"b":"7FA5FEE68000","o":"10340"},{"b":"400000","o":"EE6AFD"},{"b":"7FA5FEE68000","o":"8182"},{"b":"7FA5FEAA3000","o":"FA47D"}],"processInfo":{ "mongodbVersion" : "3.1.2-pre-", "gitVersion" : "fd21f148c02f96995ca987ee08d874b06e8d752a", "uname" : { "sysname" : "Linux", "release" : "3.16.0-33-generic", "version" : "#44~14.04.1-Ubuntu SMP Fri Mar 13 10:33:29 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "419A5DF667D9441BBE29168CAC9113FC28C8B6CC" }, { "b" : "7FFFE5FBA000", "elfType" : 3, "buildId" : "5552B9335DDE9349419BA10896C1E75C9432A946" }, { "b" : "7FA5FFAAA000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "92FCF41EFE012D6186E31A59AD05BDBB487769AB" }, { "b" : "7FA5FF8A6000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C1AE4CB7195D337A77A3C689051DABAA3980CA0C" }, { "b" : "7FA5FF5A2000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "19EFDDAB11B3BF5C71570078C59F91CF6592CE9E" }, { "b" : "7FA5FF29C000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "1D76B71E905CB867B27CEF230FCB20F01A3178F5" }, { "b" : "7FA5FF086000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "8D0AA71411580EE6C08809695C3984769F25725B" }, { "b" : "7FA5FEE68000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9318E8AF0BFBE444731BB0461202EF57F7C39542" }, { "b" : "7FA5FEAA3000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "30C94DC66A1FE95180C3D68D2B89E576D5AE213C" }, { "b" : "7FA5FFCB2000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9F00581AB3C73E3AEA35995A0C50D24D59A01D47" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x26) [0xf22b56]
       mongod(+0xB22272) [0xf22272]
       mongod(+0xB225AE) [0xf225ae]
       libpthread.so.0(+0x10340) [0x7fa5fee78340]
       mongod(+0xEE6AFD) [0x12e6afd]
       libpthread.so.0(+0x8182) [0x7fa5fee70182]
       libc.so.6(clone+0x6D) [0x7fa5feb9d47d]
      -----  END BACKTRACE  -----
      

            Assignee:
            keith.bostic@mongodb.com Keith Bostic (Inactive)
            Reporter:
            dan@mongodb.com Daniel Pasette (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: