Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17141

session->truncate failed with File Not Found in WiredTigerRecordStoreThread for local.oplog.rs

XMLWordPrintableJSON

    • Fully Compatible
    • ALL
    • Hide

      Using an r3.4xlarge instance (16 cpu, 122g ram, ephemeral storage for data) the attached mongo.conf (modded for dbpath and logpath), [https://github.com/tmcallaghan/iibench-mongodb|iibench] with the attached changes and shell script, and mongod run as follows:

      numactl --interleave=all ./mongodb-linux-x86_64-3.0.0-rc8/bin/mongod --config mongo.conf --storageEngine wiredTiger --wiredTigerCacheSizeGB 70 --master

      Workload is run on same machine. bash run.2.simple.bash 1000000

      jdk-7u25, mongo-java-driver-2.12.5.jar.

      Show
      Using an r3.4xlarge instance (16 cpu, 122g ram, ephemeral storage for data) the attached mongo.conf (modded for dbpath and logpath), [https://github.com/tmcallaghan/iibench-mongodb|iibench] with the attached changes and shell script, and mongod run as follows: numactl --interleave=all ./mongodb-linux-x86_64-3.0.0-rc8/bin/mongod --config mongo.conf --storageEngine wiredTiger --wiredTigerCacheSizeGB 70 --master Workload is run on same machine. bash run.2.simple.bash 1000000 jdk-7u25, mongo-java-driver-2.12.5.jar.

      Running insert workload, primary failed about 12 hours in.

      2015-01-31T05:26:24.818+0000 I COMMAND  [conn47] command socialite.$cmd command: insert { insert: "followers", ordered: true, documents: [ { _id: ObjectId('54cc6780e4b0aa9307e5a96e'), _f: "1036285", _t: "2457926" } ] } keyUpdates:0 writeConflicts:0 numYields:0 reslen:80 115ms
2015-01-31T05:26:25.491+0000 I QUERY    [conn689] getmore local.oplog.rs cursorid:23299506973 ntoreturn:0 keyUpdates:0 writeConflicts:0 numYields:257 nreturned:32969 reslen:4194377 200ms
2015-01-31T05:26:28.257+0000 I -        [WiredTigerRecordStoreThread for local.oplog.rs] Invariant failure: session->truncate(session, NULL, start, c, NULL) resulted in status UnknownError 2: No such file or directory at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 517
2015-01-31T05:26:28.273+0000 I QUERY    [conn690] getmore local.oplog.rs cursorid:22034715965 ntoreturn:0 keyUpdates:0 writeConflicts:0 numYields:257 nreturned:32967 reslen:4194299 205ms
2015-01-31T05:26:28.416+0000 I CONTROL  [WiredTigerRecordStoreThread for local.oplog.rs] 
 0xf3a9d9 0xee4b21 0xecaafa 0xd563fa 0xd5b4a4 0xecbbc0 0xf88114 0x7f7a624a2f18 0x7f7a615b4b9d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B3A9D9"},{"b":"400000","o":"AE4B21"},{"b":"400000","o":"ACAAFA"},{"b":"400000","o":"9563FA"},{"b":"400000","o":"95B4A4"},{"b":"400000","o":"ACBBC0"},{"b":"400000","o":"B88114"},{"b":"7F7A6249B000","o":"7F18"},{"b":"7F7A614D2000","o":"E2B9D"}],"processInfo":{ "mongodbVersion" : "3.0.0-rc7", "gitVersion" : "e4c60053b2967e16f765fa25d16aa6d629faa196", "uname" : { "sysname" : "Linux", "release" : "3.14.20-20.44.amzn1.x86_64", "version" : "#1 SMP Mon Oct 6 22:52:46 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFC7EFE000", "elfType" : 3 }, { "b" : "7F7A6249B000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7F7A62293000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7F7A6208F000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7F7A61D8B000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F7A61A8D000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7F7A61877000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F7A614D2000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7F7A626B7000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf3a9d9]
 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xee4b21]
 mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0xDA) [0xecaafa]
 mongod(_ZN5mongo21WiredTigerRecordStore27cappedDeleteAsNeeded_inlockEPNS_16OperationContextERKNS_8RecordIdE+0x78A) [0xd563fa]
 mongod(+0x95B4A4) [0xd5b4a4]
 mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0x120) [0xecbbc0]
 mongod(+0xB88114) [0xf88114]
 libpthread.so.0(+0x7F18) [0x7f7a624a2f18]
 libc.so.6(clone+0x6D) [0x7f7a615b4b9d]
-----  END BACKTRACE  -----
2015-01-31T05:26:28.422+0000 I -        [WiredTigerRecordStoreThread for local.oplog.rs] 

***aborting after invariant() failure

        1. run.simple.bash
          4 kB
        2. run.2.simple.bash
          4 kB
        3. mongo.conf
          0.5 kB
        4. jmongoiibench.java
          35 kB
        5. commandline
          0.1 kB

            Assignee:
            keith.bostic@mongodb.com Keith Bostic (Inactive)
            Reporter:
            michael.grundy Michael Grundy
            Votes:
            1 Vote for this issue
            Watchers:
            16 Start watching this issue

              Created:
              Updated:
              Resolved: