Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17141

session->truncate failed with File Not Found in WiredTigerRecordStoreThread for local.oplog.rs

    Details

    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Completed:
    • Steps To Reproduce:
      Hide

      Using an r3.4xlarge instance (16 cpu, 122g ram, ephemeral storage for data) the attached mongo.conf (modded for dbpath and logpath), [https://github.com/tmcallaghan/iibench-mongodb|iibench] with the attached changes and shell script, and mongod run as follows:

      numactl --interleave=all ./mongodb-linux-x86_64-3.0.0-rc8/bin/mongod --config mongo.conf --storageEngine wiredTiger --wiredTigerCacheSizeGB 70 --master

      Workload is run on same machine. bash run.2.simple.bash 1000000

      jdk-7u25, mongo-java-driver-2.12.5.jar.

      Show
      Using an r3.4xlarge instance (16 cpu, 122g ram, ephemeral storage for data) the attached mongo.conf (modded for dbpath and logpath), [https://github.com/tmcallaghan/iibench-mongodb|iibench] with the attached changes and shell script, and mongod run as follows: numactl --interleave=all ./mongodb-linux-x86_64-3.0.0-rc8/bin/mongod --config mongo.conf --storageEngine wiredTiger --wiredTigerCacheSizeGB 70 --master Workload is run on same machine. bash run.2.simple.bash 1000000 jdk-7u25, mongo-java-driver-2.12.5.jar.

      Description

      Running insert workload, primary failed about 12 hours in.

      2015-01-31T05:26:24.818+0000 I COMMAND  [conn47] command socialite.$cmd command: insert { insert: "followers", ordered: true, documents: [ { _id: ObjectId('54cc6780e4b0aa9307e5a96e'), _f: "1036285", _t: "2457926" } ] } keyUpdates:0 writeConflicts:0 numYields:0 reslen:80 115ms
      2015-01-31T05:26:25.491+0000 I QUERY    [conn689] getmore local.oplog.rs cursorid:23299506973 ntoreturn:0 keyUpdates:0 writeConflicts:0 numYields:257 nreturned:32969 reslen:4194377 200ms
      2015-01-31T05:26:28.257+0000 I -        [WiredTigerRecordStoreThread for local.oplog.rs] Invariant failure: session->truncate(session, NULL, start, c, NULL) resulted in status UnknownError 2: No such file or directory at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 517
      2015-01-31T05:26:28.273+0000 I QUERY    [conn690] getmore local.oplog.rs cursorid:22034715965 ntoreturn:0 keyUpdates:0 writeConflicts:0 numYields:257 nreturned:32967 reslen:4194299 205ms
      2015-01-31T05:26:28.416+0000 I CONTROL  [WiredTigerRecordStoreThread for local.oplog.rs] 
       0xf3a9d9 0xee4b21 0xecaafa 0xd563fa 0xd5b4a4 0xecbbc0 0xf88114 0x7f7a624a2f18 0x7f7a615b4b9d
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"B3A9D9"},{"b":"400000","o":"AE4B21"},{"b":"400000","o":"ACAAFA"},{"b":"400000","o":"9563FA"},{"b":"400000","o":"95B4A4"},{"b":"400000","o":"ACBBC0"},{"b":"400000","o":"B88114"},{"b":"7F7A6249B000","o":"7F18"},{"b":"7F7A614D2000","o":"E2B9D"}],"processInfo":{ "mongodbVersion" : "3.0.0-rc7", "gitVersion" : "e4c60053b2967e16f765fa25d16aa6d629faa196", "uname" : { "sysname" : "Linux", "release" : "3.14.20-20.44.amzn1.x86_64", "version" : "#1 SMP Mon Oct 6 22:52:46 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFC7EFE000", "elfType" : 3 }, { "b" : "7F7A6249B000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7F7A62293000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7F7A6208F000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7F7A61D8B000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F7A61A8D000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7F7A61877000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F7A614D2000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7F7A626B7000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf3a9d9]
       mongod(_ZN5mongo10logContextEPKc+0xE1) [0xee4b21]
       mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0xDA) [0xecaafa]
       mongod(_ZN5mongo21WiredTigerRecordStore27cappedDeleteAsNeeded_inlockEPNS_16OperationContextERKNS_8RecordIdE+0x78A) [0xd563fa]
       mongod(+0x95B4A4) [0xd5b4a4]
       mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0x120) [0xecbbc0]
       mongod(+0xB88114) [0xf88114]
       libpthread.so.0(+0x7F18) [0x7f7a624a2f18]
       libc.so.6(clone+0x6D) [0x7f7a615b4b9d]
      -----  END BACKTRACE  -----
      2015-01-31T05:26:28.422+0000 I -        [WiredTigerRecordStoreThread for local.oplog.rs] 
       
      ***aborting after invariant() failure

      1. commandline
        0.1 kB
        Mark Callaghan
      2. jmongoiibench.java
        35 kB
        Mark Callaghan
      3. mongo.conf
        0.5 kB
        Mark Callaghan
      4. run.2.simple.bash
        4 kB
        Michael Grundy
      5. run.simple.bash
        4 kB
        Mark Callaghan

        Issue Links

          Activity

          Hide
          michael.grundy Michael Grundy (Inactive) added a comment -

          I'm able to reproduce it now, takes about 1800 seconds longer. BT from rc8 below.

          2015-02-12T20:23:00.212+0000 I -        [WiredTigerRecordStoreThread for local.oplog.$main] Invariant failure: session->truncate(session, NULL, start, c, NULL) resulted in status UnknownError 2: No such file or directory at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 512
          2015-02-12T20:23:00.218+0000 I CONTROL  [WiredTigerRecordStoreThread for local.oplog.$main] 
           0xf3f059 0xee9191 0xecf16a 0xd59c9a 0xd5f4b9 0xed0230 0xf8c794 0x7f6aefb1af18 0x7f6aeec2cb2d
          ----- BEGIN BACKTRACE -----
          {"backtrace":[{"b":"400000","o":"B3F059"},{"b":"400000","o":"AE9191"},{"b":"400000","o":"ACF16A"},{"b":"400000","o":"959C9A"},{"b":"400000","o":"95F4B9"},{"b":"400000","o":"AD0230"},{"b":"400000","o":"B8C794"},{"b":"7F6AEFB13000","o":"7F18"},{"b":"7F6AEEB4A000","o":"E2B2D"}],"processInfo":{ "mongodbVersion" : "3.0.0-rc8", "gitVersion" : "9d0714cdcffadb203ff68730acedbc40875403da", "uname" : { "sysname" : "Linux", "release" : "3.14.27-25.47.amzn1.x86_64", "version" : "#1 SMP Wed Dec 17 18:36:15 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFF316FE000", "elfType" : 3 }, { "b" : "7F6AEFB13000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7F6AEF90B000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7F6AEF707000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7F6AEF403000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F6AEF105000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7F6AEEEEF000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F6AEEB4A000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7F6AEFD2F000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
           mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf3f059]
           mongod(_ZN5mongo10logContextEPKc+0xE1) [0xee9191]
           mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0xDA) [0xecf16a]
           mongod(_ZN5mongo21WiredTigerRecordStore27cappedDeleteAsNeeded_inlockEPNS_16OperationContextERKNS_8RecordIdE+0x78A) [0xd59c9a]
           mongod(+0x95F4B9) [0xd5f4b9]
           mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0x120) [0xed0230]
           mongod(+0xB8C794) [0xf8c794]
           libpthread.so.0(+0x7F18) [0x7f6aefb1af18]
           libc.so.6(clone+0x6D) [0x7f6aeec2cb2d]
          -----  END BACKTRACE  -----
          2015-02-12T20:23:00.218+0000 I -        [WiredTigerRecordStoreThread for local.oplog.$main] 
           
          ***aborting after invariant() failure

          Show
          michael.grundy Michael Grundy (Inactive) added a comment - I'm able to reproduce it now, takes about 1800 seconds longer. BT from rc8 below. 2015-02-12T20:23:00.212+0000 I - [WiredTigerRecordStoreThread for local.oplog.$main] Invariant failure: session->truncate(session, NULL, start, c, NULL) resulted in status UnknownError 2: No such file or directory at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 512 2015-02-12T20:23:00.218+0000 I CONTROL [WiredTigerRecordStoreThread for local.oplog.$main] 0xf3f059 0xee9191 0xecf16a 0xd59c9a 0xd5f4b9 0xed0230 0xf8c794 0x7f6aefb1af18 0x7f6aeec2cb2d ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"B3F059"},{"b":"400000","o":"AE9191"},{"b":"400000","o":"ACF16A"},{"b":"400000","o":"959C9A"},{"b":"400000","o":"95F4B9"},{"b":"400000","o":"AD0230"},{"b":"400000","o":"B8C794"},{"b":"7F6AEFB13000","o":"7F18"},{"b":"7F6AEEB4A000","o":"E2B2D"}],"processInfo":{ "mongodbVersion" : "3.0.0-rc8", "gitVersion" : "9d0714cdcffadb203ff68730acedbc40875403da", "uname" : { "sysname" : "Linux", "release" : "3.14.27-25.47.amzn1.x86_64", "version" : "#1 SMP Wed Dec 17 18:36:15 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFF316FE000", "elfType" : 3 }, { "b" : "7F6AEFB13000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7F6AEF90B000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7F6AEF707000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7F6AEF403000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F6AEF105000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7F6AEEEEF000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F6AEEB4A000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7F6AEFD2F000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf3f059] mongod(_ZN5mongo10logContextEPKc+0xE1) [0xee9191] mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0xDA) [0xecf16a] mongod(_ZN5mongo21WiredTigerRecordStore27cappedDeleteAsNeeded_inlockEPNS_16OperationContextERKNS_8RecordIdE+0x78A) [0xd59c9a] mongod(+0x95F4B9) [0xd5f4b9] mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0x120) [0xed0230] mongod(+0xB8C794) [0xf8c794] libpthread.so.0(+0x7F18) [0x7f6aefb1af18] libc.so.6(clone+0x6D) [0x7f6aeec2cb2d] ----- END BACKTRACE ----- 2015-02-12T20:23:00.218+0000 I - [WiredTigerRecordStoreThread for local.oplog.$main]   ***aborting after invariant() failure
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

          Message: SERVER-17141 Don't abort if ENOENT is returned from session->truncate()
          Branch: master
          https://github.com/mongodb/mongo/commit/d47529bb1ad51e65f5262ace4efcaff96062e028

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'} Message: SERVER-17141 Don't abort if ENOENT is returned from session->truncate() Branch: master https://github.com/mongodb/mongo/commit/d47529bb1ad51e65f5262ace4efcaff96062e028
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

          Message: SERVER-17141 Don't abort if ENOENT is returned from session->truncate()

          (cherry picked from commit d47529bb1ad51e65f5262ace4efcaff96062e028)
          Branch: v3.0
          https://github.com/mongodb/mongo/commit/5189c2d2e386c28b56ce8ae6ba630d98abee25bb

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'} Message: SERVER-17141 Don't abort if ENOENT is returned from session->truncate() (cherry picked from commit d47529bb1ad51e65f5262ace4efcaff96062e028) Branch: v3.0 https://github.com/mongodb/mongo/commit/5189c2d2e386c28b56ce8ae6ba630d98abee25bb
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'monkey101', u'name': u'Dan Pasette', u'email': u'dan@10gen.com'}

          Message: SERVER-17141 Cope with WT_NOTFOUND from WT_SESSION::truncate
          Branch: master
          https://github.com/mongodb/mongo/commit/4ca4224dae91ec067a38cd552b3b8c50999d7707

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'monkey101', u'name': u'Dan Pasette', u'email': u'dan@10gen.com'} Message: SERVER-17141 Cope with WT_NOTFOUND from WT_SESSION::truncate Branch: master https://github.com/mongodb/mongo/commit/4ca4224dae91ec067a38cd552b3b8c50999d7707
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'monkey101', u'name': u'Dan Pasette', u'email': u'dan@10gen.com'}

          Message: SERVER-17141 Cope with WT_NOTFOUND from WT_SESSION::truncate

          (cherry picked from commit 4ca4224dae91ec067a38cd552b3b8c50999d7707)
          Branch: v3.0
          https://github.com/mongodb/mongo/commit/74aabd913f3cafb2fd3adbafb2e7ec3d5aa8597e

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'monkey101', u'name': u'Dan Pasette', u'email': u'dan@10gen.com'} Message: SERVER-17141 Cope with WT_NOTFOUND from WT_SESSION::truncate (cherry picked from commit 4ca4224dae91ec067a38cd552b3b8c50999d7707) Branch: v3.0 https://github.com/mongodb/mongo/commit/74aabd913f3cafb2fd3adbafb2e7ec3d5aa8597e

            People

            • Votes:
              1 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: