Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-21619

sys-perf: WT crash during core_workloads_WT execution

XMLWordPrintableJSON

    • Fully Compatible
    • ALL

      There were some random crashes of mongod during core_workloads_WT test in system-perf. observations:

      • this happens in all non-sharded setups (standalone, 1-node replSet, 3-node replSet) randomly
      • Seems that it always happens during insert_ttl test
      • There was no core file generated, I am trying to figure out why. I did apply "ulimit -c unlimited" during start up mongod.
      • Manual run of insert_ttl does not re-create the crash yet, maybe need to run the whole suite?
      • The earliest SHA with this issue is 3f598f1edc (test report)

      Stack trace from the mongod log file,

       [2015/11/21 17:50:03.017] 2015-11-21T22:46:58.450+0000 I NETWORK  [conn437] end connection 10.2.0.98:51172 (7 connections now open)
 [2015/11/21 17:50:03.017] 2015-11-21T22:46:59.372+0000 F -        [thread1] Invalid access at address: 0xc8
 [2015/11/21 17:50:03.017] 2015-11-21T22:46:59.379+0000 F -        [thread1] Got signal: 11 (Segmentation fault).
 [2015/11/21 17:50:03.017]  0x12c99f2 0x12c8929 0x12c8ca8 0x7ff85a737130 0x19f9699 0x19f9937 0x19fce8f 0x19c91ad 0x19c68a4 0x19c6b5a 0x7ff85a72fdf3 0x7ff85a45d1ad
 [2015/11/21 17:50:03.017] ----- BEGIN BACKTRACE -----
 [2015/11/21 17:50:03.017] {"backtrace":[{"b":"400000","o":"EC99F2"},{"b":"400000","o":"EC8929"},{"b":"400000","o":"EC8CA8"},{"b":"7FF85A728000","o":"F130"},{"b":"400000","o":"15F9699"},{"b":"400000","o":"15F9937"},{"b":"400000","o":"15FCE8F"},{"b":"400000","o":"15C91AD"},{"b":"400000","o":"15C68A4"},{"b":"400000","o":"15C6B5A"},{"b":"7FF85A728000","o":"7DF3"},{"b":"7FF85A367000","o":"F61AD"}],"processInfo":{ "mongodbVersion" : "3.2.0-rc3-95-g6f2a7e6", "gitVersion" : "6f2a7e6cfb69e186ee2d5ca8653dda5bf0633ef7", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.14.35-28.38.amzn1.x86_64", "version" : "#1 SMP Wed Mar 11 22:50:37 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFC129D4000", "elfType" : 3 }, { "b" : "7FF85B364000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7FF85B160000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7FF85AE5C000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FF85AB5A000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7FF85A944000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FF85A728000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7FF85A367000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7FF85B56C000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
 [2015/11/21 17:50:03.017]  mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x12c99f2]
 [2015/11/21 17:50:03.017]  mongod(+0xEC8929) [0x12c8929]
 [2015/11/21 17:50:03.017]  mongod(+0xEC8CA8) [0x12c8ca8]
 [2015/11/21 17:50:03.017]  libpthread.so.0(+0xF130) [0x7ff85a737130]
 [2015/11/21 17:50:03.017]  mongod(+0x15F9699) [0x19f9699]
 [2015/11/21 17:50:03.017]  mongod(+0x15F9937) [0x19f9937]
 [2015/11/21 17:50:03.017]  mongod(__wt_reconcile+0x27F) [0x19fce8f]
 [2015/11/21 17:50:03.018]  mongod(__wt_evict+0x28D) [0x19c91ad]
 [2015/11/21 17:50:03.018]  mongod(+0x15C68A4) [0x19c68a4]
 [2015/11/21 17:50:03.018]  mongod(+0x15C6B5A) [0x19c6b5a]
 [2015/11/21 17:50:03.018]  libpthread.so.0(+0x7DF3) [0x7ff85a72fdf3]
 [2015/11/21 17:50:03.018]  libc.so.6(clone+0x6D) [0x7ff85a45d1ad]
 [2015/11/21 17:50:03.018] -----  END BACKTRACE  -----

      decode

      [ec2-user@ip-10-2-0-98 t]$ addr2line -e ./mongodb/bin/mongod 0x12c99f2 0x12c8929 0x12c8ca8 0x7ff85a737130 0x19f9699 0x19f9937 0x19fce8f 0x19c91ad 0x19c68a4 0x19c6b5a 0x7ff85a72fdf3 0x7ff85a45d1ad
/srv/10gen/mci-exec/mci/src/src/mongo/util/stacktrace_posix.cpp:172
/srv/10gen/mci-exec/mci/src/src/mongo/util/signal_handlers_synchronous.cpp:180
/srv/10gen/mci-exec/mci/src/src/mongo/util/signal_handlers_synchronous.cpp:275
??:0
/srv/10gen/mci-exec/mci/src/src/third_party/wiredtiger/src/reconcile/rec_write.c:1980
/srv/10gen/mci-exec/mci/src/src/third_party/wiredtiger/src/reconcile/rec_write.c:4572
/srv/10gen/mci-exec/mci/src/src/third_party/wiredtiger/src/reconcile/rec_write.c:412
/srv/10gen/mci-exec/mci/src/src/third_party/wiredtiger/src/evict/evict_page.c:480
/srv/10gen/mci-exec/mci/src/src/third_party/wiredtiger/src/evict/evict_lru.c:1467
/srv/10gen/mci-exec/mci/src/src/third_party/wiredtiger/src/evict/evict_lru.c:818
??:0
??:0

      more details here
      https://evergreen.mongodb.com/task_log_raw/sys_perf_linux_1_node_replSet_core_workloads_WT_6f2a7e6cfb69e186ee2d5ca8653dda5bf0633ef7_15_11_20_23_29_14/0?type=T#L482 and link to mongod.tar.gz (https://s3.amazonaws.com/mciuploads/dsi/sys_perf_6f2a7e6cfb69e186ee2d5ca8653dda5bf0633ef7/6f2a7e6cfb69e186ee2d5ca8653dda5bf0633ef7/mongod-sys_perf_6f2a7e6cfb69e186ee2d5ca8653dda5bf0633ef7.tar.gz ) which was not stripped.

      and few other crashes:

            Assignee:
            alexander.gorrod@mongodb.com Alexander Gorrod
            Reporter:
            rui.zhang Rui Zhang (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: