Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-3991

Crash on close with LSM and stable timestamp

    • Type: Icon: Bug Bug
    • Resolution: Won't Fix
    • Priority: Icon: Minor - P4 Minor - P4
    • None
    • Affects Version/s: None
    • Component/s: LSM
    • 13

      MongoDB master is now running with a stable_timestamp set (as of SERVER-29213).

      That has cause fallout in LSM tests: https://evergreen.mongodb.com/build/mongodb_mongo_master_linux_64_lsm_6ae04cd9f250fac877df94ecd4ddad33eaf5bc77_18_03_24_23_39_13

      In particular, mongod is crashing during clean shutdown with a stack like this:

      [js_test:address] 2018-03-25T03:00:02.648+0000 c20514| 2018-03-25T03:00:02.647+0000 F -        [signalProcessingThread] Invalid access at address: 0x7f6e2846d010
      [js_test:address] 2018-03-25T03:00:02.661+0000 c20514| 2018-03-25T03:00:02.661+0000 F -        [signalProcessingThread] Got signal: 11 (Segmentation fault).
      [js_test:address] 2018-03-25T03:00:02.662+0000 c20514|  0x7f6e2a7e61b1 0x7f6e2a7e53c9 0x7f6e2a7e5a36 0x7f6e2769d7e0 0x7f6e2909f2f5 0x7f6e2910b05b 0x7f6e290bcc50 0x7f6e29021394 0x7f6e29021e47 0x7f6e29021f84 0x7f6e29022ee8 0x7f6e2901c542 0x7f6e28fe944a 0x7f6e28fb9b31 0x7f6e28f85641 0x7f6e2a7e17a2 0x7f6e28f24d17 0x7f6e296207fa 0x7f6e2a8f6200 0x7f6e27695aa1 0x7f6e273e2bcd
      [js_test:address] 2018-03-25T03:00:02.662+0000 c20514| ----- BEGIN BACKTRACE -----
      [js_test:address] 2018-03-25T03:00:02.664+0000 c20514|  mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x7f6e2a7e61b1]
      [js_test:address] 2018-03-25T03:00:02.664+0000 c20514|  mongod(+0x22573C9) [0x7f6e2a7e53c9]
      [js_test:address] 2018-03-25T03:00:02.664+0000 c20514|  mongod(+0x2257A36) [0x7f6e2a7e5a36]
      [js_test:address] 2018-03-25T03:00:02.664+0000 c20514|  libpthread.so.0(+0xF7E0) [0x7f6e2769d7e0]
      [js_test:address] 2018-03-25T03:00:02.664+0000 c20514|  mongod(__wt_page_out+0x465) [0x7f6e2909f2f5]
      [js_test:address] 2018-03-25T03:00:02.664+0000 c20514|  mongod(__wt_evict_file+0x24B) [0x7f6e2910b05b]
      [js_test:address] 2018-03-25T03:00:02.664+0000 c20514|  mongod(__wt_cache_op+0x100) [0x7f6e290bcc50]
      [js_test:address] 2018-03-25T03:00:02.665+0000 c20514|  mongod(__wt_conn_dhandle_close+0x284) [0x7f6e29021394]
      [js_test:address] 2018-03-25T03:00:02.665+0000 c20514|  mongod(__wt_conn_dhandle_discard_single+0x197) [0x7f6e29021e47]
      [js_test:address] 2018-03-25T03:00:02.665+0000 c20514|  mongod(__wt_conn_dhandle_discard+0x84) [0x7f6e29021f84]
      [js_test:address] 2018-03-25T03:00:02.665+0000 c20514|  mongod(__wt_connection_close+0x1D8) [0x7f6e29022ee8]
      [js_test:address] 2018-03-25T03:00:02.665+0000 c20514|  mongod(+0xA8E542) [0x7f6e2901c542]
      [js_test:address] 2018-03-25T03:00:02.665+0000 c20514|  mongod(_ZN5mongo18WiredTigerKVEngine13cleanShutdownEv+0x23A) [0x7f6e28fe944a]
      [js_test:address] 2018-03-25T03:00:02.666+0000 c20514|  mongod(_ZN5mongo20ServiceContextMongoD34shutdownGlobalStorageEngineCleanlyEv+0x21) [0x7f6e28fb9b31]
      [js_test:address] 2018-03-25T03:00:02.666+0000 c20514|  mongod(+0x9F7641) [0x7f6e28f85641]
      [js_test:address] 2018-03-25T03:00:02.666+0000 c20514|  mongod(+0x22537A2) [0x7f6e2a7e17a2]
      [js_test:address] 2018-03-25T03:00:02.666+0000 c20514|  mongod(_ZN5mongo8shutdownENS_8ExitCodeE+0x35E) [0x7f6e28f24d17]
      [js_test:address] 2018-03-25T03:00:02.666+0000 c20514|  mongod(+0x10927FA) [0x7f6e296207fa]
      [js_test:address] 2018-03-25T03:00:02.666+0000 c20514|  mongod(+0x2368200) [0x7f6e2a8f6200]
      [js_test:address] 2018-03-25T03:00:02.666+0000 c20514|  libpthread.so.0(+0x7AA1) [0x7f6e27695aa1]
      [js_test:address] 2018-03-25T03:00:02.667+0000 c20514|  libc.so.6(clone+0x6D) [0x7f6e273e2bcd]
      [js_test:address] 2018-03-25T03:00:02.667+0000 c20514| -----  END BACKTRACE  -----
      

      Investigate (1) whether setting mmap=false avoids this crash, then (2) what is causing it.

            Assignee:
            backlog-server-storage-engines [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            michael.cahill@mongodb.com Michael Cahill (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: