Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17324

NULL pointer exception during insert only stress under low memory condition

    • Type: Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.0.0-rc8
    • Component/s: WiredTiger
    • Labels:
    • Fully Compatible
    • ALL

      v3.0.0-rc9-pre
      GitHub # - c7d4eac9eb895886086eb6f0b763147ec7f3b90

      Environment:

      • Single machine replica set with 2 nodes.
      • Windows 8 w/ limited amount of available memory (VM w/o page file, limited to 2GB RAM)
      • Each instance uses default 1GB RAM

      Workload:

      • Used hammer.mongo to do insert only workload
      • The machine complained about out of memory before crashing
      • When cache fills up, eventually hit a NPE. Location varies according to run.

      Stack trace:

      0:015> k
      Child-SP          RetAddr           Call Site
      00000051`8cb1e7f0 00007ff6`bc5a86e0 mongod!__rec_txn_read+0x70 [c:\data\mci\shell\src\src\third_party\wiredtiger\src\reconcile\rec_write.c @ 855]
      00000051`8cb1e890 00007ff6`bc5a852c mongod!__rec_row_leaf_insert+0x60 [c:\data\mci\shell\src\src\third_party\wiredtiger\src\reconcile\rec_write.c @ 4619]
      00000051`8cb1e8f0 00007ff6`bc5ad15c mongod!__rec_row_leaf+0x70c [c:\data\mci\shell\src\src\third_party\wiredtiger\src\reconcile\rec_write.c @ 4589]
      00000051`8cb1ea10 00007ff6`bc58c9de mongod!__wt_reconcile+0x19c [c:\data\mci\shell\src\src\third_party\wiredtiger\src\reconcile\rec_write.c @ 414]
      00000051`8cb1ea70 00007ff6`bc58cbfd mongod!__evict_review+0x21e [c:\data\mci\shell\src\src\third_party\wiredtiger\src\evict\evict_page.c @ 481]
      00000051`8cb1ead0 00007ff6`bc58c298 mongod!__wt_evict+0xcd [c:\data\mci\shell\src\src\third_party\wiredtiger\src\evict\evict_page.c @ 59]
      00000051`8cb1eb50 00007ff6`bc568a78 mongod!__wt_evict_page+0x38 [c:\data\mci\shell\src\src\third_party\wiredtiger\src\evict\evict_lru.c @ 630]
      00000051`8cb1eb80 00007ff6`bc55acb5 mongod!__wt_page_release_evict+0x98 [c:\data\mci\shell\src\src\third_party\wiredtiger\src\include\btree.i @ 972]
      00000051`8cb1ebc0 00007ff6`bc56a55a mongod!__wt_page_in_func+0x1d5 [c:\data\mci\shell\src\src\third_party\wiredtiger\src\btree\bt_page.c @ 144]
      00000051`8cb1ec10 00007ff6`bc56ac75 mongod!__wt_page_swap_func+0x2a [c:\data\mci\shell\src\src\third_party\wiredtiger\src\include\btree.i @ 1063]
      00000051`8cb1ec50 00007ff6`bc554044 mongod!__wt_row_search+0x405 [c:\data\mci\shell\src\src\third_party\wiredtiger\src\btree\row_srch.c @ 297]
      00000051`8cb1ed10 00007ff6`bc57c3fc mongod!__wt_btcur_insert+0x214 [c:\data\mci\shell\src\src\third_party\wiredtiger\src\btree\bt_cursor.c @ 484]
      00000051`8cb1ed70 00007ff6`bc1b9852 mongod!__curfile_insert+0x2ac [c:\data\mci\shell\src\src\third_party\wiredtiger\src\cursor\cur_file.c @ 245]
      00000051`8cb1ee10 00007ff6`bbe50bdb mongod!mongo::WiredTigerRecordStore::insertRecord+0x3b2 [c:\data\mci\shell\src\src\mongo\db\storage\wiredtiger\wiredtiger_record_store.cpp @ 579]
      00000051`8cb1ef60 00007ff6`bbe51a67 mongod!mongo::Collection::_insertDocument+0x6b [c:\data\mci\shell\src\src\mongo\db\catalog\collection.cpp @ 245]
      00000051`8cb1efe0 00007ff6`bc0d2bea mongod!mongo::Collection::insertDocument+0x1b7 [c:\data\mci\shell\src\src\mongo\db\catalog\collection.cpp @ 201]
      00000051`8cb1f0e0 00007ff6`bc11afe6 mongod!mongo::repl::writeOpsToOplog+0x3ca [c:\data\mci\shell\src\src\mongo\db\repl\oplog.cpp @ 445]
      00000051`8cb1f380 00007ff6`bc11b654 mongod!mongo::repl::SyncTail::multiApply+0x2b6 [c:\data\mci\shell\src\src\mongo\db\repl\sync_tail.cpp @ 300]
      00000051`8cb1f530 00007ff6`bc11579c mongod!mongo::repl::SyncTail::oplogApplication+0x3d4 [c:\data\mci\shell\src\src\mongo\db\repl\sync_tail.cpp @ 510]
      00000051`8cb1f6a0 00007ff6`bc307d81 mongod!mongo::repl::runSyncThread+0x37c [c:\data\mci\shell\src\src\mongo\db\repl\rs_sync.cpp @ 135]
      00000051`8cb1f900 00007ff6`bc60e207 mongod!boost::`anonymous namespace'::thread_start_function+0x21 [c:\data\mci\shell\src\src\third_party\boost\libs\thread\src\win32\thread.cpp @ 185]
      00000051`8cb1f940 00007ff6`bc60e3ae mongod!_callthreadstartex+0x17 [f:\dd\vctools\crt\crtw32\startup\threadex.c @ 376]
      *** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\Windows\system32\KERNEL32.DLL - 
      00000051`8cb1f970 00007fff`e37615bd mongod!_threadstartex+0x102 [f:\dd\vctools\crt\crtw32\startup\threadex.c @ 354]
      *** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\Windows\SYSTEM32\ntdll.dll - 
      00000051`8cb1f9a0 00007fff`e5b843d1 KERNEL32!BaseThreadInitThunk+0xd
      00000051`8cb1f9d0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
      
      0:015> .srcpath \\eitan6\f$\src\mongo\src
      

      Variables:

      > dump variables
              session = 0x00000051`8f992e10
                    r = 0x00007ff6`bc5bce2e
                  ins = 0x00000051`9912e030
                  rip = 0x00000000`00000000
                vpack = 0x00000000`00000000
                 updp = 0x00000051`8cb1e8f8
              notused = 0x00007ff6`bc5bce2e
             upd_ovfl = 0x00000051`8cb1e8f8
                 ovfl = struct __wt_item
      
      > dump pointer 0x00000051`8cb1e8f8
      00000051`8cb1e8f8  00000000`00000000 00000051`8f992e10
      00000051`8cb1e908  00000051`8cb1e9a9 00000051`8cb1ea10
      00000051`8cb1e918 
      

      Code at point of crash:

      for (max_txn = WT_TXN_NONE, min_txn = UINT64_MAX, upd = upd_list;
              upd != NULL; upd = upd->next) {  <== upd->next is NULL
              if ((txnid = upd->txnid) == WT_TXN_ABORTED)
                  continue;
      

      Windows dumpfile:

            Assignee:
            michael.cahill@mongodb.com Michael Cahill (Inactive)
            Reporter:
            eitan.klein Eitan Klein
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: