Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16967

WT_WITH_TABLE_LOCK assert failure during compact

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-rc7
    • Component/s: Storage
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL

      Description

      In a debug build, we are now turning on HAVE_DIAGNOSTIC, which activates WT_ASSERT checking. One such check failed during the parallel test suite, where we were running the Compact command at the same time as some other operations:

      http://buildlogger-test-2014121502-1.10gen-mci.4085.mongodbdns.com:8080/build/54bfbcac7fe2dd09ac008055/test/54bfbeed7fe2dd09ac0087f9

      2015-01-21T10:02:20.092-0500 E STORAGE  [conn82] WiredTiger (0) [1421852540:92613][30694:0x48592940], session.checkpoint: src/third_party/wiredtiger/src/schema/schema_list.c, 28: F_ISSET(session, WT_SESSION_TABLE_LOCKED) || !F_ISSET(session, WT_SESSION_HANDLE_LIST_LOCKED
      2015-01-21T10:02:20.092-0500 E STORAGE  [conn82] WiredTiger (0) [1421852540:92713][30694:0x48592940], session.checkpoint: aborting WiredTiger library
      2015-01-21T10:02:20.106-0500 F -        [conn82] Got signal: 6 (Aborted).
       0xf5bb99 0xf5b242 0xf5b5c6 0x7fda488122d0 0x7fda48812265 0x7fda48813d10 0x15423df 0x157638b 0x1563e74 0x1570c9e 0x157e405 0x157f2fd 0x15728a6 0x1574712 0xd79157 0x91878d 0x94faf9 0x9dfdb4 0x9e0cf3 0x9e17ab 0xbb1b57 0xac1ab8 0x80ab32 0xf19991 0x7fda496dd73d 0x7fda488b5d1d
      ----- BEGIN BACKTRACE -----
       mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf5bb99]
       mongod(+0xB5B242) [0xf5b242]
       mongod(+0xB5B5C6) [0xf5b5c6]
       libc.so.6(+0x302D0) [0x7fda488122d0]
       libc.so.6(gsignal+0x35) [0x7fda48812265]
       libc.so.6(abort+0x110) [0x7fda48813d10]
       mongod(+0x11423DF) [0x15423df]
       mongod(+0x117638B) [0x157638b]
       mongod(__wt_schema_get_table+0x244) [0x1563e74]
       mongod(__wt_schema_worker+0x3AE) [0x1570c9e]
       mongod(+0x117E405) [0x157e405]
       mongod(__wt_txn_checkpoint+0x29D) [0x157f2fd]
       mongod(+0x11728A6) [0x15728a6]
       mongod(__wt_session_compact+0x3E2) [0x1574712]
       mongod(_ZN5mongo21WiredTigerRecordStore7compactEPNS_16OperationContextEPNS_25RecordStoreCompactAdaptorEPKNS_14CompactOptionsEPNS_12CompactStatsE+0x37) [0xd79157]
       mongod(_ZN5mongo10Collection7compactEPNS_16OperationContextEPKNS_14CompactOptionsE+0x22D) [0x91878d]
       mongod(_ZN5mongo10CompactCmd3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x7B9) [0x94faf9]
       mongod(_ZN5mongo12_execCommandEPNS_16OperationContextEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x34) [0x9dfdb4]
       mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_iPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xC13) [0x9e0cf3]
       mongod(_ZN5mongo12_runCommandsEPNS_16OperationContextEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x28B) [0x9e17ab]
       mongod(_ZN5mongo8runQueryEPNS_16OperationContextERNS_7MessageERNS_12QueryMessageERKNS_15NamespaceStringERNS_5CurOpES3_b+0x787) [0xbb1b57]
       mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortEb+0xAC8) [0xac1ab8]
       mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xE2) [0x80ab32]
       mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x321) [0xf19991]
       libpthread.so.0(+0x673D) [0x7fda496dd73d]
       libc.so.6(clone+0x6D) [0x7fda488b5d1d]
      		
      -----  END BACKTRACE  -----

      1. server16967.js
        0.5 kB
        Benety Goh

        Issue Links

          Activity

          Show
          milkie Eric Milkie added a comment - From this build: https://mci.10gen.com/task/mongodb_mongo_master_linux_64_debug_7d504b66098d1cfcafdb536ff0db3f7eb53f8c99_15_01_21_14_17_02_parallel_WT_linux_64_debug
          Hide
          milkie Eric Milkie added a comment -

          Another failure, from Windows debug this time (easier to read backtrace):
          http://buildlogs.mongodb.org/MCI_windows-64-2k8-debug/builds/601793/test/parallel_WT_0/basic.js

          2015-01-21T15:47:38.814+0000 E STORAGE  [conn30] WiredTiger (0) [1421855258:814400][3748:1999844256], session.checkpoint: src\third_party\wiredtiger\src\schema\schema_list.c, 28: F_ISSET(session, WT_SESSION_TABLE_LOCKED) || !F_ISSET(session, WT_SESSION_HANDLE_LIST_LOCKED)
          2015-01-21T15:47:38.814+0000 E STORAGE  [conn30] WiredTiger (0) [1421855258:814400][3748:1999844256], session.checkpoint: aborting WiredTiger library
          2015-01-21T15:47:39.351+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\util\stacktrace_win.cpp(175)                            mongo::printStackTrace+0x43
          2015-01-21T15:47:39.351+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\util\signal_handlers_synchronous.cpp(127)               mongo::`anonymous namespace'::printSignalAndBacktrace+0x73
          2015-01-21T15:47:39.351+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\util\signal_handlers_synchronous.cpp(194)               mongo::`anonymous namespace'::abruptQuit+0x2e
          2015-01-21T15:47:39.351+0000 I CONTROL  [conn30] mongod.exe    f:\dd\vctools\crt\crtw32\misc\winsig.c(594)                           raise+0x35f
          2015-01-21T15:47:39.351+0000 I CONTROL  [conn30] mongod.exe    f:\dd\vctools\crt\crtw32\misc\abort.c(82)                             abort+0x40
          2015-01-21T15:47:39.351+0000 I CONTROL  [conn30] mongod.exe    ...\src\third_party\wiredtiger\src\os_posix\os_abort.c(25)            __wt_abort+0x22
          2015-01-21T15:47:39.351+0000 I CONTROL  [conn30] mongod.exe    ...\src\third_party\wiredtiger\src\support\err.c(451)                 __wt_assert+0x37
          2015-01-21T15:47:39.352+0000 I CONTROL  [conn30] mongod.exe    ...\src\third_party\wiredtiger\src\schema\schema_list.c(28)           __schema_add_table+0x6c
          2015-01-21T15:47:39.352+0000 I CONTROL  [conn30] mongod.exe    ...\src\third_party\wiredtiger\src\schema\schema_list.c(99)           __wt_schema_get_table+0x64
          2015-01-21T15:47:39.352+0000 I CONTROL  [conn30] mongod.exe    ...\src\third_party\wiredtiger\src\schema\schema_worker.c(83)         __wt_schema_worker+0x2ee
          2015-01-21T15:47:39.352+0000 I CONTROL  [conn30] mongod.exe    ...\src\third_party\wiredtiger\src\txn\txn_ckpt.c(132)                __checkpoint_apply_all+0x1d1
          2015-01-21T15:47:39.352+0000 I CONTROL  [conn30] mongod.exe    ...\src\third_party\wiredtiger\src\txn\txn_ckpt.c(346)                __wt_txn_checkpoint+0x1ca
          2015-01-21T15:47:39.352+0000 I CONTROL  [conn30] mongod.exe    ...\src\third_party\wiredtiger\src\session\session_api.c(889)         __session_checkpoint+0x1b3
          2015-01-21T15:47:39.352+0000 I CONTROL  [conn30] mongod.exe    ...\src\third_party\wiredtiger\src\session\session_compact.c(185)     __compact_file+0x101
          2015-01-21T15:47:39.352+0000 I CONTROL  [conn30] mongod.exe    ...\src\third_party\wiredtiger\src\session\session_compact.c(235)     __wt_session_compact+0x250
          2015-01-21T15:47:39.353+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\db\storage\wiredtiger\wiredtiger_record_store.cpp(654)  mongo::WiredTigerRecordStore::compact+0x62
          2015-01-21T15:47:39.353+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\db\catalog\collection_compact.cpp(118)                  mongo::Collection::compact+0x1cf
          2015-01-21T15:47:39.353+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\db\commands\compact.cpp(168)                            mongo::CompactCmd::run+0x617
          2015-01-21T15:47:39.353+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\db\dbcommands.cpp(1267)                                 mongo::_execCommand+0x72
          2015-01-21T15:47:39.353+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\db\dbcommands.cpp(1485)                                 mongo::Command::execCommand+0xf01
          2015-01-21T15:47:39.353+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\db\dbcommands.cpp(1560)                                 mongo::_runCommands+0x413
          2015-01-21T15:47:39.353+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\db\query\find.cpp(137)                                  mongo::runCommands+0x43
          2015-01-21T15:47:39.353+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\db\query\find.cpp(606)                                  mongo::runQuery+0x3c1
          2015-01-21T15:47:39.354+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\db\instance.cpp(220)                                    mongo::receivedQuery+0x2ec
          2015-01-21T15:47:39.354+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\db\instance.cpp(403)                                    mongo::assembleResponse+0x36c
          2015-01-21T15:47:39.354+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\db\db.cpp(206)                                          mongo::MyMessageHandler::process+0xb6
          2015-01-21T15:47:39.354+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\util\net\message_server_port.cpp(229)                   mongo::PortMessageServer::handleIncomingMsg+0x670
          2015-01-21T15:47:39.354+0000 I CONTROL  [conn30] mongod.exe    ...\src\third_party\boost\libs\thread\src\win32\thread.cpp(185)       boost::`anonymous namespace'::thread_start_function+0x21
          2015-01-21T15:47:39.354+0000 I CONTROL  [conn30] mongod.exe    f:\dd\vctools\crt\crtw32\startup\threadex.c(376)                      _callthreadstartex+0x25
          2015-01-21T15:47:39.354+0000 I CONTROL  [conn30] mongod.exe    f:\dd\vctools\crt\crtw32\startup\threadex.c(359)                      _threadstartex+0xe7
          2015-01-21T15:47:39.354+0000 I CONTROL  [conn30] kernel32.dll                                                                        BaseThreadInitThunk+0xd
          2015-01-21T15:47:39.354+0000 F -        [conn30] Got signal: 22 (SIGABRT).

          Show
          milkie Eric Milkie added a comment - Another failure, from Windows debug this time (easier to read backtrace): http://buildlogs.mongodb.org/MCI_windows-64-2k8-debug/builds/601793/test/parallel_WT_0/basic.js 2015-01-21T15:47:38.814+0000 E STORAGE [conn30] WiredTiger (0) [1421855258:814400][3748:1999844256], session.checkpoint: src\third_party\wiredtiger\src\schema\schema_list.c, 28: F_ISSET(session, WT_SESSION_TABLE_LOCKED) || !F_ISSET(session, WT_SESSION_HANDLE_LIST_LOCKED) 2015-01-21T15:47:38.814+0000 E STORAGE [conn30] WiredTiger (0) [1421855258:814400][3748:1999844256], session.checkpoint: aborting WiredTiger library 2015-01-21T15:47:39.351+0000 I CONTROL [conn30] mongod.exe ...\src\mongo\util\stacktrace_win.cpp(175) mongo::printStackTrace+0x43 2015-01-21T15:47:39.351+0000 I CONTROL [conn30] mongod.exe ...\src\mongo\util\signal_handlers_synchronous.cpp(127) mongo::`anonymous namespace'::printSignalAndBacktrace+0x73 2015-01-21T15:47:39.351+0000 I CONTROL [conn30] mongod.exe ...\src\mongo\util\signal_handlers_synchronous.cpp(194) mongo::`anonymous namespace'::abruptQuit+0x2e 2015-01-21T15:47:39.351+0000 I CONTROL [conn30] mongod.exe f:\dd\vctools\crt\crtw32\misc\winsig.c(594) raise+0x35f 2015-01-21T15:47:39.351+0000 I CONTROL [conn30] mongod.exe f:\dd\vctools\crt\crtw32\misc\abort.c(82) abort+0x40 2015-01-21T15:47:39.351+0000 I CONTROL [conn30] mongod.exe ...\src\third_party\wiredtiger\src\os_posix\os_abort.c(25) __wt_abort+0x22 2015-01-21T15:47:39.351+0000 I CONTROL [conn30] mongod.exe ...\src\third_party\wiredtiger\src\support\err.c(451) __wt_assert+0x37 2015-01-21T15:47:39.352+0000 I CONTROL [conn30] mongod.exe ...\src\third_party\wiredtiger\src\schema\schema_list.c(28) __schema_add_table+0x6c 2015-01-21T15:47:39.352+0000 I CONTROL [conn30] mongod.exe ...\src\third_party\wiredtiger\src\schema\schema_list.c(99) __wt_schema_get_table+0x64 2015-01-21T15:47:39.352+0000 I CONTROL [conn30] mongod.exe ...\src\third_party\wiredtiger\src\schema\schema_worker.c(83) __wt_schema_worker+0x2ee 2015-01-21T15:47:39.352+0000 I CONTROL [conn30] mongod.exe ...\src\third_party\wiredtiger\src\txn\txn_ckpt.c(132) __checkpoint_apply_all+0x1d1 2015-01-21T15:47:39.352+0000 I CONTROL [conn30] mongod.exe ...\src\third_party\wiredtiger\src\txn\txn_ckpt.c(346) __wt_txn_checkpoint+0x1ca 2015-01-21T15:47:39.352+0000 I CONTROL [conn30] mongod.exe ...\src\third_party\wiredtiger\src\session\session_api.c(889) __session_checkpoint+0x1b3 2015-01-21T15:47:39.352+0000 I CONTROL [conn30] mongod.exe ...\src\third_party\wiredtiger\src\session\session_compact.c(185) __compact_file+0x101 2015-01-21T15:47:39.352+0000 I CONTROL [conn30] mongod.exe ...\src\third_party\wiredtiger\src\session\session_compact.c(235) __wt_session_compact+0x250 2015-01-21T15:47:39.353+0000 I CONTROL [conn30] mongod.exe ...\src\mongo\db\storage\wiredtiger\wiredtiger_record_store.cpp(654) mongo::WiredTigerRecordStore::compact+0x62 2015-01-21T15:47:39.353+0000 I CONTROL [conn30] mongod.exe ...\src\mongo\db\catalog\collection_compact.cpp(118) mongo::Collection::compact+0x1cf 2015-01-21T15:47:39.353+0000 I CONTROL [conn30] mongod.exe ...\src\mongo\db\commands\compact.cpp(168) mongo::CompactCmd::run+0x617 2015-01-21T15:47:39.353+0000 I CONTROL [conn30] mongod.exe ...\src\mongo\db\dbcommands.cpp(1267) mongo::_execCommand+0x72 2015-01-21T15:47:39.353+0000 I CONTROL [conn30] mongod.exe ...\src\mongo\db\dbcommands.cpp(1485) mongo::Command::execCommand+0xf01 2015-01-21T15:47:39.353+0000 I CONTROL [conn30] mongod.exe ...\src\mongo\db\dbcommands.cpp(1560) mongo::_runCommands+0x413 2015-01-21T15:47:39.353+0000 I CONTROL [conn30] mongod.exe ...\src\mongo\db\query\find.cpp(137) mongo::runCommands+0x43 2015-01-21T15:47:39.353+0000 I CONTROL [conn30] mongod.exe ...\src\mongo\db\query\find.cpp(606) mongo::runQuery+0x3c1 2015-01-21T15:47:39.354+0000 I CONTROL [conn30] mongod.exe ...\src\mongo\db\instance.cpp(220) mongo::receivedQuery+0x2ec 2015-01-21T15:47:39.354+0000 I CONTROL [conn30] mongod.exe ...\src\mongo\db\instance.cpp(403) mongo::assembleResponse+0x36c 2015-01-21T15:47:39.354+0000 I CONTROL [conn30] mongod.exe ...\src\mongo\db\db.cpp(206) mongo::MyMessageHandler::process+0xb6 2015-01-21T15:47:39.354+0000 I CONTROL [conn30] mongod.exe ...\src\mongo\util\net\message_server_port.cpp(229) mongo::PortMessageServer::handleIncomingMsg+0x670 2015-01-21T15:47:39.354+0000 I CONTROL [conn30] mongod.exe ...\src\third_party\boost\libs\thread\src\win32\thread.cpp(185) boost::`anonymous namespace'::thread_start_function+0x21 2015-01-21T15:47:39.354+0000 I CONTROL [conn30] mongod.exe f:\dd\vctools\crt\crtw32\startup\threadex.c(376) _callthreadstartex+0x25 2015-01-21T15:47:39.354+0000 I CONTROL [conn30] mongod.exe f:\dd\vctools\crt\crtw32\startup\threadex.c(359) _threadstartex+0xe7 2015-01-21T15:47:39.354+0000 I CONTROL [conn30] kernel32.dll BaseThreadInitThunk+0xd 2015-01-21T15:47:39.354+0000 F - [conn30] Got signal: 22 (SIGABRT).
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

          Message: SERVER-16967 extend collection compact test to drop collection in the background
          Branch: master
          https://github.com/mongodb/mongo/commit/373b844920028ac35815f99172ac95684e5e2be8

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'} Message: SERVER-16967 extend collection compact test to drop collection in the background Branch: master https://github.com/mongodb/mongo/commit/373b844920028ac35815f99172ac95684e5e2be8
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

          Message: SERVER-16967 extend collection compact test to drop collection in the background

          (cherry picked from commit 373b844920028ac35815f99172ac95684e5e2be8)
          Branch: v3.0
          https://github.com/mongodb/mongo/commit/90bbd525a01e710ec843bacebf82a62928c7458f

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'} Message: SERVER-16967 extend collection compact test to drop collection in the background (cherry picked from commit 373b844920028ac35815f99172ac95684e5e2be8) Branch: v3.0 https://github.com/mongodb/mongo/commit/90bbd525a01e710ec843bacebf82a62928c7458f

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: