[SERVER-16967] WT_WITH_TABLE_LOCK assert failure during compact Created: 21/Jan/15  Updated: 18/Sep/15  Resolved: 27/Jan/15

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: 3.0.0-rc7

Type: Bug Priority: Major - P3
Reporter: Eric Milkie Assignee: Benety Goh
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File server16967.js    
Issue Links:
Duplicate
is duplicated by SERVER-16934 WT crashed at index building due to "... Closed
Related
related to SERVER-17336 fix core/compact_keeps_indexes.js in ... Closed
is related to SERVER-16676 Compact with WiredTiger does not rebu... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:
Linked BF Score: 0

 Description   

In a debug build, we are now turning on HAVE_DIAGNOSTIC, which activates WT_ASSERT checking. One such check failed during the parallel test suite, where we were running the Compact command at the same time as some other operations:

http://buildlogger-test-2014121502-1.10gen-mci.4085.mongodbdns.com:8080/build/54bfbcac7fe2dd09ac008055/test/54bfbeed7fe2dd09ac0087f9

2015-01-21T10:02:20.092-0500 E STORAGE  [conn82] WiredTiger (0) [1421852540:92613][30694:0x48592940], session.checkpoint: src/third_party/wiredtiger/src/schema/schema_list.c, 28: F_ISSET(session, WT_SESSION_TABLE_LOCKED) || !F_ISSET(session, WT_SESSION_HANDLE_LIST_LOCKED
2015-01-21T10:02:20.092-0500 E STORAGE  [conn82] WiredTiger (0) [1421852540:92713][30694:0x48592940], session.checkpoint: aborting WiredTiger library
2015-01-21T10:02:20.106-0500 F -        [conn82] Got signal: 6 (Aborted).
 0xf5bb99 0xf5b242 0xf5b5c6 0x7fda488122d0 0x7fda48812265 0x7fda48813d10 0x15423df 0x157638b 0x1563e74 0x1570c9e 0x157e405 0x157f2fd 0x15728a6 0x1574712 0xd79157 0x91878d 0x94faf9 0x9dfdb4 0x9e0cf3 0x9e17ab 0xbb1b57 0xac1ab8 0x80ab32 0xf19991 0x7fda496dd73d 0x7fda488b5d1d
----- BEGIN BACKTRACE -----
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf5bb99]
 mongod(+0xB5B242) [0xf5b242]
 mongod(+0xB5B5C6) [0xf5b5c6]
 libc.so.6(+0x302D0) [0x7fda488122d0]
 libc.so.6(gsignal+0x35) [0x7fda48812265]
 libc.so.6(abort+0x110) [0x7fda48813d10]
 mongod(+0x11423DF) [0x15423df]
 mongod(+0x117638B) [0x157638b]
 mongod(__wt_schema_get_table+0x244) [0x1563e74]
 mongod(__wt_schema_worker+0x3AE) [0x1570c9e]
 mongod(+0x117E405) [0x157e405]
 mongod(__wt_txn_checkpoint+0x29D) [0x157f2fd]
 mongod(+0x11728A6) [0x15728a6]
 mongod(__wt_session_compact+0x3E2) [0x1574712]
 mongod(_ZN5mongo21WiredTigerRecordStore7compactEPNS_16OperationContextEPNS_25RecordStoreCompactAdaptorEPKNS_14CompactOptionsEPNS_12CompactStatsE+0x37) [0xd79157]
 mongod(_ZN5mongo10Collection7compactEPNS_16OperationContextEPKNS_14CompactOptionsE+0x22D) [0x91878d]
 mongod(_ZN5mongo10CompactCmd3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x7B9) [0x94faf9]
 mongod(_ZN5mongo12_execCommandEPNS_16OperationContextEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x34) [0x9dfdb4]
 mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_iPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xC13) [0x9e0cf3]
 mongod(_ZN5mongo12_runCommandsEPNS_16OperationContextEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x28B) [0x9e17ab]
 mongod(_ZN5mongo8runQueryEPNS_16OperationContextERNS_7MessageERNS_12QueryMessageERKNS_15NamespaceStringERNS_5CurOpES3_b+0x787) [0xbb1b57]
 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortEb+0xAC8) [0xac1ab8]
 mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xE2) [0x80ab32]
 mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x321) [0xf19991]
 libpthread.so.0(+0x673D) [0x7fda496dd73d]
 libc.so.6(clone+0x6D) [0x7fda488b5d1d]
		
-----  END BACKTRACE  -----



 Comments   
Comment by Githook User [ 27/Jan/15 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-16967 extend collection compact test to drop collection in the background

(cherry picked from commit 373b844920028ac35815f99172ac95684e5e2be8)
Branch: v3.0
https://github.com/mongodb/mongo/commit/90bbd525a01e710ec843bacebf82a62928c7458f

Comment by Githook User [ 27/Jan/15 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: SERVER-16967 extend collection compact test to drop collection in the background
Branch: master
https://github.com/mongodb/mongo/commit/373b844920028ac35815f99172ac95684e5e2be8

Comment by Eric Milkie [ 21/Jan/15 ]

Another failure, from Windows debug this time (easier to read backtrace):
http://buildlogs.mongodb.org/MCI_windows-64-2k8-debug/builds/601793/test/parallel_WT_0/basic.js

2015-01-21T15:47:38.814+0000 E STORAGE  [conn30] WiredTiger (0) [1421855258:814400][3748:1999844256], session.checkpoint: src\third_party\wiredtiger\src\schema\schema_list.c, 28: F_ISSET(session, WT_SESSION_TABLE_LOCKED) || !F_ISSET(session, WT_SESSION_HANDLE_LIST_LOCKED)
2015-01-21T15:47:38.814+0000 E STORAGE  [conn30] WiredTiger (0) [1421855258:814400][3748:1999844256], session.checkpoint: aborting WiredTiger library
2015-01-21T15:47:39.351+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\util\stacktrace_win.cpp(175)                            mongo::printStackTrace+0x43
2015-01-21T15:47:39.351+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\util\signal_handlers_synchronous.cpp(127)               mongo::`anonymous namespace'::printSignalAndBacktrace+0x73
2015-01-21T15:47:39.351+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\util\signal_handlers_synchronous.cpp(194)               mongo::`anonymous namespace'::abruptQuit+0x2e
2015-01-21T15:47:39.351+0000 I CONTROL  [conn30] mongod.exe    f:\dd\vctools\crt\crtw32\misc\winsig.c(594)                           raise+0x35f
2015-01-21T15:47:39.351+0000 I CONTROL  [conn30] mongod.exe    f:\dd\vctools\crt\crtw32\misc\abort.c(82)                             abort+0x40
2015-01-21T15:47:39.351+0000 I CONTROL  [conn30] mongod.exe    ...\src\third_party\wiredtiger\src\os_posix\os_abort.c(25)            __wt_abort+0x22
2015-01-21T15:47:39.351+0000 I CONTROL  [conn30] mongod.exe    ...\src\third_party\wiredtiger\src\support\err.c(451)                 __wt_assert+0x37
2015-01-21T15:47:39.352+0000 I CONTROL  [conn30] mongod.exe    ...\src\third_party\wiredtiger\src\schema\schema_list.c(28)           __schema_add_table+0x6c
2015-01-21T15:47:39.352+0000 I CONTROL  [conn30] mongod.exe    ...\src\third_party\wiredtiger\src\schema\schema_list.c(99)           __wt_schema_get_table+0x64
2015-01-21T15:47:39.352+0000 I CONTROL  [conn30] mongod.exe    ...\src\third_party\wiredtiger\src\schema\schema_worker.c(83)         __wt_schema_worker+0x2ee
2015-01-21T15:47:39.352+0000 I CONTROL  [conn30] mongod.exe    ...\src\third_party\wiredtiger\src\txn\txn_ckpt.c(132)                __checkpoint_apply_all+0x1d1
2015-01-21T15:47:39.352+0000 I CONTROL  [conn30] mongod.exe    ...\src\third_party\wiredtiger\src\txn\txn_ckpt.c(346)                __wt_txn_checkpoint+0x1ca
2015-01-21T15:47:39.352+0000 I CONTROL  [conn30] mongod.exe    ...\src\third_party\wiredtiger\src\session\session_api.c(889)         __session_checkpoint+0x1b3
2015-01-21T15:47:39.352+0000 I CONTROL  [conn30] mongod.exe    ...\src\third_party\wiredtiger\src\session\session_compact.c(185)     __compact_file+0x101
2015-01-21T15:47:39.352+0000 I CONTROL  [conn30] mongod.exe    ...\src\third_party\wiredtiger\src\session\session_compact.c(235)     __wt_session_compact+0x250
2015-01-21T15:47:39.353+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\db\storage\wiredtiger\wiredtiger_record_store.cpp(654)  mongo::WiredTigerRecordStore::compact+0x62
2015-01-21T15:47:39.353+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\db\catalog\collection_compact.cpp(118)                  mongo::Collection::compact+0x1cf
2015-01-21T15:47:39.353+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\db\commands\compact.cpp(168)                            mongo::CompactCmd::run+0x617
2015-01-21T15:47:39.353+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\db\dbcommands.cpp(1267)                                 mongo::_execCommand+0x72
2015-01-21T15:47:39.353+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\db\dbcommands.cpp(1485)                                 mongo::Command::execCommand+0xf01
2015-01-21T15:47:39.353+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\db\dbcommands.cpp(1560)                                 mongo::_runCommands+0x413
2015-01-21T15:47:39.353+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\db\query\find.cpp(137)                                  mongo::runCommands+0x43
2015-01-21T15:47:39.353+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\db\query\find.cpp(606)                                  mongo::runQuery+0x3c1
2015-01-21T15:47:39.354+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\db\instance.cpp(220)                                    mongo::receivedQuery+0x2ec
2015-01-21T15:47:39.354+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\db\instance.cpp(403)                                    mongo::assembleResponse+0x36c
2015-01-21T15:47:39.354+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\db\db.cpp(206)                                          mongo::MyMessageHandler::process+0xb6
2015-01-21T15:47:39.354+0000 I CONTROL  [conn30] mongod.exe    ...\src\mongo\util\net\message_server_port.cpp(229)                   mongo::PortMessageServer::handleIncomingMsg+0x670
2015-01-21T15:47:39.354+0000 I CONTROL  [conn30] mongod.exe    ...\src\third_party\boost\libs\thread\src\win32\thread.cpp(185)       boost::`anonymous namespace'::thread_start_function+0x21
2015-01-21T15:47:39.354+0000 I CONTROL  [conn30] mongod.exe    f:\dd\vctools\crt\crtw32\startup\threadex.c(376)                      _callthreadstartex+0x25
2015-01-21T15:47:39.354+0000 I CONTROL  [conn30] mongod.exe    f:\dd\vctools\crt\crtw32\startup\threadex.c(359)                      _threadstartex+0xe7
2015-01-21T15:47:39.354+0000 I CONTROL  [conn30] kernel32.dll                                                                        BaseThreadInitThunk+0xd
2015-01-21T15:47:39.354+0000 F -        [conn30] Got signal: 22 (SIGABRT).

Comment by Eric Milkie [ 21/Jan/15 ]

From this build: https://mci.10gen.com/task/mongodb_mongo_master_linux_64_debug_7d504b66098d1cfcafdb536ff0db3f7eb53f8c99_15_01_21_14_17_02_parallel_WT_linux_64_debug

Generated at Thu Feb 08 03:42:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.