|
2017-11-13T17:03:40.886+0000 I NETWORK [initandlisten] connection accepted from 161.202.16.42:40336 #978 (943 connections now open)
|
2017-11-13T17:03:40.887+0000 E STORAGE [repl writer worker 4] WiredTiger (22) [1510592620:887915][10630:0x7fc141a4f700], file:index-31062--824032195446804458.wt, WT_SESSION.open_cursor: live.avail: m
|
erge range 40960-53248 overlaps with existing range 49152-53248: Invalid argument
|
2017-11-13T17:03:40.887+0000 E STORAGE [repl writer worker 4] WiredTiger (-31804) [1510592620:887951][10630:0x7fc141a4f700], file:index-31062--824032195446804458.wt, WT_SESSION.open_cursor: the proce
|
ss must exit and restart: WT_PANIC: WiredTiger library panic
|
2017-11-13T17:03:40.887+0000 I - [repl writer worker 4] Fatal Assertion 28558
|
2017-11-13T17:03:40.888+0000 I - [repl writer worker 4]
|
|
***aborting after fassert() failure
|
|
|
2017-11-13T17:03:40.888+0000 I ACCESS [conn936] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.890+0000 I - [repl writer worker 5] Fatal Assertion 28559
|
2017-11-13T17:03:40.890+0000 I - [repl writer worker 5]
|
|
***aborting after fassert() failure
|
|
|
2017-11-13T17:03:40.890+0000 I - [repl writer worker 6] Fatal Assertion 28559
|
2017-11-13T17:03:40.890+0000 I - [repl writer worker 6]
|
|
***aborting after fassert() failure
|
|
|
2017-11-13T17:03:40.890+0000 I - [repl writer worker 7] Fatal Assertion 28559
|
2017-11-13T17:03:40.890+0000 I - [repl writer worker 7]
|
|
***aborting after fassert() failure
|
|
|
2017-11-13T17:03:40.890+0000 I - [repl writer worker 8] Fatal Assertion 28559
|
2017-11-13T17:03:40.890+0000 I - [repl writer worker 8]
|
|
***aborting after fassert() failure
|
|
|
2017-11-13T17:03:40.891+0000 I - [repl writer worker 9] Fatal Assertion 28559
|
2017-11-13T17:03:40.891+0000 I - [repl writer worker 9]
|
|
***aborting after fassert() failure
|
|
|
2017-11-13T17:03:40.891+0000 I - [repl writer worker 10] Fatal Assertion 28559
|
2017-11-13T17:03:40.891+0000 I - [repl writer worker 10]
|
|
***aborting after fassert() failure
|
|
|
2017-11-13T17:03:40.891+0000 I - [repl writer worker 11] Fatal Assertion 28559
|
2017-11-13T17:03:40.891+0000 I - [repl writer worker 11]
|
|
***aborting after fassert() failure
|
|
|
2017-11-13T17:03:40.891+0000 I - [repl writer worker 3] Fatal Assertion 28559
|
2017-11-13T17:03:40.891+0000 I - [repl writer worker 3]
|
|
***aborting after fassert() failure
|
|
|
2017-11-13T17:03:40.891+0000 I - [repl writer worker 2] Fatal Assertion 28559
|
2017-11-13T17:03:40.891+0000 I - [repl writer worker 2]
|
|
***aborting after fassert() failure
|
|
|
2017-11-13T17:03:40.892+0000 I NETWORK [initandlisten] connection accepted from 161.202.16.41:42326 #979 (944 connections now open)
|
2017-11-13T17:03:40.893+0000 I ACCESS [conn962] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.895+0000 I ACCESS [conn971] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.896+0000 I ACCESS [conn967] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.901+0000 I ACCESS [conn970] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.902+0000 I NETWORK [initandlisten] connection accepted from 161.202.16.41:42328 #980 (945 connections now open)
|
2017-11-13T17:03:40.904+0000 I ACCESS [conn966] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.905+0000 I ACCESS [conn969] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.908+0000 I NETWORK [initandlisten] connection accepted from 161.202.16.41:42330 #981 (946 connections now open)
|
2017-11-13T17:03:40.914+0000 I NETWORK [initandlisten] connection accepted from 161.202.16.42:40338 #982 (947 connections now open)
|
2017-11-13T17:03:40.916+0000 I NETWORK [initandlisten] connection accepted from 161.202.16.41:42336 #983 (948 connections now open)
|
2017-11-13T17:03:40.916+0000 I NETWORK [initandlisten] connection accepted from 161.202.16.41:42340 #984 (949 connections now open)
|
2017-11-13T17:03:40.916+0000 I NETWORK [initandlisten] connection accepted from 161.202.16.41:42338 #985 (950 connections now open)
|
2017-11-13T17:03:40.916+0000 I NETWORK [initandlisten] connection accepted from 161.202.16.41:42334 #986 (951 connections now open)
|
2017-11-13T17:03:40.917+0000 I ACCESS [conn977] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.918+0000 I ACCESS [conn899] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.918+0000 I ACCESS [conn913] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.918+0000 I ACCESS [conn907] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.920+0000 I NETWORK [initandlisten] connection accepted from 161.202.16.41:42344 #987 (952 connections now open)
|
2017-11-13T17:03:40.921+0000 I NETWORK [initandlisten] connection accepted from 161.202.16.41:42342 #988 (953 connections now open)
|
2017-11-13T17:03:40.922+0000 I ACCESS [conn904] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.923+0000 I ACCESS [conn927] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.923+0000 I ACCESS [conn972] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.925+0000 I NETWORK [initandlisten] connection accepted from 161.202.16.41:42346 #989 (954 connections now open)
|
2017-11-13T17:03:40.927+0000 I ACCESS [conn895] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.927+0000 I ACCESS [conn924] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.928+0000 I ACCESS [conn918] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.928+0000 I ACCESS [conn942] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.928+0000 I ACCESS [conn876] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.929+0000 I ACCESS [conn919] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.930+0000 I ACCESS [conn903] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.932+0000 I ACCESS [conn976] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.932+0000 I ACCESS [conn974] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.932+0000 I ACCESS [conn973] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.933+0000 I ACCESS [conn877] Successfully authenticated as principal admin on admin
|
2017-11-13T17:03:40.934+0000 F - [repl writer worker 4] Got signal: 6 (Aborted).
|
|
|
0x130e1e2 0x130d119 0x130d922 0x7fc15ab4c100 0x7fc15a7b05f7 0x7fc15a7b1ce8 0x1295b32 0x108b3e3 0x1a57e3c 0x1a57fdd 0x1a583c4 0x198f81d 0x19909a4 0x1990bf3 0x198dbd7 0x19919de 0x19ae1a8 0x19e2140 0x1a
|
56bfb 0x1a57117 0x1a5726b 0x19f02f8 0x1a54415 0x1a061a3 0x1a54299 0x1a54829 0x1085b14 0x1084460 0x106856b 0xc87055 0xbc96fa 0xbfefc4 0xe0c6d5 0xe0cd99 0xe0ce95 0xd17139 0xe9d024 0xf34c6e 0xf2ea60 0xf3
|
07e6 0xf313e6 0xf345ab 0x12a0651 0x12a0fb9 0x12a1b10 0x1ac9600 0x7fc15ab44dc5 0x7fc15a871c9d
|
----- BEGIN BACKTRACE -----
|
{"backtrace":[{"b":"400000","o":"F0E1E2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F0D119"},{"b":"400000","o":"F0D922"},{"b":"7FC15AB3D000","o":"F100"},{"b":"7FC15A77B000","o":"355F7","
|
s":"gsignal"},{"b":"7FC15A77B000","o":"36CE8","s":"abort"},{"b":"400000","o":"E95B32","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"C8B3E3"},{"b":"400000","o":"1657E3C","s":"__wt_eventv"},{"b":
|
"400000","o":"1657FDD","s":"__wt_err"},{"b":"400000","o":"16583C4","s":"__wt_panic"},{"b":"400000","o":"158F81D"},{"b":"400000","o":"15909A4","s":"__wt_block_extlist_read"},{"b":"400000","o":"1590BF3"
|
,"s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"158DBD7","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"15919DE"},{"b":"400000","o":"15AE1A8","s":"__wt_btree_open"},{"b":"400000","o":"1
|
5E2140","s":"__wt_conn_btree_open"},{"b":"400000","o":"1656BFB","s":"__wt_session_get_btree"},{"b":"400000","o":"1657117","s":"__wt_session_get_btree"},{"b":"400000","o":"165726B","s":"__wt_session_ge
|
t_btree_ckpt"},{"b":"400000","o":"15F02F8","s":"__wt_curfile_open"},{"b":"400000","o":"1654415"},{"b":"400000","o":"16061A3","s":"__wt_curtable_open"},{"b":"400000","o":"1654299"},{"b":"400000","o":"1
|
654829"},{"b":"400000","o":"C85B14","s":"_ZN5mongo17WiredTigerSession9getCursorERKSsmb"},{"b":"400000","o":"C84460","s":"_ZN5mongo16WiredTigerCursorC1ERKSsmbPNS_16OperationContextE"},{"b":"400000","o"
|
:"C6856B","s":"_ZNK5mongo21WiredTigerIndexUnique9newCursorEPNS_16OperationContextEb"},{"b":"400000","o":"887055","s":"_ZNK5mongo17IndexAccessMethod10findSingleEPNS_16OperationContextERKNS_7BSONObjE"},
|
{"b":"400000","o":"7C96FA","s":"_ZN5mongo11IDHackStage4workEPm"},{"b":"400000","o":"7FEFC4","s":"_ZN5mongo11UpdateStage4workEPm"},{"b":"400000","o":"A0C6D5","s":"_ZN5mongo12PlanExecutor11getNextImplEP
|
NS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE"},{"b":"400000","o":"A0CD99","s":"_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE"},{"b":"400000","o":"A0CE95","s":"_ZN5mongo12PlanExecutor11e
|
xecutePlanEv"},{"b":"400000","o":"917139","s":"_ZN5mongo6updateEPNS_16OperationContextEPNS_8DatabaseERKNS_13UpdateRequestEPNS_7OpDebugE"},{"b":"400000","o":"A9D024","s":"_ZN5mongo4repl21applyOperation
|
_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEb"},{"b":"400000","o":"B34C6E","s":"_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseERKNS0_7BSONObjEbEPS9_E9_M
|
_invokeERKSt9_Any_dataS3_S5_S8_b"},{"b":"400000","o":"B2EA60"},{"b":"400000","o":"B307E6","s":"_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEbSt8functionIFNS_6StatusES3_PNS_8D
|
atabaseES6_bEES7_IFS8_S3_S6_EES7_IFvvEE"},{"b":"400000","o":"B313E6","s":"_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb"},{"b":"400000","o":"B345AB","s":"_ZN5mongo4repl14mul
|
tiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE"},{"b":"400000","o":"EA0651","s":"_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE"},{"b":"400000","o":"EA0FB9","s":"_ZN5mongo10
|
ThreadPool13_consumeTasksEv"},{"b":"400000","o":"EA1B10","s":"_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKSs"},{"b":"400000","o":"16C9600"},{"b":"7FC15AB3D000","o":"7DC5"},{"b":"7FC15A77B000","o":"
|
F6C9D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.3", "gitVersion" : "b326ba837cf6f49d65c2f85e1b70f6f31ece7937", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.11
|
-23.53.amzn1.x86_64", "version" : "#1 SMP Wed Jun 1 22:22:50 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "B389239A9108B604FCA797D0F968E253E7BF9F5E" }, {
|
"b" : "7FFDD2F89000", "elfType" : 3, "buildId" : "41B0E5BE826EA13B746A0EEA2F99E39A5E936B59" }, { "b" : "7FC15BD68000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "C9DB6FDB87F3CAAF
|
209AEACD2EC345B152605135" }, { "b" : "7FC15B982000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "09CACF79947C91081F28FCCAB5AF524D51751AF0" }, { "b" : "7FC15B77A000", "path" : "/lib6
|
4/librt.so.1", "elfType" : 3, "buildId" : "42833B65941483A611C40EA7D32F56EA83EA6E93" }, { "b" : "7FC15B576000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "6335077ACD51527BE9F2F18451A88E
|
2B7350C5B6" }, { "b" : "7FC15B271000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "0A90C35D3174805453EA67A785446D628E298B59" }, { "b" : "7FC15AF6F000", "path" : "/lib64/libm.so.6
|
", "elfType" : 3, "buildId" : "BB312C4A65B8FD830C148612CBEACEACC8B08E4F" }, { "b" : "7FC15AD59000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "00FA2883FB47B1327397BBF167C52F51A723D01
|
3" }, { "b" : "7FC15AB3D000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "E5E575776DAD20ADE8FD0CAF17897C9D89020A87" }, { "b" : "7FC15A77B000", "path" : "/lib64/libc.so.6", "elfType"
|
: 3, "buildId" : "7128B52741C2494951D285A80DAF25B2BC9C7A0D" }, { "b" : "7FC15BFD5000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "7B7BF8FEEF1A9C627EF90CA5C9188EFD4DA2DDD2" },
|
{ "b" : "7FC15A52F000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "E203354E7F907ACC8C3028FE465541B666DCFBA0" }, { "b" : "7FC15A24A000", "path" : "/lib64/libkrb5.so.3", "elfTyp
|
e" : 3, "buildId" : "D769C8FFAF8772FDA55031ABF2E167DF2207E378" }, { "b" : "7FC15A047000", "path" : "/usr/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "5C01209C5AE1B1714F19B07EB58F2A1274B69DC8" }
|
, { "b" : "7FC159E15000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "6C2243D37143F7FD1E16ED1F6CE4D7F16C2D7EF1" }, { "b" : "7FC159BFF000", "path" : "/lib64/libz.so.1", "elfType" :
|
3, "buildId" : "89C6AF118B6B4FB6A73AE1813E2C8BDD722956D1" }, { "b" : "7FC1599F0000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "744272FEAAABCE629AB9E11FAA4A96AEBE8BC2B4" }, { "
|
b" : "7FC1597ED000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "37A58210FA50C91E09387765408A92909468D25B" }, { "b" : "7FC1595D3000", "path" : "/lib64/libresolv.so.2", "elfType" :
|
3, "buildId" : "EE46CA824A1683662BCDA445CC1A4122BBE552BA" }, { "b" : "7FC1593B2000", "path" : "/usr/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "F5054DC94443326819FBF3065CFDF5E4726F57EE" } ] }}
|
mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x130e1e2]
|
mongod(+0xF0D119) [0x130d119]
|
mongod(+0xF0D922) [0x130d922]
|
libpthread.so.0(+0xF100) [0x7fc15ab4c100]
|
libc.so.6(gsignal+0x37) [0x7fc15a7b05f7]
|
libc.so.6(abort+0x148) [0x7fc15a7b1ce8]
|
mongod(_ZN5mongo13fassertFailedEi+0x82) [0x1295b32]
|
mongod(+0xC8B3E3) [0x108b3e3]
|
mongod(__wt_eventv+0x40C) [0x1a57e3c]
|
mongod(__wt_err+0x8D) [0x1a57fdd]
|
mongod(__wt_panic+0x24) [0x1a583c4]
|
mongod(+0x158F81D) [0x198f81d]
|
mongod(__wt_block_extlist_read+0x394) [0x19909a4]
|
mongod(__wt_block_extlist_read_avail+0x33) [0x1990bf3]
|
mongod(__wt_block_checkpoint_load+0x3B7) [0x198dbd7]
|
mongod(+0x15919DE) [0x19919de]
|
mongod(__wt_btree_open+0xC68) [0x19ae1a8]
|
mongod(__wt_conn_btree_open+0x140) [0x19e2140]
|
mongod(__wt_session_get_btree+0xEB) [0x1a56bfb]
|
mongod(__wt_session_get_btree+0x607) [0x1a57117]
|
mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a5726b]
|
mongod(__wt_curfile_open+0x1E8) [0x19f02f8]
|
mongod(+0x1654415) [0x1a54415]
|
mongod(__wt_curtable_open+0x393) [0x1a061a3]
|
mongod(+0x1654299) [0x1a54299]
|
mongod(+0x1654829) [0x1a54829]
|
mongod(_ZN5mongo17WiredTigerSession9getCursorERKSsmb+0x94) [0x1085b14]
|
mongod(_ZN5mongo16WiredTigerCursorC1ERKSsmbPNS_16OperationContextE+0x50) [0x1084460]
|
mongod(_ZNK5mongo21WiredTigerIndexUnique9newCursorEPNS_16OperationContextEb+0x15B) [0x106856b]
|
mongod(_ZNK5mongo17IndexAccessMethod10findSingleEPNS_16OperationContextERKNS_7BSONObjE+0x25) [0xc87055]
|
mongod(_ZN5mongo11IDHackStage4workEPm+0x11A) [0xbc96fa]
|
mongod(_ZN5mongo11UpdateStage4workEPm+0x394) [0xbfefc4]
|
mongod(_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x275) [0xe0c6d5]
|
mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x39) [0xe0cd99]
|
mongod(_ZN5mongo12PlanExecutor11executePlanEv+0x55) [0xe0ce95]
|
mongod(_ZN5mongo6updateEPNS_16OperationContextEPNS_8DatabaseERKNS_13UpdateRequestEPNS_7OpDebugE+0x179) [0xd17139]
|
mongod(_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEb+0x924) [0xe9d024]
|
mongod(_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseERKNS0_7BSONObjEbEPS9_E9_M_invokeERKSt9_Any_dataS3_S5_S8_b+0x1E) [0xf34c6e]
|
mongod(+0xB2EA60) [0xf2ea60]
|
mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEbSt8functionIFNS_6StatusES3_PNS_8DatabaseES6_bEES7_IFS8_S3_S6_EES7_IFvvEE+0x336) [0xf307e6]
|
mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb+0xE6) [0xf313e6]
|
mongod(_ZN5mongo4repl14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x9B) [0xf345ab]
|
mongod(_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE+0x121) [0x12a0651]
|
mongod(_ZN5mongo10ThreadPool13_consumeTasksEv+0xA9) [0x12a0fb9]
|
mongod(_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKSs+0x100) [0x12a1b10]
|
mongod(+0x16C9600) [0x1ac9600]
|
libpthread.so.0(+0x7DC5) [0x7fc15ab44dc5]
|
libc.so.6(clone+0x6D) [0x7fc15a871c9d]
|
----- END BACKTRACE -----
|
2017-11-13T17:03:40.936+0000 I ACCESS [conn979] Successfully authenticated as principal admin on admin
|