Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-31939

MongoDB secondary in 3.2.3 crashes and does not restart - m erge range 40960-53248 overlaps with existing range 49152-53248: Invalid argument

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.2.3
    • Component/s: None
    • Labels:
      None
    • ALL

      2017-11-13T17:03:40.886+0000 I NETWORK  [initandlisten] connection accepted from 161.202.16.42:40336 #978 (943 connections now open)
      2017-11-13T17:03:40.887+0000 E STORAGE  [repl writer worker 4] WiredTiger (22) [1510592620:887915][10630:0x7fc141a4f700], file:index-31062--824032195446804458.wt, WT_SESSION.open_cursor: live.avail: m
      erge range 40960-53248 overlaps with existing range 49152-53248: Invalid argument
      2017-11-13T17:03:40.887+0000 E STORAGE  [repl writer worker 4] WiredTiger (-31804) [1510592620:887951][10630:0x7fc141a4f700], file:index-31062--824032195446804458.wt, WT_SESSION.open_cursor: the proce
      ss must exit and restart: WT_PANIC: WiredTiger library panic
      2017-11-13T17:03:40.887+0000 I -        [repl writer worker 4] Fatal Assertion 28558
      2017-11-13T17:03:40.888+0000 I -        [repl writer worker 4]
      
      ***aborting after fassert() failure
      
      
      2017-11-13T17:03:40.888+0000 I ACCESS   [conn936] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.890+0000 I -        [repl writer worker 5] Fatal Assertion 28559
      2017-11-13T17:03:40.890+0000 I -        [repl writer worker 5]
      
      ***aborting after fassert() failure
      
      
      2017-11-13T17:03:40.890+0000 I -        [repl writer worker 6] Fatal Assertion 28559
      2017-11-13T17:03:40.890+0000 I -        [repl writer worker 6]
      
      ***aborting after fassert() failure
      
      
      2017-11-13T17:03:40.890+0000 I -        [repl writer worker 7] Fatal Assertion 28559
      2017-11-13T17:03:40.890+0000 I -        [repl writer worker 7]
      
      ***aborting after fassert() failure
      
      
      2017-11-13T17:03:40.890+0000 I -        [repl writer worker 8] Fatal Assertion 28559
      2017-11-13T17:03:40.890+0000 I -        [repl writer worker 8]
      
      ***aborting after fassert() failure
      
      
      2017-11-13T17:03:40.891+0000 I -        [repl writer worker 9] Fatal Assertion 28559
      2017-11-13T17:03:40.891+0000 I -        [repl writer worker 9]
      
      ***aborting after fassert() failure
      
      
      2017-11-13T17:03:40.891+0000 I -        [repl writer worker 10] Fatal Assertion 28559
      2017-11-13T17:03:40.891+0000 I -        [repl writer worker 10]
      
      ***aborting after fassert() failure
      
      
      2017-11-13T17:03:40.891+0000 I -        [repl writer worker 11] Fatal Assertion 28559
      2017-11-13T17:03:40.891+0000 I -        [repl writer worker 11]
      
      ***aborting after fassert() failure
      
      
      2017-11-13T17:03:40.891+0000 I -        [repl writer worker 3] Fatal Assertion 28559
      2017-11-13T17:03:40.891+0000 I -        [repl writer worker 3]
      
      ***aborting after fassert() failure
      
      
      2017-11-13T17:03:40.891+0000 I -        [repl writer worker 2] Fatal Assertion 28559
      2017-11-13T17:03:40.891+0000 I -        [repl writer worker 2]
      
      ***aborting after fassert() failure
      
      
      2017-11-13T17:03:40.892+0000 I NETWORK  [initandlisten] connection accepted from 161.202.16.41:42326 #979 (944 connections now open)
      2017-11-13T17:03:40.893+0000 I ACCESS   [conn962] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.895+0000 I ACCESS   [conn971] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.896+0000 I ACCESS   [conn967] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.901+0000 I ACCESS   [conn970] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.902+0000 I NETWORK  [initandlisten] connection accepted from 161.202.16.41:42328 #980 (945 connections now open)
      2017-11-13T17:03:40.904+0000 I ACCESS   [conn966] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.905+0000 I ACCESS   [conn969] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.908+0000 I NETWORK  [initandlisten] connection accepted from 161.202.16.41:42330 #981 (946 connections now open)
      2017-11-13T17:03:40.914+0000 I NETWORK  [initandlisten] connection accepted from 161.202.16.42:40338 #982 (947 connections now open)
      2017-11-13T17:03:40.916+0000 I NETWORK  [initandlisten] connection accepted from 161.202.16.41:42336 #983 (948 connections now open)
      2017-11-13T17:03:40.916+0000 I NETWORK  [initandlisten] connection accepted from 161.202.16.41:42340 #984 (949 connections now open)
      2017-11-13T17:03:40.916+0000 I NETWORK  [initandlisten] connection accepted from 161.202.16.41:42338 #985 (950 connections now open)
      2017-11-13T17:03:40.916+0000 I NETWORK  [initandlisten] connection accepted from 161.202.16.41:42334 #986 (951 connections now open)
      2017-11-13T17:03:40.917+0000 I ACCESS   [conn977] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.918+0000 I ACCESS   [conn899] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.918+0000 I ACCESS   [conn913] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.918+0000 I ACCESS   [conn907] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.920+0000 I NETWORK  [initandlisten] connection accepted from 161.202.16.41:42344 #987 (952 connections now open)
      2017-11-13T17:03:40.921+0000 I NETWORK  [initandlisten] connection accepted from 161.202.16.41:42342 #988 (953 connections now open)
      2017-11-13T17:03:40.922+0000 I ACCESS   [conn904] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.923+0000 I ACCESS   [conn927] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.923+0000 I ACCESS   [conn972] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.925+0000 I NETWORK  [initandlisten] connection accepted from 161.202.16.41:42346 #989 (954 connections now open)
      2017-11-13T17:03:40.927+0000 I ACCESS   [conn895] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.927+0000 I ACCESS   [conn924] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.928+0000 I ACCESS   [conn918] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.928+0000 I ACCESS   [conn942] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.928+0000 I ACCESS   [conn876] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.929+0000 I ACCESS   [conn919] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.930+0000 I ACCESS   [conn903] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.932+0000 I ACCESS   [conn976] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.932+0000 I ACCESS   [conn974] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.932+0000 I ACCESS   [conn973] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.933+0000 I ACCESS   [conn877] Successfully authenticated as principal admin on admin
      2017-11-13T17:03:40.934+0000 F -        [repl writer worker 4] Got signal: 6 (Aborted).
      
      
       0x130e1e2 0x130d119 0x130d922 0x7fc15ab4c100 0x7fc15a7b05f7 0x7fc15a7b1ce8 0x1295b32 0x108b3e3 0x1a57e3c 0x1a57fdd 0x1a583c4 0x198f81d 0x19909a4 0x1990bf3 0x198dbd7 0x19919de 0x19ae1a8 0x19e2140 0x1a
      56bfb 0x1a57117 0x1a5726b 0x19f02f8 0x1a54415 0x1a061a3 0x1a54299 0x1a54829 0x1085b14 0x1084460 0x106856b 0xc87055 0xbc96fa 0xbfefc4 0xe0c6d5 0xe0cd99 0xe0ce95 0xd17139 0xe9d024 0xf34c6e 0xf2ea60 0xf3
      07e6 0xf313e6 0xf345ab 0x12a0651 0x12a0fb9 0x12a1b10 0x1ac9600 0x7fc15ab44dc5 0x7fc15a871c9d
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"F0E1E2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F0D119"},{"b":"400000","o":"F0D922"},{"b":"7FC15AB3D000","o":"F100"},{"b":"7FC15A77B000","o":"355F7","
      s":"gsignal"},{"b":"7FC15A77B000","o":"36CE8","s":"abort"},{"b":"400000","o":"E95B32","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"C8B3E3"},{"b":"400000","o":"1657E3C","s":"__wt_eventv"},{"b":
      "400000","o":"1657FDD","s":"__wt_err"},{"b":"400000","o":"16583C4","s":"__wt_panic"},{"b":"400000","o":"158F81D"},{"b":"400000","o":"15909A4","s":"__wt_block_extlist_read"},{"b":"400000","o":"1590BF3"
      ,"s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"158DBD7","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"15919DE"},{"b":"400000","o":"15AE1A8","s":"__wt_btree_open"},{"b":"400000","o":"1
      5E2140","s":"__wt_conn_btree_open"},{"b":"400000","o":"1656BFB","s":"__wt_session_get_btree"},{"b":"400000","o":"1657117","s":"__wt_session_get_btree"},{"b":"400000","o":"165726B","s":"__wt_session_ge
      t_btree_ckpt"},{"b":"400000","o":"15F02F8","s":"__wt_curfile_open"},{"b":"400000","o":"1654415"},{"b":"400000","o":"16061A3","s":"__wt_curtable_open"},{"b":"400000","o":"1654299"},{"b":"400000","o":"1
      654829"},{"b":"400000","o":"C85B14","s":"_ZN5mongo17WiredTigerSession9getCursorERKSsmb"},{"b":"400000","o":"C84460","s":"_ZN5mongo16WiredTigerCursorC1ERKSsmbPNS_16OperationContextE"},{"b":"400000","o"
      :"C6856B","s":"_ZNK5mongo21WiredTigerIndexUnique9newCursorEPNS_16OperationContextEb"},{"b":"400000","o":"887055","s":"_ZNK5mongo17IndexAccessMethod10findSingleEPNS_16OperationContextERKNS_7BSONObjE"},
      {"b":"400000","o":"7C96FA","s":"_ZN5mongo11IDHackStage4workEPm"},{"b":"400000","o":"7FEFC4","s":"_ZN5mongo11UpdateStage4workEPm"},{"b":"400000","o":"A0C6D5","s":"_ZN5mongo12PlanExecutor11getNextImplEP
      NS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE"},{"b":"400000","o":"A0CD99","s":"_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE"},{"b":"400000","o":"A0CE95","s":"_ZN5mongo12PlanExecutor11e
      xecutePlanEv"},{"b":"400000","o":"917139","s":"_ZN5mongo6updateEPNS_16OperationContextEPNS_8DatabaseERKNS_13UpdateRequestEPNS_7OpDebugE"},{"b":"400000","o":"A9D024","s":"_ZN5mongo4repl21applyOperation
      _inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEb"},{"b":"400000","o":"B34C6E","s":"_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseERKNS0_7BSONObjEbEPS9_E9_M
      _invokeERKSt9_Any_dataS3_S5_S8_b"},{"b":"400000","o":"B2EA60"},{"b":"400000","o":"B307E6","s":"_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEbSt8functionIFNS_6StatusES3_PNS_8D
      atabaseES6_bEES7_IFS8_S3_S6_EES7_IFvvEE"},{"b":"400000","o":"B313E6","s":"_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb"},{"b":"400000","o":"B345AB","s":"_ZN5mongo4repl14mul
      tiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE"},{"b":"400000","o":"EA0651","s":"_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE"},{"b":"400000","o":"EA0FB9","s":"_ZN5mongo10
      ThreadPool13_consumeTasksEv"},{"b":"400000","o":"EA1B10","s":"_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKSs"},{"b":"400000","o":"16C9600"},{"b":"7FC15AB3D000","o":"7DC5"},{"b":"7FC15A77B000","o":"
      F6C9D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.3", "gitVersion" : "b326ba837cf6f49d65c2f85e1b70f6f31ece7937", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.11
      -23.53.amzn1.x86_64", "version" : "#1 SMP Wed Jun 1 22:22:50 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "B389239A9108B604FCA797D0F968E253E7BF9F5E" }, {
       "b" : "7FFDD2F89000", "elfType" : 3, "buildId" : "41B0E5BE826EA13B746A0EEA2F99E39A5E936B59" }, { "b" : "7FC15BD68000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "C9DB6FDB87F3CAAF
      209AEACD2EC345B152605135" }, { "b" : "7FC15B982000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "09CACF79947C91081F28FCCAB5AF524D51751AF0" }, { "b" : "7FC15B77A000", "path" : "/lib6
      4/librt.so.1", "elfType" : 3, "buildId" : "42833B65941483A611C40EA7D32F56EA83EA6E93" }, { "b" : "7FC15B576000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "6335077ACD51527BE9F2F18451A88E
      2B7350C5B6" }, { "b" : "7FC15B271000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "0A90C35D3174805453EA67A785446D628E298B59" }, { "b" : "7FC15AF6F000", "path" : "/lib64/libm.so.6
      ", "elfType" : 3, "buildId" : "BB312C4A65B8FD830C148612CBEACEACC8B08E4F" }, { "b" : "7FC15AD59000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "00FA2883FB47B1327397BBF167C52F51A723D01
      3" }, { "b" : "7FC15AB3D000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "E5E575776DAD20ADE8FD0CAF17897C9D89020A87" }, { "b" : "7FC15A77B000", "path" : "/lib64/libc.so.6", "elfType"
       : 3, "buildId" : "7128B52741C2494951D285A80DAF25B2BC9C7A0D" }, { "b" : "7FC15BFD5000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "7B7BF8FEEF1A9C627EF90CA5C9188EFD4DA2DDD2" },
       { "b" : "7FC15A52F000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "E203354E7F907ACC8C3028FE465541B666DCFBA0" }, { "b" : "7FC15A24A000", "path" : "/lib64/libkrb5.so.3", "elfTyp
      e" : 3, "buildId" : "D769C8FFAF8772FDA55031ABF2E167DF2207E378" }, { "b" : "7FC15A047000", "path" : "/usr/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "5C01209C5AE1B1714F19B07EB58F2A1274B69DC8" }
      , { "b" : "7FC159E15000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "6C2243D37143F7FD1E16ED1F6CE4D7F16C2D7EF1" }, { "b" : "7FC159BFF000", "path" : "/lib64/libz.so.1", "elfType" :
      3, "buildId" : "89C6AF118B6B4FB6A73AE1813E2C8BDD722956D1" }, { "b" : "7FC1599F0000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "744272FEAAABCE629AB9E11FAA4A96AEBE8BC2B4" }, { "
      b" : "7FC1597ED000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "37A58210FA50C91E09387765408A92909468D25B" }, { "b" : "7FC1595D3000", "path" : "/lib64/libresolv.so.2", "elfType" :
      3, "buildId" : "EE46CA824A1683662BCDA445CC1A4122BBE552BA" }, { "b" : "7FC1593B2000", "path" : "/usr/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "F5054DC94443326819FBF3065CFDF5E4726F57EE" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x130e1e2]
       mongod(+0xF0D119) [0x130d119]
       mongod(+0xF0D922) [0x130d922]
       libpthread.so.0(+0xF100) [0x7fc15ab4c100]
       libc.so.6(gsignal+0x37) [0x7fc15a7b05f7]
       libc.so.6(abort+0x148) [0x7fc15a7b1ce8]
       mongod(_ZN5mongo13fassertFailedEi+0x82) [0x1295b32]
       mongod(+0xC8B3E3) [0x108b3e3]
       mongod(__wt_eventv+0x40C) [0x1a57e3c]
       mongod(__wt_err+0x8D) [0x1a57fdd]
       mongod(__wt_panic+0x24) [0x1a583c4]
       mongod(+0x158F81D) [0x198f81d]
       mongod(__wt_block_extlist_read+0x394) [0x19909a4]
       mongod(__wt_block_extlist_read_avail+0x33) [0x1990bf3]
       mongod(__wt_block_checkpoint_load+0x3B7) [0x198dbd7]
       mongod(+0x15919DE) [0x19919de]
       mongod(__wt_btree_open+0xC68) [0x19ae1a8]
       mongod(__wt_conn_btree_open+0x140) [0x19e2140]
       mongod(__wt_session_get_btree+0xEB) [0x1a56bfb]
       mongod(__wt_session_get_btree+0x607) [0x1a57117]
       mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a5726b]
       mongod(__wt_curfile_open+0x1E8) [0x19f02f8]
       mongod(+0x1654415) [0x1a54415]
       mongod(__wt_curtable_open+0x393) [0x1a061a3]
       mongod(+0x1654299) [0x1a54299]
       mongod(+0x1654829) [0x1a54829]
       mongod(_ZN5mongo17WiredTigerSession9getCursorERKSsmb+0x94) [0x1085b14]
       mongod(_ZN5mongo16WiredTigerCursorC1ERKSsmbPNS_16OperationContextE+0x50) [0x1084460]
       mongod(_ZNK5mongo21WiredTigerIndexUnique9newCursorEPNS_16OperationContextEb+0x15B) [0x106856b]
       mongod(_ZNK5mongo17IndexAccessMethod10findSingleEPNS_16OperationContextERKNS_7BSONObjE+0x25) [0xc87055]
       mongod(_ZN5mongo11IDHackStage4workEPm+0x11A) [0xbc96fa]
       mongod(_ZN5mongo11UpdateStage4workEPm+0x394) [0xbfefc4]
       mongod(_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x275) [0xe0c6d5]
       mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x39) [0xe0cd99]
       mongod(_ZN5mongo12PlanExecutor11executePlanEv+0x55) [0xe0ce95]
       mongod(_ZN5mongo6updateEPNS_16OperationContextEPNS_8DatabaseERKNS_13UpdateRequestEPNS_7OpDebugE+0x179) [0xd17139]
       mongod(_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEb+0x924) [0xe9d024]
       mongod(_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseERKNS0_7BSONObjEbEPS9_E9_M_invokeERKSt9_Any_dataS3_S5_S8_b+0x1E) [0xf34c6e]
       mongod(+0xB2EA60) [0xf2ea60]
       mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEbSt8functionIFNS_6StatusES3_PNS_8DatabaseES6_bEES7_IFS8_S3_S6_EES7_IFvvEE+0x336) [0xf307e6]
       mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb+0xE6) [0xf313e6]
       mongod(_ZN5mongo4repl14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x9B) [0xf345ab]
       mongod(_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE+0x121) [0x12a0651]
       mongod(_ZN5mongo10ThreadPool13_consumeTasksEv+0xA9) [0x12a0fb9]
       mongod(_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKSs+0x100) [0x12a1b10]
       mongod(+0x16C9600) [0x1ac9600]
       libpthread.so.0(+0x7DC5) [0x7fc15ab44dc5]
       libc.so.6(clone+0x6D) [0x7fc15a871c9d]
      -----  END BACKTRACE  -----
      2017-11-13T17:03:40.936+0000 I ACCESS   [conn979] Successfully authenticated as principal admin on admin
      

            Assignee:
            Unassigned Unassigned
            Reporter:
            dharshanr@scalegrid.net Dharshan Rangegowda
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: