Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-27863

Reschedule early alarms in NetworkInterfaceASIO to avoid mongos crash

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical - P2
    • Resolution: Fixed
    • Affects Version/s: 3.4.1
    • Fix Version/s: 3.4.3, 3.5.4
    • Component/s: Shell
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Requested:
      v3.4
    • Steps To Reproduce:
      Hide

      It happens in non busy hours

      Show
      It happens in non busy hours
    • Sprint:
      HELP/DWS/BF Tracking Sprint

      Description

      Mongos servers are crashing with the following log messages

      2017-01-31T05:16:32.309+0400 I -        [ReplicaSetMonitor-TaskExecutor-0] Invariant failure now() >= when src/mongo/executor/thread_pool_task_executor.cpp 294
      2017-01-31T05:16:32.309+0400 I -        [ReplicaSetMonitor-TaskExecutor-0] 
       
      ***aborting after invariant() failure
       
       
      2017-01-31T05:16:32.349+0400 F -        [ReplicaSetMonitor-TaskExecutor-0] Got signal: 6 (Aborted).
       
       0x7f8bca5635b1 0x7f8bca5626a9 0x7f8bca562b8d 0x7f8bc8629330 0x7f8bc828ac37 0x7f8bc828e028 0x7f8bc9d9f204 0x7f8bca2b8f18 0x7f8bca290225 0x7f8bca641164 0x7f8bca641361 0x7f8bca28f32f 0x7f8bca941ba0 0x7f8bc8621184 0x7f8bc834e37d
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"7F8BC99BF000","o":"BA45B1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7F8BC99BF000","o":"BA36A9"},{"b":"7F8BC99BF000","o":"BA3B8D"},{"b":"7F8BC8619000","o":"10330"},{"b":"7F8BC8254000","o":"36C37","s":"gsignal"},{"b":"7F8BC8254000","o":"3A028","s":"abort"},{"b":"7F8BC99BF000","o":"3E0204","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},{"b":"7F8BC99BF000","o":"8F9F18"},{"b":"7F8BC99BF000","o":"8D1225"},{"b":"7F8BC99BF000","o":"C82164","s":"_ZN4asio6detail9scheduler10do_run_oneERNS0_11scoped_lockINS0_11posix_mutexEEERNS0_21scheduler_thread_infoERKSt10error_code"},{"b":"7F8BC99BF000","o":"C82361","s":"_ZN4asio6detail9scheduler3runERSt10error_code"},{"b":"7F8BC99BF000","o":"8D032F"},{"b":"7F8BC99BF000","o":"F82BA0"},{"b":"7F8BC8619000","o":"8184"},{"b":"7F8BC8254000","o":"FA37D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.1", "gitVersion" : "5e103c4f5583e2566a45d740225dc250baacfbd7", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.19.0-33-generic", "version" : "#38~14.04.1-Ubuntu SMP Fri Nov 6 18:17:28 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "b" : "7F8BC99BF000", "elfType" : 3, "buildId" : "0E781D7641A3474BB32487BBA122619FE8606F8F" }, { "b" : "7FFCC0BE6000", "elfType" : 3, "buildId" : "3AA714083E1D7D719D6C5A7B1A4A8D3649727CB8" }, { "b" : "7F8BC953B000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "15A6A69B862FAC0AA589921DB1B0DF8A673FEC81" }, { "b" : "7F8BC915F000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "607C4A6F3428B6F065D55691AFDC4E343786A8A5" }, { "b" : "7F8BC8F57000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "E2A6DD5048A0A051FD61043BDB69D8CC68192AB7" }, { "b" : "7F8BC8D53000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "DA9B8C234D0FE9FD8CAAC8970A7EC1B6C8F6623F" }, { "b" : "7F8BC8A4D000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "D144258E614900B255A31F3FD2283A878670D5BC" }, { "b" : "7F8BC8837000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7F8BC8619000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "31E9F21AE8C10396171F1E13DA15780986FA696C" }, { "b" : "7F8BC8254000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "CF699A15CAAE64F50311FC4655B86DC39A479789" }, { "b" : "7F8BC979A000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D0F537904076D73F29E4A37341F8A449E2EF6CD0" }, { "b" : "7F8BC8049000", "path" : "/lib/x86_64-linux-gnu/libnss_files.so.2", "elfType" : 3, "buildId" : "A0BDDA36D5AD1390D4DF158D7AD7CC866E534C28" }, { "b" : "7F8BC7E43000", "path" : "/lib/x86_64-linux-gnu/libnss_dns.so.2", "elfType" : 3, "buildId" : "A5721A2FACA0CB423F6C851DF8B7A96C52E94691" }, { "b" : "7F8BC7C28000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "FE705AA7E3872B43EA9E99FAF1897F7F85FC0AC9" } ] }}
       mongos(_ZN5mongo15printStackTraceERSo+0x41) [0x7f8bca5635b1]
       mongos(+0xBA36A9) [0x7f8bca5626a9]
       mongos(+0xBA3B8D) [0x7f8bca562b8d]
       libpthread.so.0(+0x10330) [0x7f8bc8629330]
       libc.so.6(gsignal+0x37) [0x7f8bc828ac37]
       libc.so.6(abort+0x148) [0x7f8bc828e028]
       mongos(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0x0) [0x7f8bc9d9f204]
       mongos(+0x8F9F18) [0x7f8bca2b8f18]
       mongos(+0x8D1225) [0x7f8bca290225]
       mongos(_ZN4asio6detail9scheduler10do_run_oneERNS0_11scoped_lockINS0_11posix_mutexEEERNS0_21scheduler_thread_infoERKSt10error_code+0x324) [0x7f8bca641164]
       mongos(_ZN4asio6detail9scheduler3runERSt10error_code+0xD1) [0x7f8bca641361]
       mongos(+0x8D032F) [0x7f8bca28f32f]
       mongos(+0xF82BA0) [0x7f8bca941ba0]
       libpthread.so.0(+0x8184) [0x7f8bc8621184]
       libc.so.6(clone+0x6D) [0x7f8bc834e37d]
      -----  END BACKTRACE  -----
      

        Activity

        Hide
        ramon.fernandez Ramon Fernandez added a comment -

        Thanks for your report Edik Mkoyan, and sorry you're having issues with mongos. We're investigating this issue and will post updates in this ticket.

        Regards,
        Ramón.

        Show
        ramon.fernandez Ramon Fernandez added a comment - Thanks for your report Edik Mkoyan , and sorry you're having issues with mongos . We're investigating this issue and will post updates in this ticket. Regards, Ramón.
        Hide
        schwerin Andy Schwerin added a comment -

        I don't think it matters too much what the stack trace is, but I would look for evidence in the logs of the clock jumping around. I have three hypotheses:

        1. The wall clock jittered backward after the alarm was scheduled and before it executed
        2. The network interface triggered the alarm early
        3. There are two different clocks in use by the network interface, and they round slightly differently.

        I think we should

        1. check for early-trigger type bugs in the network interface used by the ReplicaSetMonitor's TaskExecutor. If we find any, fix them
        2. check to see if two different clocks are in play here. If so, develop a plan to get down to one and implement it
        3. Regardless of the above, replace the invariant with a log message and reschedule the task at the original deadline. If we see an explosion of the log messages, we'll know to look more deeply.
        Show
        schwerin Andy Schwerin added a comment - I don't think it matters too much what the stack trace is, but I would look for evidence in the logs of the clock jumping around. I have three hypotheses: The wall clock jittered backward after the alarm was scheduled and before it executed The network interface triggered the alarm early There are two different clocks in use by the network interface, and they round slightly differently. I think we should check for early-trigger type bugs in the network interface used by the ReplicaSetMonitor's TaskExecutor. If we find any, fix them check to see if two different clocks are in play here. If so, develop a plan to get down to one and implement it Regardless of the above, replace the invariant with a log message and reschedule the task at the original deadline. If we see an explosion of the log messages, we'll know to look more deeply.
        Hide
        edikmkoyan Edik Mkoyan added a comment - - edited

        I have upgraded my cluster to 3.4.2 and it happened today, too.

        2017-02-09T08:33:26.073+0400 I -        [ReplicaSetMonitor-TaskExecutor-0] Invariant failure now() >= when src/mongo/executor/thread_pool_task_executor.cpp 295
        2017-02-09T08:33:26.073+0400 I -        [ReplicaSetMonitor-TaskExecutor-0] 
         
        ***aborting after invariant() failure
         
         
        2017-02-09T08:33:26.105+0400 F -        [ReplicaSetMonitor-TaskExecutor-0] Got signal: 6 (Aborted).
         
         0x7f07c83fa6e1 0x7f07c83f97d9 0x7f07c83f9cbd 0x7f07c64c0330 0x7f07c6121c37 0x7f07c6125028 0x7f07c7c327be 0x7f07c814e8b8 0x7f07c8125b15 0x7f07c84d8294 0x7f07c84d8491 0x7f07c8124c1f 0x7f07c87d8960 0x7f07c64b8184 0x7f07c61e537d
        ----- BEGIN BACKTRACE -----
        {"backtrace":[{"b":"7F07C7856000","o":"BA46E1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7F07C7856000","o":"BA37D9"},{"b":"7F07C7856000","o":"BA3CBD"},{"b":"7F07C64B0000","o":"10330"},{"b":"7F07C60EB000","o":"36C37","s":"gsignal"},{"b":"7F07C60EB000","o":"3A028","s":"abort"},{"b":"7F07C7856000","o":"3DC7BE","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},{"b":"7F07C7856000","o":"8F88B8"},{"b":"7F07C7856000","o":"8CFB15"},{"b":"7F07C7856000","o":"C82294","s":"_ZN4asio6detail9scheduler10do_run_oneERNS0_11scoped_lockINS0_11posix_mutexEEERNS0_21scheduler_thread_infoERKSt10error_code"},{"b":"7F07C7856000","o":"C82491","s":"_ZN4asio6detail9scheduler3runERSt10error_code"},{"b":"7F07C7856000","o":"8CEC1F"},{"b":"7F07C7856000","o":"F82960"},{"b":"7F07C64B0000","o":"8184"},{"b":"7F07C60EB000","o":"FA37D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.2", "gitVersion" : "3f76e40c105fc223b3e5aac3e20dcd026b83b38b", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.19.0-33-generic", "version" : "#38~14.04.1-Ubuntu SMP Fri Nov 6 18:17:28 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "b" : "7F07C7856000", "elfType" : 3, "buildId" : "149E88146384D65E439E13324AF97C149CF35650" }, { "b" : "7FFFEAFE8000", "elfType" : 3, "buildId" : "3AA714083E1D7D719D6C5A7B1A4A8D3649727CB8" }, { "b" : "7F07C73D2000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "48A664AE6B0B4918A3EB0156C6364C4F084232FD" }, { "b" : "7F07C6FF6000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "6B8997EA892A7FF37AC8CAA8F239D595251889BB" }, { "b" : "7F07C6DEE000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "E2A6DD5048A0A051FD61043BDB69D8CC68192AB7" }, { "b" : "7F07C6BEA000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "DA9B8C234D0FE9FD8CAAC8970A7EC1B6C8F6623F" }, { "b" : "7F07C68E4000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "D144258E614900B255A31F3FD2283A878670D5BC" }, { "b" : "7F07C66CE000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7F07C64B0000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "31E9F21AE8C10396171F1E13DA15780986FA696C" }, { "b" : "7F07C60EB000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "CF699A15CAAE64F50311FC4655B86DC39A479789" }, { "b" : "7F07C7631000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D0F537904076D73F29E4A37341F8A449E2EF6CD0" }, { "b" : "7F07C5EE0000", "path" : "/lib/x86_64-linux-gnu/libnss_files.so.2", "elfType" : 3, "buildId" : "A0BDDA36D5AD1390D4DF158D7AD7CC866E534C28" }, { "b" : "7F07C5CDA000", "path" : "/lib/x86_64-linux-gnu/libnss_dns.so.2", "elfType" : 3, "buildId" : "A5721A2FACA0CB423F6C851DF8B7A96C52E94691" }, { "b" : "7F07C5ABF000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "FE705AA7E3872B43EA9E99FAF1897F7F85FC0AC9" } ] }}
         mongos(_ZN5mongo15printStackTraceERSo+0x41) [0x7f07c83fa6e1]
         mongos(+0xBA37D9) [0x7f07c83f97d9]
         mongos(+0xBA3CBD) [0x7f07c83f9cbd]
         libpthread.so.0(+0x10330) [0x7f07c64c0330]
         libc.so.6(gsignal+0x37) [0x7f07c6121c37]
         libc.so.6(abort+0x148) [0x7f07c6125028]
         mongos(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0x0) [0x7f07c7c327be]
         mongos(+0x8F88B8) [0x7f07c814e8b8]
         mongos(+0x8CFB15) [0x7f07c8125b15]
         mongos(_ZN4asio6detail9scheduler10do_run_oneERNS0_11scoped_lockINS0_11posix_mutexEEERNS0_21scheduler_thread_infoERKSt10error_code+0x324) [0x7f07c84d8294]
         mongos(_ZN4asio6detail9scheduler3runERSt10error_code+0xD1) [0x7f07c84d8491]
         mongos(+0x8CEC1F) [0x7f07c8124c1f]
         mongos(+0xF82960) [0x7f07c87d8960]
         libpthread.so.0(+0x8184) [0x7f07c64b8184]
         libc.so.6(clone+0x6D) [0x7f07c61e537d]
        -----  END BACKTRACE  -----
        

        Show
        edikmkoyan Edik Mkoyan added a comment - - edited I have upgraded my cluster to 3.4.2 and it happened today, too. 2017-02-09T08:33:26.073+0400 I - [ReplicaSetMonitor-TaskExecutor-0] Invariant failure now() >= when src/mongo/executor/thread_pool_task_executor.cpp 295 2017-02-09T08:33:26.073+0400 I - [ReplicaSetMonitor-TaskExecutor-0]   ***aborting after invariant() failure     2017-02-09T08:33:26.105+0400 F - [ReplicaSetMonitor-TaskExecutor-0] Got signal: 6 (Aborted).   0x7f07c83fa6e1 0x7f07c83f97d9 0x7f07c83f9cbd 0x7f07c64c0330 0x7f07c6121c37 0x7f07c6125028 0x7f07c7c327be 0x7f07c814e8b8 0x7f07c8125b15 0x7f07c84d8294 0x7f07c84d8491 0x7f07c8124c1f 0x7f07c87d8960 0x7f07c64b8184 0x7f07c61e537d ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"7F07C7856000","o":"BA46E1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7F07C7856000","o":"BA37D9"},{"b":"7F07C7856000","o":"BA3CBD"},{"b":"7F07C64B0000","o":"10330"},{"b":"7F07C60EB000","o":"36C37","s":"gsignal"},{"b":"7F07C60EB000","o":"3A028","s":"abort"},{"b":"7F07C7856000","o":"3DC7BE","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},{"b":"7F07C7856000","o":"8F88B8"},{"b":"7F07C7856000","o":"8CFB15"},{"b":"7F07C7856000","o":"C82294","s":"_ZN4asio6detail9scheduler10do_run_oneERNS0_11scoped_lockINS0_11posix_mutexEEERNS0_21scheduler_thread_infoERKSt10error_code"},{"b":"7F07C7856000","o":"C82491","s":"_ZN4asio6detail9scheduler3runERSt10error_code"},{"b":"7F07C7856000","o":"8CEC1F"},{"b":"7F07C7856000","o":"F82960"},{"b":"7F07C64B0000","o":"8184"},{"b":"7F07C60EB000","o":"FA37D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.2", "gitVersion" : "3f76e40c105fc223b3e5aac3e20dcd026b83b38b", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.19.0-33-generic", "version" : "#38~14.04.1-Ubuntu SMP Fri Nov 6 18:17:28 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "b" : "7F07C7856000", "elfType" : 3, "buildId" : "149E88146384D65E439E13324AF97C149CF35650" }, { "b" : "7FFFEAFE8000", "elfType" : 3, "buildId" : "3AA714083E1D7D719D6C5A7B1A4A8D3649727CB8" }, { "b" : "7F07C73D2000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "48A664AE6B0B4918A3EB0156C6364C4F084232FD" }, { "b" : "7F07C6FF6000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "6B8997EA892A7FF37AC8CAA8F239D595251889BB" }, { "b" : "7F07C6DEE000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "E2A6DD5048A0A051FD61043BDB69D8CC68192AB7" }, { "b" : "7F07C6BEA000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "DA9B8C234D0FE9FD8CAAC8970A7EC1B6C8F6623F" }, { "b" : "7F07C68E4000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "D144258E614900B255A31F3FD2283A878670D5BC" }, { "b" : "7F07C66CE000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7F07C64B0000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "31E9F21AE8C10396171F1E13DA15780986FA696C" }, { "b" : "7F07C60EB000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "CF699A15CAAE64F50311FC4655B86DC39A479789" }, { "b" : "7F07C7631000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D0F537904076D73F29E4A37341F8A449E2EF6CD0" }, { "b" : "7F07C5EE0000", "path" : "/lib/x86_64-linux-gnu/libnss_files.so.2", "elfType" : 3, "buildId" : "A0BDDA36D5AD1390D4DF158D7AD7CC866E534C28" }, { "b" : "7F07C5CDA000", "path" : "/lib/x86_64-linux-gnu/libnss_dns.so.2", "elfType" : 3, "buildId" : "A5721A2FACA0CB423F6C851DF8B7A96C52E94691" }, { "b" : "7F07C5ABF000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "FE705AA7E3872B43EA9E99FAF1897F7F85FC0AC9" } ] }} mongos(_ZN5mongo15printStackTraceERSo+0x41) [0x7f07c83fa6e1] mongos(+0xBA37D9) [0x7f07c83f97d9] mongos(+0xBA3CBD) [0x7f07c83f9cbd] libpthread.so.0(+0x10330) [0x7f07c64c0330] libc.so.6(gsignal+0x37) [0x7f07c6121c37] libc.so.6(abort+0x148) [0x7f07c6125028] mongos(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0x0) [0x7f07c7c327be] mongos(+0x8F88B8) [0x7f07c814e8b8] mongos(+0x8CFB15) [0x7f07c8125b15] mongos(_ZN4asio6detail9scheduler10do_run_oneERNS0_11scoped_lockINS0_11posix_mutexEEERNS0_21scheduler_thread_infoERKSt10error_code+0x324) [0x7f07c84d8294] mongos(_ZN4asio6detail9scheduler3runERSt10error_code+0xD1) [0x7f07c84d8491] mongos(+0x8CEC1F) [0x7f07c8124c1f] mongos(+0xF82960) [0x7f07c87d8960] libpthread.so.0(+0x8184) [0x7f07c64b8184] libc.so.6(clone+0x6D) [0x7f07c61e537d] ----- END BACKTRACE -----
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'jbreams', u'name': u'Jonathan Reams', u'email': u'jbreams@mongodb.com'}

        Message: SERVER-27863 Reschedule early alarms in NetworkInterfaceASIO
        Branch: master
        https://github.com/mongodb/mongo/commit/f725e5137561ba5a521d0f5eb6a60bdeebf34c24

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'jbreams', u'name': u'Jonathan Reams', u'email': u'jbreams@mongodb.com'} Message: SERVER-27863 Reschedule early alarms in NetworkInterfaceASIO Branch: master https://github.com/mongodb/mongo/commit/f725e5137561ba5a521d0f5eb6a60bdeebf34c24
        Hide
        jonathan.reams Jonathan Reams added a comment -

        Edik Mkoyan, there may be a deeper issue here, but we should definitely not crash the server if an alarm fires early, so I've just pushed a fix that logs when an alarm fires early and reschedules it. I'll get it back-ported to 3.4 and it should be in the next release.

        Show
        jonathan.reams Jonathan Reams added a comment - Edik Mkoyan , there may be a deeper issue here, but we should definitely not crash the server if an alarm fires early, so I've just pushed a fix that logs when an alarm fires early and reschedules it. I'll get it back-ported to 3.4 and it should be in the next release.
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'jbreams', u'name': u'Jonathan Reams', u'email': u'jbreams@mongodb.com'}

        Message: SERVER-27863 Reschedule early alarms in NetworkInterfaceASIO

        (cherry picked from commit f725e5137561ba5a521d0f5eb6a60bdeebf34c24)
        Branch: v3.4
        https://github.com/mongodb/mongo/commit/65bc8f25d7e76e23e7380f81ee033537ba9e8866

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'jbreams', u'name': u'Jonathan Reams', u'email': u'jbreams@mongodb.com'} Message: SERVER-27863 Reschedule early alarms in NetworkInterfaceASIO (cherry picked from commit f725e5137561ba5a521d0f5eb6a60bdeebf34c24) Branch: v3.4 https://github.com/mongodb/mongo/commit/65bc8f25d7e76e23e7380f81ee033537ba9e8866

          People

          • Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

                Agile