Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-81784

Pass MSG_WAITALL to send/recv when doing sync networking

    • Service Arch
    • Service Arch 2023-11-13, Service Arch 2023-11-27

      This tells the kernel not to bother waking user-space until the whole message has been sent/received. Without it, the kernel wakes us each time there is a partially complete send/recv just so that we can call send/recv again with an advanced buffer slice. findOne latency for a 1MB document went from ~750μs to ~550μs, so that is a lot of wasted work.

      Unfortunately asio doesn't know to set it, so we will need to bypass asio and do it ourselves.

      While we are at it, we should make the sourceMessage a bit more optimal for small messages as well. Currently we do a recv of 16 bytes to read the header, then do another recv to read the rest of the message. Instead we should allocate a buffer on the stack (maybe 1, 4, or 16KB?) and do a recv into that, only looping until we have the size (in general we won't loop at all). If we got lucky and got a full message on our first try, we can just copy that into a Message and move on without doing a second syscall. If we didn't get a full message then we should to a recv(MSG_WAITALL) for the remainder.

      sinkMessage is simpler and we can just unconditionally use MSG_WAITALL when sync because we always have a full message ready to go.

      Note, this will only show an improvement for large messages, with the impact proportional to the size of the message. So be sure to run benchmarks that involve very large messages.

            Assignee:
            erin.mcnulty@mongodb.com Erin McNulty
            Reporter:
            mathias@mongodb.com Mathias Stearn
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: