A deadlock is induced on the SessionCatalog:
- The operation performing the merging half of the pipeline checks out the Session for that lsid.
- That operation includes a $mergeCursors, which performs getMores on remote hosts, one of which is the same host performing the $mergeCursors.
- That operation will attempt to check out the same session once the getMore includes the lsid - blocking on a mutex in the SessionCatalog.
As a short term fix, we should do the following:
- Only check out the Session if the operation includes a transaction number.
- Ban aggregations with a transaction number on mongos.
As a long term fix, we will investigate either not using getMores over the network for what are really local reads. If that proves difficult, we will have to re-evaluate.