[SERVER-68039] Old pymongo version 3.10.1 on MongoDB v5.0 causes Invariant failure (message.operation() == dbMsg) after connection reset by peer Created: 14/Jul/22  Updated: 29/Oct/23  Resolved: 15/Aug/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 5.0.9
Fix Version/s: 4.4.17, 5.0.12

Type: Bug Priority: Critical - P2
Reporter: Anders Wenhaug Assignee: George Wangensteen
Resolution: Fixed Votes: 2
Labels: crash
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File diagnostic.data.tar.gz     PNG File image-2022-07-20-16-06-43-364.png    
Issue Links:
Backports
Related
is related to SERVER-62147 Exhaust query using the OP_QUERY prot... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.4
Sprint: Service Arch 2022-08-22
Participants:

 Description   

A request over a very slow connection (South America to Northern Europe: RTT 220ms, throughput ~10Mbps) seems to have triggered an invariant failure after the remote end hung up.

Connection 1:

{"t":{"$date":"2022-07-13T13:50:25.648Z"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn66597","msg":"client metadata","attr":{"remote":"10.20.55.1:62396","client":"conn66597","doc":{"driver":{"name":"PyMongo","version":"3.10.1"},"os":{"type":"Windows","name":"Windows 10","architecture":"AMD64","version":"10.0.22000-SP0"},"platform":"CPython 3.9.13.final.0"}}}
{"t":{"$date":"2022-07-13T13:50:26.666Z"},"s":"I",  "c":"ACCESS",   "id":20250,   "ctx":"conn66597","msg":"Authentication succeeded","attr":{"mechanism":"SCRAM-SHA-256","speculative":false,"principalName":"reader","authenticationDatabase":"<DATABASE>","remote":"10.20.55.1:62396","extraInfo":{}}}
{"t":{"$date":"2022-07-13T13:58:09.753Z"},"s":"I",  "c":"NETWORK",  "id":22989,   "ctx":"conn66597","msg":"Error sending response to client. Ending connection from remote","attr":{"error":{"code":6,"codeName":"HostUnreachable","errmsg":"Connection reset by peer"},"remote":"10.20.55.1:62396","connectionId":66597}}
{"t":{"$date":"2022-07-13T13:58:09.756Z"},"s":"F",  "c":"-",        "id":23079,   "ctx":"conn66597","msg":"Invariant failure","attr":{"expr":"message.operation() == dbMsg","file":"src/mongo/rpc/op_msg.cpp","line":133}}
{"t":{"$date":"2022-07-13T13:58:09.757Z"},"s":"F",  "c":"-",        "id":23080,   "ctx":"conn66597","msg":"\n\n***aborting after invariant() failure\n\n"}
{"t":{"$date":"2022-07-13T13:58:09.758Z"},"s":"F",  "c":"CONTROL",  "id":4757800, "ctx":"conn66597","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
{"t":{"$date":"2022-07-13T13:58:09.964Z"},"s":"I",  "c":"CONTROL",  "id":31380,   "ctx":"conn66597","msg":"BACKTRACE","attr":{"bt":{"backtrace":[{"a":"55BB2BF5BE75","b":"55BB2809E000","o":"3EBDE75","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.357","s+":"215"},{"a":"55BB2BF5E909","b":"55BB2809E000","o":"3EC0909","s":"_ZN5mongo15printStackTraceEv","s+":"29"},{"a":"55BB2BF56F46","b":"55BB2809E000","o":"3EB8F46","s":"abruptQuit","s+":"66"},{"a":"7FB3E1A51420","b":"7FB3E1A3D000","o":"14420","s":"funlockfile","s+":"60"},{"a":"7FB3E188E00B","b":"7FB3E184B000","o":"4300B","s":"gsignal","s+":"CB"},{"a":"7FB3E186D859","b":"7FB3E184B000","o":"22859","s":"abort","s+":"12B"},{"a":"55BB29468FAE","b":"55BB2809E000","o":"13CAFAE","s":"_ZN5mongo15invariantFailedEPKcS1_j","s+":"F7"},{"a":"55BB293B05A2","b":"55BB2809E000","o":"13125A2","s":"_ZN5mongo5OpMsg5parseERKNS_7MessageE.cold.429","s+":"644"},{"a":"55BB29670CE6","b":"55BB2809E000","o":"15D2CE6","s":"_ZN5mongo9transport19ServiceStateMachine4Impl23cleanupExhaustResourcesEv","s+":"66"},{"a":"55BB29671545","b":"55BB2809E000","o":"15D3545","s":"_ZN5mongo9transport19ServiceStateMachine4Impl14cleanupSessionERKNS_6StatusE","s+":"95"},{"a":"55BB28F4B908","b":"55BB2809E000","o":"EAD908","s":"_ZN5mongo9transport19ServiceStateMachine4Impl15scheduleNewLoopENS_6StatusE.cold.860","s+":"150"},{"a":"55BB29671AB4","b":"55BB2809E000","o":"15D3AB4","s":"_ZN5mongo14future_details4callIRZNS_9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusEEUlS5_E2_EEDaOT_NS_10StatusWithINS0_8FakeVoidEEE","s+":"34"},{"a":"55BB29671E03","b":"55BB2809E000","o":"15D3E03","s":"_ZN5mongo9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusE","s+":"283"},{"a":"55BB2967222F","b":"55BB2809E000","o":"15D422F","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_9transport19ServiceStateMachine4Impl15scheduleNewLoopES1_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1_","s+":"7F"},{"a":"55BB29D51ED3","b":"55BB2809E000","o":"1CB3ED3","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport15ServiceExecutor8scheduleENS0_IFvNS_6StatusEEEEEUlvE_EEDaOT_EN12SpecificImpl4callEv","s+":"33"},{"a":"55BB2B70DB7B","b":"55BB2809E000","o":"366FB7B","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport26ServiceExecutorSynchronous12scheduleTaskES2_NS4_15ServiceExecutor13ScheduleFlagsEEUlvE0_EEDaOT_EN12SpecificImpl4callEv","s+":"BB"},{"a":"55BB2B711A2C","b":"55BB2809E000","o":"3673A2C","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_25launchServiceWorkerThreadES2_EUlvE2_EEDaOT_EN12SpecificImpl4callEv","s+":"5C"},{"a":"55BB2B711A9C","b":"55BB2809E000","o":"3673A9C","s":"_ZN5mongo12_GLOBAL__N_17runFuncEPv","s+":"1C"},{"a":"7FB3E1A45609","b":"7FB3E1A3D000","o":"8609","s":"start_thread","s+":"D9"},{"a":"7FB3E196A133","b":"7FB3E184B000","o":"11F133","s":"clone","s+":"43"}],"processInfo":{"mongodbVersion":"5.0.9","gitVersion":"6f7dae919422dcd7f4892c10ff20cdc721ad00e6","compiledModules":[],"uname":{"sysname":"Linux","release":"5.4.0-121-generic","version":"#137-Ubuntu SMP Wed Jun 15 13:33:07 UTC 2022","machine":"x86_64"},"somap":[{"b":"55BB2809E000","elfType":3,"buildId":"EEB6C6E4767A2FF4A2F77AE4A23FCBFE4981C02D"},{"b":"7FB3E1A3D000","path":"/lib/x86_64-linux-gnu/libpthread.so.0","elfType":3,"buildId":"7B4536F41CDAA5888408E82D0836E33DCF436466"},{"b":"7FB3E184B000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"1878E6B475720C7C51969E69AB2D276FAE6D1DEE"}]}}}}
{"t":{"$date":"2022-07-13T13:58:09.965Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn66597","msg":"Frame","attr":{"frame":{"a":"55BB2BF5BE75","b":"55BB2809E000","o":"3EBDE75","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.357","s+":"215"}}}
{"t":{"$date":"2022-07-13T13:58:09.965Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn66597","msg":"Frame","attr":{"frame":{"a":"55BB2BF5E909","b":"55BB2809E000","o":"3EC0909","s":"_ZN5mongo15printStackTraceEv","s+":"29"}}}
{"t":{"$date":"2022-07-13T13:58:09.965Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn66597","msg":"Frame","attr":{"frame":{"a":"55BB2BF56F46","b":"55BB2809E000","o":"3EB8F46","s":"abruptQuit","s+":"66"}}}
{"t":{"$date":"2022-07-13T13:58:09.965Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn66597","msg":"Frame","attr":{"frame":{"a":"7FB3E1A51420","b":"7FB3E1A3D000","o":"14420","s":"funlockfile","s+":"60"}}}
{"t":{"$date":"2022-07-13T13:58:09.965Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn66597","msg":"Frame","attr":{"frame":{"a":"7FB3E188E00B","b":"7FB3E184B000","o":"4300B","s":"gsignal","s+":"CB"}}}
{"t":{"$date":"2022-07-13T13:58:09.965Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn66597","msg":"Frame","attr":{"frame":{"a":"7FB3E186D859","b":"7FB3E184B000","o":"22859","s":"abort","s+":"12B"}}}
{"t":{"$date":"2022-07-13T13:58:09.965Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn66597","msg":"Frame","attr":{"frame":{"a":"55BB29468FAE","b":"55BB2809E000","o":"13CAFAE","s":"_ZN5mongo15invariantFailedEPKcS1_j","s+":"F7"}}}
{"t":{"$date":"2022-07-13T13:58:09.965Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn66597","msg":"Frame","attr":{"frame":{"a":"55BB293B05A2","b":"55BB2809E000","o":"13125A2","s":"_ZN5mongo5OpMsg5parseERKNS_7MessageE.cold.429","s+":"644"}}}
{"t":{"$date":"2022-07-13T13:58:09.965Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn66597","msg":"Frame","attr":{"frame":{"a":"55BB29670CE6","b":"55BB2809E000","o":"15D2CE6","s":"_ZN5mongo9transport19ServiceStateMachine4Impl23cleanupExhaustResourcesEv","s+":"66"}}}
{"t":{"$date":"2022-07-13T13:58:09.965Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn66597","msg":"Frame","attr":{"frame":{"a":"55BB29671545","b":"55BB2809E000","o":"15D3545","s":"_ZN5mongo9transport19ServiceStateMachine4Impl14cleanupSessionERKNS_6StatusE","s+":"95"}}}
{"t":{"$date":"2022-07-13T13:58:09.965Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn66597","msg":"Frame","attr":{"frame":{"a":"55BB28F4B908","b":"55BB2809E000","o":"EAD908","s":"_ZN5mongo9transport19ServiceStateMachine4Impl15scheduleNewLoopENS_6StatusE.cold.860","s+":"150"}}}
{"t":{"$date":"2022-07-13T13:58:09.965Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn66597","msg":"Frame","attr":{"frame":{"a":"55BB29671AB4","b":"55BB2809E000","o":"15D3AB4","s":"_ZN5mongo14future_details4callIRZNS_9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusEEUlS5_E2_EEDaOT_NS_10StatusWithINS0_8FakeVoidEEE","s+":"34"}}}
{"t":{"$date":"2022-07-13T13:58:09.965Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn66597","msg":"Frame","attr":{"frame":{"a":"55BB29671E03","b":"55BB2809E000","o":"15D3E03","s":"_ZN5mongo9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusE","s+":"283"}}}
{"t":{"$date":"2022-07-13T13:58:09.965Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn66597","msg":"Frame","attr":{"frame":{"a":"55BB2967222F","b":"55BB2809E000","o":"15D422F","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_9transport19ServiceStateMachine4Impl15scheduleNewLoopES1_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1_","s+":"7F"}}}
{"t":{"$date":"2022-07-13T13:58:09.965Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn66597","msg":"Frame","attr":{"frame":{"a":"55BB29D51ED3","b":"55BB2809E000","o":"1CB3ED3","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport15ServiceExecutor8scheduleENS0_IFvNS_6StatusEEEEEUlvE_EEDaOT_EN12SpecificImpl4callEv","s+":"33"}}}
{"t":{"$date":"2022-07-13T13:58:09.965Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn66597","msg":"Frame","attr":{"frame":{"a":"55BB2B70DB7B","b":"55BB2809E000","o":"366FB7B","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport26ServiceExecutorSynchronous12scheduleTaskES2_NS4_15ServiceExecutor13ScheduleFlagsEEUlvE0_EEDaOT_EN12SpecificImpl4callEv","s+":"BB"}}}
{"t":{"$date":"2022-07-13T13:58:09.965Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn66597","msg":"Frame","attr":{"frame":{"a":"55BB2B711A2C","b":"55BB2809E000","o":"3673A2C","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_25launchServiceWorkerThreadES2_EUlvE2_EEDaOT_EN12SpecificImpl4callEv","s+":"5C"}}}
{"t":{"$date":"2022-07-13T13:58:09.965Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn66597","msg":"Frame","attr":{"frame":{"a":"55BB2B711A9C","b":"55BB2809E000","o":"3673A9C","s":"_ZN5mongo12_GLOBAL__N_17runFuncEPv","s+":"1C"}}}
{"t":{"$date":"2022-07-13T13:58:09.965Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn66597","msg":"Frame","attr":{"frame":{"a":"7FB3E1A45609","b":"7FB3E1A3D000","o":"8609","s":"start_thread","s+":"D9"}}}
{"t":{"$date":"2022-07-13T13:58:09.965Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn66597","msg":"Frame","attr":{"frame":{"a":"7FB3E196A133","b":"7FB3E184B000","o":"11F133","s":"clone","s+":"43"}}},

Connection 2:

{"t":{"$date":"2022-07-13T13:50:29.255Z"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn66604","msg":"client metadata","attr":{"remote":"10.20.55.1:55624","client":"conn66604","doc":{"driver":{"name":"PyMongo","version":"3.10.1"},"os":{"type":"Windows","name":"Windows 10","architecture":"AMD64","version":"10.0.22000-SP0"},"platform":"CPython 3.9.13.final.0"}}}
{"t":{"$date":"2022-07-13T13:50:30.238Z"},"s":"I",  "c":"ACCESS",   "id":20250,   "ctx":"conn66604","msg":"Authentication succeeded","attr":{"mechanism":"SCRAM-SHA-256","speculative":false,"principalName":"reader","authenticationDatabase":"<DATABASE>","remote":"10.20.55.1:55624","extraInfo":{}}}
{"t":{"$date":"2022-07-13T13:58:09.753Z"},"s":"I",  "c":"NETWORK",  "id":22989,   "ctx":"conn66604","msg":"Error sending response to client. Ending connection from remote","attr":{"error":{"code":6,"codeName":"HostUnreachable","errmsg":"Connection reset by peer"},"remote":"10.20.55.1:55624","connectionId":66604}}
{"t":{"$date":"2022-07-13T13:58:09.757Z"},"s":"F",  "c":"-",        "id":23079,   "ctx":"conn66604","msg":"Invariant failure","attr":{"expr":"message.operation() == dbMsg","file":"src/mongo/rpc/op_msg.cpp","line":133}}
{"t":{"$date":"2022-07-13T13:58:09.757Z"},"s":"F",  "c":"-",        "id":23080,   "ctx":"conn66604","msg":"\n\n***aborting after invariant() failure\n\n"}
{"t":{"$date":"2022-07-13T13:58:09.966Z"},"s":"F",  "c":"CONTROL",  "id":4757800, "ctx":"conn66604","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}



 Comments   
Comment by Jason Chan [ 16/Aug/22 ]

Author:

{'name': 'George Wangensteen', 'email': 'george.wangensteen@mongodb.com', 'username': 'gewa24'}

Message: SERVER-68039 Support cleaning up dbQuery/dbGetMore exhaust cursors
Branch: v4.4
https://github.com/mongodb/mongo/commit/954ae7c6e4d33766170f767a0ad4807b8a7b215e

Comment by Jason Chan [ 15/Aug/22 ]

Author:

{'name': 'George Wangensteen', 'email': 'george.wangensteen@mongodb.com', 'username': 'gewa24'}

Message: SERVER-68039 Support cleaning up dbQuery/dbGetMore exhaust cursors
Branch: v5.0
https://github.com/mongodb/mongo/commit/effae26eda3c0f4f71d1e7b48bb1291497dab25d

Comment by Chris Kelly [ 09/Aug/22 ]

Anders,

I was unable to reproduce your issue because I am using a newer version of pymongo (3.12.3) that did not cause the issues in these environments:

  • Windows 10 running your docker reproduction
  • WSL2 running Ubuntu 20.04 LTS with docker reproduction or manually installed reproduction (with m)

In all cases, inserts were successful, there was no crash, and no log output to indicate the invariant failure.

However, with version 3.10.1 of pymongo (which you are using), I reproduce your issue. I would just upgrade to a newer version of that so you no longer have the problem.
PyMongo 3.10.1 only supports MongoDB 2.6, 3.0, 3.2, 3.4, 3.6, 4.0 and 4.2.

EDIT: This is probably going to warrant additional investigation to figure out why it causes a crash though, since this really shouldn't be happening. An error should be presented on the driver side instead (like pymongo.errors.OperationFailure: database error: OP_QUERY is no longer supported)

 Thank you for taking the time to create not one, but two helpful and concise reproductions for your issue though, as well as supplying the logs!

Regards,
Christopher

Comment by Anders Wenhaug [ 21/Jul/22 ]

Managed to reproduce using Docker. Just run this Python code, it will run mongo:5.0.9 in a new docker container (without volume mounts, only exposing a port for communication), then kill the MongoDB process using the bug.

import sys
import uuid
import subprocess
import pymongo as pm
 
print(f"{sys.version_info}")  # sys.version_info(major=3, minor=9, micro=13, releaselevel='final', serial=0)
print(f"{pm.__version__}")  # 3.10.1
 
 
mongo_port = 27017
n_docs = 10 ** 3
 
 
def spawn_mongodb():
    return subprocess.Popen(["docker", "run", "-p", f"{mongo_port}:27017", "mongo:5.0.9"])
 
 
def query_and_kill():
    for doc in co.find({}).add_option(pm.CursorType.EXHAUST):
        print(doc["i"])
        if doc["i"] > n_docs // 2:
            return
 
 
def insert_data():
    bs = bytes([123] * 10 ** 5)
 
    docs = []
    for i in range(n_docs):
        docs.append({"i": i, "bytes": bs})
 
    co.insert_many(docs)
 
 
def crash_mongodb_5_0_9():
    insert_data()
    query_and_kill()
 
 
cl = pm.MongoClient("localhost", mongo_port)
db = cl[str(uuid.uuid4())]
co = db["crash-co"]
 
 
mongo = spawn_mongodb()
crash_mongodb_5_0_9()
 
print(f"MongoDB output:")
print(mongo.stdout) 

docker --version
# Docker version 20.10.17, build 100c70180f 

Anders

Comment by Anders Wenhaug [ 20/Jul/22 ]

Sure! I managed to produce a minimal example for how to crash MongoDB (which is actually trivial, although I wasn't able to reproduce when running MongoDB in a container locally):

  1. New Ubuntu 20.04 LTS VM in GCP (t2d-standard-4 - AMD Milan as that is the same CPU family we have on our bare metal servers where the problem initially occurred, however I have no idea if the CPU type is relevant or not). RTT to this GCP DC is around 33ms.
  2. Follow guide on how to install MongoDB 5 (https://www.mongodb.com/docs/v5.0/tutorial/install-mongodb-on-ubuntu/)
  3. Use default mongod.conf, except for setting bindIp to 0.0.0.0
  4. Run the python code below:

import os
import sys
import pymongo as pm
 
 
print(f"{sys.version_info}")  # sys.version_info(major=3, minor=9, micro=13, releaselevel='final', serial=0)
print(f"{pm.__version__}")    # 3.10.1
 
 
cl = pm.MongoClient("<ip-of-instance>", 27017)
db = cl["crash-db"]
co = db["crash-co"]
co.drop()
 
 
n_docs = 10 ** 3
 
 
def query_and_kill():
    for doc in co.find({}).add_option(pm.CursorType.EXHAUST):
        print(doc["i"])
        if doc["i"] > n_docs // 2:
            os.kill(os.getpid(), 9)
 
 
def insert_data():
    bs = bytes([123]*10**5)
 
    docs = []
    for i in range(n_docs):
        docs.append({"i": i, "bytes": bs})
 
    co.insert_many(docs)
 
 
def crash_mongodb_5_0_9():
    insert_data()
    query_and_kill()
 
 
crash_mongodb_5_0_9() 

Log is the same as the others I posted it seems:

{"t":{"$date":"2022-07-20T13:52:46.329+00:00"},"s":"W",  "c":"COMMAND",  "id":5578800, "ctx":"conn2","msg":"Deprecated operation requested. For more details see https://dochub.mongodb.org/core/legacy-opcode-compatibility","attr":{"op":"
query","clientInfo":{"driver":{"name":"PyMongo","version":"3.10.1"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"5.18.12-arch1-1"},"platform":"CPython 3.9.13.final.0"}}}
 
{"t":{"$date":"2022-07-20T13:52:47.312+00:00"},"s":"I",  "c":"NETWORK",  "id":22989,   "ctx":"conn2","msg":"Error sending response to client. Ending connection from remote","attr":{"error":{"code":6,"codeName":"HostUnreachable","errmsg"
:"Connection reset by peer"},"remote":"193.69.209.190:47514","connectionId":2}}
 
{"t":{"$date":"2022-07-20T13:52:47.312+00:00"},"s":"F",  "c":"-",        "id":23079,   "ctx":"conn2","msg":"Invariant failure","attr":{"expr":"message.operation() == dbMsg","file":"src/mongo/rpc/op_msg.cpp","line":133}}
 
{"t":{"$date":"2022-07-20T13:52:47.313+00:00"},"s":"F",  "c":"-",        "id":23080,   "ctx":"conn2","msg":"\n\n***aborting after invariant() failure\n\n"}
 
{"t":{"$date":"2022-07-20T13:52:47.313+00:00"},"s":"F",  "c":"CONTROL",  "id":4757800, "ctx":"conn2","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
 
{"t":{"$date":"2022-07-20T13:52:47.313+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn1","msg":"Connection ended","attr":{"remote":"193.69.209.190:47500","uuid":"7ba65e32-5899-4da3-ae03-50d36de1f067","connectionId":1,"connec
tionCount":1}}
 
{"t":{"$date":"2022-07-20T13:52:47.362+00:00"},"s":"I",  "c":"CONTROL",  "id":31380,   "ctx":"conn2","msg":"BACKTRACE","attr":{"bt":{"backtrace":[{"a":"555F514A4E75","b":"555F4D5E7000","o":"3EBDE75","s":"_ZN5mongo18stack_trace_detail12_
GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.357","s+":"215"},{"a":"555F514A7909","b":"555F4D5E7000","o":"3EC0909","s":"_ZN5mongo15printStackTraceEv","s+":"29"},{"a":"555F5149FF46","b":"555F4D5E7000","
o":"3EB8F46","s":"abruptQuit","s+":"66"},{"a":"7FD448EA0420","b":"7FD448E8C000","o":"14420","s":"funlockfile","s+":"60"},{"a":"7FD448CDD00B","b":"7FD448C9A000","o":"4300B","s":"gsignal","s+":"CB"},{"a":"7FD448CBC859","b":"7FD448C9A000",
"o":"22859","s":"abort","s+":"12B"},{"a":"555F4E9B1FAE","b":"555F4D5E7000","o":"13CAFAE","s":"_ZN5mongo15invariantFailedEPKcS1_j","s+":"F7"},{"a":"555F4E8F95A2","b":"555F4D5E7000","o":"13125A2","s":"_ZN5mongo5OpMsg5parseERKNS_7MessageE.
cold.429","s+":"644"},{"a":"555F4EBB9CE6","b":"555F4D5E7000","o":"15D2CE6","s":"_ZN5mongo9transport19ServiceStateMachine4Impl23cleanupExhaustResourcesEv","s+":"66"},{"a":"555F4EBBA545","b":"555F4D5E7000","o":"15D3545","s":"_ZN5mongo9tra
nsport19ServiceStateMachine4Impl14cleanupSessionERKNS_6StatusE","s+":"95"},{"a":"555F4E494908","b":"555F4D5E7000","o":"EAD908","s":"_ZN5mongo9transport19ServiceStateMachine4Impl15scheduleNewLoopENS_6StatusE.cold.860","s+":"150"},{"a":"5
55F4EBBAAB4","b":"555F4D5E7000","o":"15D3AB4","s":"_ZN5mongo14future_details4callIRZNS_9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusEEUlS5_E2_EEDaOT_NS_10StatusWithINS0_8FakeVoidEEE","s+":"34"},{"a":"555F4EBBAE03","b":
"555F4D5E7000","o":"15D3E03","s":"_ZN5mongo9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusE","s+":"283"},{"a":"555F4EBBB22F","b":"555F4D5E7000","o":"15D422F","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_
9transport19ServiceStateMachine4Impl15scheduleNewLoopES1_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1_","s+":"7F"},{"a":"555F4F29AED3","b":"555F4D5E7000","o":"1CB3ED3","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport15ServiceE
xecutor8scheduleENS0_IFvNS_6StatusEEEEEUlvE_EEDaOT_EN12SpecificImpl4callEv","s+":"33"},{"a":"555F50C56B7B","b":"555F4D5E7000","o":"366FB7B","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport26ServiceExecutorSynchronous12sche
duleTaskES2_NS4_15ServiceExecutor13ScheduleFlagsEEUlvE0_EEDaOT_EN12SpecificImpl4callEv","s+":"BB"},{"a":"555F50C5AA2C","b":"555F4D5E7000","o":"3673A2C","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_25launchServiceWorkerThreadES2_E
UlvE2_EEDaOT_EN12SpecificImpl4callEv","s+":"5C"},{"a":"555F50C5AA9C","b":"555F4D5E7000","o":"3673A9C","s":"_ZN5mongo12_GLOBAL__N_17runFuncEPv","s+":"1C"},{"a":"7FD448E94609","b":"7FD448E8C000","o":"8609","s":"start_thread","s+":"D9"},{"
a":"7FD448DB9133","b":"7FD448C9A000","o":"11F133","s":"clone","s+":"43"}],"processInfo":{"mongodbVersion":"5.0.9","gitVersion":"6f7dae919422dcd7f4892c10ff20cdc721ad00e6","compiledModules":[],"uname":{"sysname":"Linux","release":"5.15.0-
1013-gcp","version":"#18~20.04.1-Ubuntu SMP Sun Jul 3 08:20:07 UTC 2022","machine":"x86_64"},"somap":[{"b":"555F4D5E7000","elfType":3,"buildId":"EEB6C6E4767A2FF4A2F77AE4A23FCBFE4981C02D"},{"b":"7FD448E8C000","path":"/lib/x86_64-linux-gn
u/libpthread.so.0","elfType":3,"buildId":"7B4536F41CDAA5888408E82D0836E33DCF436466"},{"b":"7FD448C9A000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"1878E6B475720C7C51969E69AB2D276FAE6D1DEE"}]}}}}
 
{"t":{"$date":"2022-07-20T13:52:47.362+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn2","msg":"Frame","attr":{"frame":{"a":"555F514A4E75","b":"555F4D5E7000","o":"3EBDE75","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119pr
intStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.357","s+":"215"}}}
 
{"t":{"$date":"2022-07-20T13:52:47.362+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn2","msg":"Frame","attr":{"frame":{"a":"555F514A7909","b":"555F4D5E7000","o":"3EC0909","s":"_ZN5mongo15printStackTraceEv","s+":"29"}}}
 
{"t":{"$date":"2022-07-20T13:52:47.362+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn2","msg":"Frame","attr":{"frame":{"a":"555F5149FF46","b":"555F4D5E7000","o":"3EB8F46","s":"abruptQuit","s+":"66"}}}
 
{"t":{"$date":"2022-07-20T13:52:47.362+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn2","msg":"Frame","attr":{"frame":{"a":"7FD448EA0420","b":"7FD448E8C000","o":"14420","s":"funlockfile","s+":"60"}}}
 
{"t":{"$date":"2022-07-20T13:52:47.362+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn2","msg":"Frame","attr":{"frame":{"a":"7FD448CDD00B","b":"7FD448C9A000","o":"4300B","s":"gsignal","s+":"CB"}}}
 
{"t":{"$date":"2022-07-20T13:52:47.362+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn2","msg":"Frame","attr":{"frame":{"a":"7FD448CBC859","b":"7FD448C9A000","o":"22859","s":"abort","s+":"12B"}}}
 
{"t":{"$date":"2022-07-20T13:52:47.362+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn2","msg":"Frame","attr":{"frame":{"a":"555F4E9B1FAE","b":"555F4D5E7000","o":"13CAFAE","s":"_ZN5mongo15invariantFailedEPKcS1_j","s+":"F7"}}
}
 
{"t":{"$date":"2022-07-20T13:52:47.362+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn2","msg":"Frame","attr":{"frame":{"a":"555F4E8F95A2","b":"555F4D5E7000","o":"13125A2","s":"_ZN5mongo5OpMsg5parseERKNS_7MessageE.cold.429",
"s+":"644"}}}
 
{"t":{"$date":"2022-07-20T13:52:47.362+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn2","msg":"Frame","attr":{"frame":{"a":"555F4EBB9CE6","b":"555F4D5E7000","o":"15D2CE6","s":"_ZN5mongo9transport19ServiceStateMachine4Impl23
cleanupExhaustResourcesEv","s+":"66"}}}
 
{"t":{"$date":"2022-07-20T13:52:47.362+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn2","msg":"Frame","attr":{"frame":{"a":"555F4EBBA545","b":"555F4D5E7000","o":"15D3545","s":"_ZN5mongo9transport19ServiceStateMachine4Impl14
cleanupSessionERKNS_6StatusE","s+":"95"}}}
 
{"t":{"$date":"2022-07-20T13:52:47.362+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn2","msg":"Frame","attr":{"frame":{"a":"555F4E494908","b":"555F4D5E7000","o":"EAD908","s":"_ZN5mongo9transport19ServiceStateMachine4Impl15s
cheduleNewLoopENS_6StatusE.cold.860","s+":"150"}}}
 
{"t":{"$date":"2022-07-20T13:52:47.362+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn2","msg":"Frame","attr":{"frame":{"a":"555F4EBBAAB4","b":"555F4D5E7000","o":"15D3AB4","s":"_ZN5mongo14future_details4callIRZNS_9transport1
9ServiceStateMachine4Impl12startNewLoopERKNS_6StatusEEUlS5_E2_EEDaOT_NS_10StatusWithINS0_8FakeVoidEEE","s+":"34"}}}
 
{"t":{"$date":"2022-07-20T13:52:47.362+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn2","msg":"Frame","attr":{"frame":{"a":"555F4EBBAE03","b":"555F4D5E7000","o":"15D3E03","s":"_ZN5mongo9transport19ServiceStateMachine4Impl12
startNewLoopERKNS_6StatusE","s+":"283"}}}
 
{"t":{"$date":"2022-07-20T13:52:47.362+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn2","msg":"Frame","attr":{"frame":{"a":"555F4EBBB22F","b":"555F4D5E7000","o":"15D422F","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8mak
eImplIZNS_9transport19ServiceStateMachine4Impl15scheduleNewLoopES1_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1_","s+":"7F"}}}
 
{"t":{"$date":"2022-07-20T13:52:47.362+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn2","msg":"Frame","attr":{"frame":{"a":"555F4F29AED3","b":"555F4D5E7000","o":"1CB3ED3","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_
9transport15ServiceExecutor8scheduleENS0_IFvNS_6StatusEEEEEUlvE_EEDaOT_EN12SpecificImpl4callEv","s+":"33"}}}
 
{"t":{"$date":"2022-07-20T13:52:47.362+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn2","msg":"Frame","attr":{"frame":{"a":"555F50C56B7B","b":"555F4D5E7000","o":"366FB7B","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_
9transport26ServiceExecutorSynchronous12scheduleTaskES2_NS4_15ServiceExecutor13ScheduleFlagsEEUlvE0_EEDaOT_EN12SpecificImpl4callEv","s+":"BB"}}}
 
{"t":{"$date":"2022-07-20T13:52:47.362+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn2","msg":"Frame","attr":{"frame":{"a":"555F50C5AA2C","b":"555F4D5E7000","o":"3673A2C","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_
25launchServiceWorkerThreadES2_EUlvE2_EEDaOT_EN12SpecificImpl4callEv","s+":"5C"}}}
 
{"t":{"$date":"2022-07-20T13:52:47.362+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn2","msg":"Frame","attr":{"frame":{"a":"555F50C5AA9C","b":"555F4D5E7000","o":"3673A9C","s":"_ZN5mongo12_GLOBAL__N_17runFuncEPv","s+":"1C"}}
}
 
{"t":{"$date":"2022-07-20T13:52:47.362+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn2","msg":"Frame","attr":{"frame":{"a":"7FD448E94609","b":"7FD448E8C000","o":"8609","s":"start_thread","s+":"D9"}}}
 
{"t":{"$date":"2022-07-20T13:52:47.362+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn2","msg":"Frame","attr":{"frame":{"a":"7FD448DB9133","b":"7FD448C9A000","o":"11F133","s":"clone","s+":"43"}}} 

diagnostic.data.tar.gz

See attached archive of diagnostic.data, however this should be very easy to reproduce on your end.

MongoDB package info:

Let me know if I can be of additional help!

Anders

Comment by Chris Kelly [ 20/Jul/22 ]

Hi Anders,

Thanks for your report. Just for context, can you describe the steps you are taking to reproduce this problem more specifically? It sounds like the connection speed isn't related. If you're running any specific queries before this, that would be helpful to note as well.

For each node in the replica set spanning a time period that includes the incident, would you please archive (tar or zip) and upload to the ticket:

  • the mongod log
  • the $dbpath/diagnostic.data directory (the contents are described here)

Thank you!
Christopher

Comment by Anders Wenhaug [ 14/Jul/22 ]

I realized I didn't provide information about the host:
OS: Ubuntu 20.04.4 LTS
Linux: 5.4.0-121-generic

Comment by Anders Wenhaug [ 14/Jul/22 ]

Seems like the issue is related to using exhaust-cursors. I can reproduce the issue by stopping the Python process (using pymongo) while it is running a query using exhaust-cursors, but I am not able to reproduce when running without the exhaust option.

Comment by Anders Wenhaug [ 14/Jul/22 ]

Two more instances of the same issue, but this time over a much faster connection (1ms, 1Gbps).

{"t":{"$date":"2022-07-14T08:08:09.539Z"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn67114","msg":"client metadata","attr":{"remote":"10.20.55.1:31786","client":"conn67114","doc":{"driver":{"name":"PyMongo","version":"3.10.1"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"5.11.0-49-generic"},"platform":"CPython 3.9.13.final.0"}}}
{"t":{"$date":"2022-07-14T08:08:09.570Z"},"s":"I",  "c":"ACCESS",   "id":20250,   "ctx":"conn67114","msg":"Authentication succeeded","attr":{"mechanism":"SCRAM-SHA-256","speculative":false,"principalName":"writer","authenticationDatabase":"<DATABASE>","remote":"10.20.55.1:31786","extraInfo":{}}}
{"t":{"$date":"2022-07-14T08:24:14.459Z"},"s":"I",  "c":"NETWORK",  "id":22989,   "ctx":"conn67114","msg":"Error sending response to client. Ending connection from remote","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"Connection timed out"},"remote":"10.20.55.1:31786","connectionId":67114}}
{"t":{"$date":"2022-07-14T08:24:14.464Z"},"s":"F",  "c":"-",        "id":23079,   "ctx":"conn67114","msg":"Invariant failure","attr":{"expr":"message.operation() == dbMsg","file":"src/mongo/rpc/op_msg.cpp","line":133}}
{"t":{"$date":"2022-07-14T08:24:14.466Z"},"s":"F",  "c":"-",        "id":23080,   "ctx":"conn67114","msg":"\n\n***aborting after invariant() failure\n\n"}
{"t":{"$date":"2022-07-14T08:24:14.467Z"},"s":"F",  "c":"CONTROL",  "id":4757800, "ctx":"conn67114","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
{"t":{"$date":"2022-07-14T08:24:14.655Z"},"s":"I",  "c":"CONTROL",  "id":31380,   "ctx":"conn67114","msg":"BACKTRACE","attr":{"bt":{"backtrace":[{"a":"556A5435DE75","b":"556A504A0000","o":"3EBDE75","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.357","s+":"215"},{"a":"556A54360909","b":"556A504A0000","o":"3EC0909","s":"_ZN5mongo15printStackTraceEv","s+":"29"},{"a":"556A54358F46","b":"556A504A0000","o":"3EB8F46","s":"abruptQuit","s+":"66"},{"a":"7F4A2990C420","b":"7F4A298F8000","o":"14420","s":"funlockfile","s+":"60"},{"a":"7F4A2974900B","b":"7F4A29706000","o":"4300B","s":"gsignal","s+":"CB"},{"a":"7F4A29728859","b":"7F4A29706000","o":"22859","s":"abort","s+":"12B"},{"a":"556A5186AFAE","b":"556A504A0000","o":"13CAFAE","s":"_ZN5mongo15invariantFailedEPKcS1_j","s+":"F7"},{"a":"556A517B25A2","b":"556A504A0000","o":"13125A2","s":"_ZN5mongo5OpMsg5parseERKNS_7MessageE.cold.429","s+":"644"},{"a":"556A51A72CE6","b":"556A504A0000","o":"15D2CE6","s":"_ZN5mongo9transport19ServiceStateMachine4Impl23cleanupExhaustResourcesEv","s+":"66"},{"a":"556A51A73545","b":"556A504A0000","o":"15D3545","s":"_ZN5mongo9transport19ServiceStateMachine4Impl14cleanupSessionERKNS_6StatusE","s+":"95"},{"a":"556A5134D908","b":"556A504A0000","o":"EAD908","s":"_ZN5mongo9transport19ServiceStateMachine4Impl15scheduleNewLoopENS_6StatusE.cold.860","s+":"150"},{"a":"556A51A73AB4","b":"556A504A0000","o":"15D3AB4","s":"_ZN5mongo14future_details4callIRZNS_9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusEEUlS5_E2_EEDaOT_NS_10StatusWithINS0_8FakeVoidEEE","s+":"34"},{"a":"556A51A73E03","b":"556A504A0000","o":"15D3E03","s":"_ZN5mongo9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusE","s+":"283"},{"a":"556A51A7422F","b":"556A504A0000","o":"15D422F","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_9transport19ServiceStateMachine4Impl15scheduleNewLoopES1_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1_","s+":"7F"},{"a":"556A52153ED3","b":"556A504A0000","o":"1CB3ED3","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport15ServiceExecutor8scheduleENS0_IFvNS_6StatusEEEEEUlvE_EEDaOT_EN12SpecificImpl4callEv","s+":"33"},{"a":"556A53B0FB7B","b":"556A504A0000","o":"366FB7B","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport26ServiceExecutorSynchronous12scheduleTaskES2_NS4_15ServiceExecutor13ScheduleFlagsEEUlvE0_EEDaOT_EN12SpecificImpl4callEv","s+":"BB"},{"a":"556A53B13A2C","b":"556A504A0000","o":"3673A2C","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_25launchServiceWorkerThreadES2_EUlvE2_EEDaOT_EN12SpecificImpl4callEv","s+":"5C"},{"a":"556A53B13A9C","b":"556A504A0000","o":"3673A9C","s":"_ZN5mongo12_GLOBAL__N_17runFuncEPv","s+":"1C"},{"a":"7F4A29900609","b":"7F4A298F8000","o":"8609","s":"start_thread","s+":"D9"},{"a":"7F4A29825133","b":"7F4A29706000","o":"11F133","s":"clone","s+":"43"}],"processInfo":{"mongodbVersion":"5.0.9","gitVersion":"6f7dae919422dcd7f4892c10ff20cdc721ad00e6","compiledModules":[],"uname":{"sysname":"Linux","release":"5.4.0-121-generic","version":"#137-Ubuntu SMP Wed Jun 15 13:33:07 UTC 2022","machine":"x86_64"},"somap":[{"b":"556A504A0000","elfType":3,"buildId":"EEB6C6E4767A2FF4A2F77AE4A23FCBFE4981C02D"},{"b":"7F4A298F8000","path":"/lib/x86_64-linux-gnu/libpthread.so.0","elfType":3,"buildId":"7B4536F41CDAA5888408E82D0836E33DCF436466"},{"b":"7F4A29706000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"1878E6B475720C7C51969E69AB2D276FAE6D1DEE"}]}}}}
{"t":{"$date":"2022-07-14T08:24:14.656Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn67114","msg":"Frame","attr":{"frame":{"a":"556A5435DE75","b":"556A504A0000","o":"3EBDE75","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.357","s+":"215"}}}
{"t":{"$date":"2022-07-14T08:24:14.656Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn67114","msg":"Frame","attr":{"frame":{"a":"556A54360909","b":"556A504A0000","o":"3EC0909","s":"_ZN5mongo15printStackTraceEv","s+":"29"}}}
{"t":{"$date":"2022-07-14T08:24:14.656Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn67114","msg":"Frame","attr":{"frame":{"a":"556A54358F46","b":"556A504A0000","o":"3EB8F46","s":"abruptQuit","s+":"66"}}}
{"t":{"$date":"2022-07-14T08:24:14.656Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn67114","msg":"Frame","attr":{"frame":{"a":"7F4A2990C420","b":"7F4A298F8000","o":"14420","s":"funlockfile","s+":"60"}}}
{"t":{"$date":"2022-07-14T08:24:14.656Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn67114","msg":"Frame","attr":{"frame":{"a":"7F4A2974900B","b":"7F4A29706000","o":"4300B","s":"gsignal","s+":"CB"}}}
{"t":{"$date":"2022-07-14T08:24:14.656Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn67114","msg":"Frame","attr":{"frame":{"a":"7F4A29728859","b":"7F4A29706000","o":"22859","s":"abort","s+":"12B"}}}
{"t":{"$date":"2022-07-14T08:24:14.656Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn67114","msg":"Frame","attr":{"frame":{"a":"556A5186AFAE","b":"556A504A0000","o":"13CAFAE","s":"_ZN5mongo15invariantFailedEPKcS1_j","s+":"F7"}}}
{"t":{"$date":"2022-07-14T08:24:14.656Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn67114","msg":"Frame","attr":{"frame":{"a":"556A517B25A2","b":"556A504A0000","o":"13125A2","s":"_ZN5mongo5OpMsg5parseERKNS_7MessageE.cold.429","s+":"644"}}}
{"t":{"$date":"2022-07-14T08:24:14.656Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn67114","msg":"Frame","attr":{"frame":{"a":"556A51A72CE6","b":"556A504A0000","o":"15D2CE6","s":"_ZN5mongo9transport19ServiceStateMachine4Impl23cleanupExhaustResourcesEv","s+":"66"}}}
{"t":{"$date":"2022-07-14T08:24:14.656Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn67114","msg":"Frame","attr":{"frame":{"a":"556A51A73545","b":"556A504A0000","o":"15D3545","s":"_ZN5mongo9transport19ServiceStateMachine4Impl14cleanupSessionERKNS_6StatusE","s+":"95"}}}
{"t":{"$date":"2022-07-14T08:24:14.656Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn67114","msg":"Frame","attr":{"frame":{"a":"556A5134D908","b":"556A504A0000","o":"EAD908","s":"_ZN5mongo9transport19ServiceStateMachine4Impl15scheduleNewLoopENS_6StatusE.cold.860","s+":"150"}}}
{"t":{"$date":"2022-07-14T08:24:14.656Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn67114","msg":"Frame","attr":{"frame":{"a":"556A51A73AB4","b":"556A504A0000","o":"15D3AB4","s":"_ZN5mongo14future_details4callIRZNS_9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusEEUlS5_E2_EEDaOT_NS_10StatusWithINS0_8FakeVoidEEE","s+":"34"}}}
{"t":{"$date":"2022-07-14T08:24:14.656Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn67114","msg":"Frame","attr":{"frame":{"a":"556A51A73E03","b":"556A504A0000","o":"15D3E03","s":"_ZN5mongo9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusE","s+":"283"}}}
{"t":{"$date":"2022-07-14T08:24:14.656Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn67114","msg":"Frame","attr":{"frame":{"a":"556A51A7422F","b":"556A504A0000","o":"15D422F","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_9transport19ServiceStateMachine4Impl15scheduleNewLoopES1_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1_","s+":"7F"}}}
{"t":{"$date":"2022-07-14T08:24:14.656Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn67114","msg":"Frame","attr":{"frame":{"a":"556A52153ED3","b":"556A504A0000","o":"1CB3ED3","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport15ServiceExecutor8scheduleENS0_IFvNS_6StatusEEEEEUlvE_EEDaOT_EN12SpecificImpl4callEv","s+":"33"}}}
{"t":{"$date":"2022-07-14T08:24:14.656Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn67114","msg":"Frame","attr":{"frame":{"a":"556A53B0FB7B","b":"556A504A0000","o":"366FB7B","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport26ServiceExecutorSynchronous12scheduleTaskES2_NS4_15ServiceExecutor13ScheduleFlagsEEUlvE0_EEDaOT_EN12SpecificImpl4callEv","s+":"BB"}}}
{"t":{"$date":"2022-07-14T08:24:14.656Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn67114","msg":"Frame","attr":{"frame":{"a":"556A53B13A2C","b":"556A504A0000","o":"3673A2C","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_25launchServiceWorkerThreadES2_EUlvE2_EEDaOT_EN12SpecificImpl4callEv","s+":"5C"}}}
{"t":{"$date":"2022-07-14T08:24:14.656Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn67114","msg":"Frame","attr":{"frame":{"a":"556A53B13A9C","b":"556A504A0000","o":"3673A9C","s":"_ZN5mongo12_GLOBAL__N_17runFuncEPv","s+":"1C"}}}
{"t":{"$date":"2022-07-14T08:24:14.656Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn67114","msg":"Frame","attr":{"frame":{"a":"7F4A29900609","b":"7F4A298F8000","o":"8609","s":"start_thread","s+":"D9"}}}
{"t":{"$date":"2022-07-14T08:24:14.656Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn67114","msg":"Frame","attr":{"frame":{"a":"7F4A29825133","b":"7F4A29706000","o":"11F133","s":"clone","s+":"43"}}},

{"t":{"$date":"2022-07-14T09:36:23.832Z"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn15034","msg":"client metadata","attr":{"remote":"10.20.55.1:63147","client":"conn15034","doc":{"driver":{"name":"PyMongo","version":"3.10.1"},"os":{"type":"Darwin","name":"Darwin","architecture":"x86_64","version":"12.3.1"},"platform":"CPython 3.9.12.final.0"}}}
{"t":{"$date":"2022-07-14T09:36:24.437Z"},"s":"I",  "c":"ACCESS",   "id":20250,   "ctx":"conn15034","msg":"Authentication succeeded","attr":{"mechanism":"SCRAM-SHA-256","speculative":false,"principalName":"reader","authenticationDatabase":"<DATABASE>","remote":"10.20.55.1:63147","extraInfo":{}}}
{"t":{"$date":"2022-07-14T09:38:50.902Z"},"s":"I",  "c":"NETWORK",  "id":22989,   "ctx":"conn15034","msg":"Error sending response to client. Ending connection from remote","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"Broken pipe"},"remote":"10.20.55.1:63147","connectionId":15034}}
{"t":{"$date":"2022-07-14T09:38:50.903Z"},"s":"F",  "c":"-",        "id":23079,   "ctx":"conn15034","msg":"Invariant failure","attr":{"expr":"message.operation() == dbMsg","file":"src/mongo/rpc/op_msg.cpp","line":133}}
{"t":{"$date":"2022-07-14T09:38:50.904Z"},"s":"F",  "c":"-",        "id":23080,   "ctx":"conn15034","msg":"\n\n***aborting after invariant() failure\n\n"}
{"t":{"$date":"2022-07-14T09:38:50.904Z"},"s":"F",  "c":"CONTROL",  "id":4757800, "ctx":"conn15034","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
{"t":{"$date":"2022-07-14T09:38:51.092Z"},"s":"I",  "c":"CONTROL",  "id":31380,   "ctx":"conn15034","msg":"BACKTRACE","attr":{"bt":{"backtrace":[{"a":"563BEE49AE75","b":"563BEA5DD000","o":"3EBDE75","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.357","s+":"215"},{"a":"563BEE49D909","b":"563BEA5DD000","o":"3EC0909","s":"_ZN5mongo15printStackTraceEv","s+":"29"},{"a":"563BEE495F46","b":"563BEA5DD000","o":"3EB8F46","s":"abruptQuit","s+":"66"},{"a":"7F09B36AD420","b":"7F09B3699000","o":"14420","s":"funlockfile","s+":"60"},{"a":"7F09B34EA00B","b":"7F09B34A7000","o":"4300B","s":"gsignal","s+":"CB"},{"a":"7F09B34C9859","b":"7F09B34A7000","o":"22859","s":"abort","s+":"12B"},{"a":"563BEB9A7FAE","b":"563BEA5DD000","o":"13CAFAE","s":"_ZN5mongo15invariantFailedEPKcS1_j","s+":"F7"},{"a":"563BEB8EF5A2","b":"563BEA5DD000","o":"13125A2","s":"_ZN5mongo5OpMsg5parseERKNS_7MessageE.cold.429","s+":"644"},{"a":"563BEBBAFCE6","b":"563BEA5DD000","o":"15D2CE6","s":"_ZN5mongo9transport19ServiceStateMachine4Impl23cleanupExhaustResourcesEv","s+":"66"},{"a":"563BEBBB0545","b":"563BEA5DD000","o":"15D3545","s":"_ZN5mongo9transport19ServiceStateMachine4Impl14cleanupSessionERKNS_6StatusE","s+":"95"},{"a":"563BEB48A908","b":"563BEA5DD000","o":"EAD908","s":"_ZN5mongo9transport19ServiceStateMachine4Impl15scheduleNewLoopENS_6StatusE.cold.860","s+":"150"},{"a":"563BEBBB0AB4","b":"563BEA5DD000","o":"15D3AB4","s":"_ZN5mongo14future_details4callIRZNS_9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusEEUlS5_E2_EEDaOT_NS_10StatusWithINS0_8FakeVoidEEE","s+":"34"},{"a":"563BEBBB0E03","b":"563BEA5DD000","o":"15D3E03","s":"_ZN5mongo9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusE","s+":"283"},{"a":"563BEBBB122F","b":"563BEA5DD000","o":"15D422F","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_9transport19ServiceStateMachine4Impl15scheduleNewLoopES1_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1_","s+":"7F"},{"a":"563BEDC4C9C1","b":"563BEA5DD000","o":"366F9C1","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_9transport26ServiceExecutorSynchronous18runOnDataAvailableERKSt10shared_ptrINS5_7SessionEES3_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1_","s+":"41"},{"a":"563BEC290ED3","b":"563BEA5DD000","o":"1CB3ED3","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport15ServiceExecutor8scheduleENS0_IFvNS_6StatusEEEEEUlvE_EEDaOT_EN12SpecificImpl4callEv","s+":"33"},{"a":"563BEDC4CB7B","b":"563BEA5DD000","o":"366FB7B","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport26ServiceExecutorSynchronous12scheduleTaskES2_NS4_15ServiceExecutor13ScheduleFlagsEEUlvE0_EEDaOT_EN12SpecificImpl4callEv","s+":"BB"},{"a":"563BEDC50A2C","b":"563BEA5DD000","o":"3673A2C","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_25launchServiceWorkerThreadES2_EUlvE2_EEDaOT_EN12SpecificImpl4callEv","s+":"5C"},{"a":"563BEDC50A9C","b":"563BEA5DD000","o":"3673A9C","s":"_ZN5mongo12_GLOBAL__N_17runFuncEPv","s+":"1C"},{"a":"7F09B36A1609","b":"7F09B3699000","o":"8609","s":"start_thread","s+":"D9"},{"a":"7F09B35C6133","b":"7F09B34A7000","o":"11F133","s":"clone","s+":"43"}],"processInfo":{"mongodbVersion":"5.0.9","gitVersion":"6f7dae919422dcd7f4892c10ff20cdc721ad00e6","compiledModules":[],"uname":{"sysname":"Linux","release":"5.4.0-121-generic","version":"#137-Ubuntu SMP Wed Jun 15 13:33:07 UTC 2022","machine":"x86_64"},"somap":[{"b":"563BEA5DD000","elfType":3,"buildId":"EEB6C6E4767A2FF4A2F77AE4A23FCBFE4981C02D"},{"b":"7F09B3699000","path":"/lib/x86_64-linux-gnu/libpthread.so.0","elfType":3,"buildId":"7B4536F41CDAA5888408E82D0836E33DCF436466"},{"b":"7F09B34A7000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"1878E6B475720C7C51969E69AB2D276FAE6D1DEE"}]}}}}
{"t":{"$date":"2022-07-14T09:38:51.092Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn15034","msg":"Frame","attr":{"frame":{"a":"563BEE49AE75","b":"563BEA5DD000","o":"3EBDE75","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.357","s+":"215"}}}
{"t":{"$date":"2022-07-14T09:38:51.092Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn15034","msg":"Frame","attr":{"frame":{"a":"563BEE49D909","b":"563BEA5DD000","o":"3EC0909","s":"_ZN5mongo15printStackTraceEv","s+":"29"}}}
{"t":{"$date":"2022-07-14T09:38:51.092Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn15034","msg":"Frame","attr":{"frame":{"a":"563BEE495F46","b":"563BEA5DD000","o":"3EB8F46","s":"abruptQuit","s+":"66"}}}
{"t":{"$date":"2022-07-14T09:38:51.092Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn15034","msg":"Frame","attr":{"frame":{"a":"7F09B36AD420","b":"7F09B3699000","o":"14420","s":"funlockfile","s+":"60"}}}
{"t":{"$date":"2022-07-14T09:38:51.092Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn15034","msg":"Frame","attr":{"frame":{"a":"7F09B34EA00B","b":"7F09B34A7000","o":"4300B","s":"gsignal","s+":"CB"}}}
{"t":{"$date":"2022-07-14T09:38:51.092Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn15034","msg":"Frame","attr":{"frame":{"a":"7F09B34C9859","b":"7F09B34A7000","o":"22859","s":"abort","s+":"12B"}}}
{"t":{"$date":"2022-07-14T09:38:51.092Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn15034","msg":"Frame","attr":{"frame":{"a":"563BEB9A7FAE","b":"563BEA5DD000","o":"13CAFAE","s":"_ZN5mongo15invariantFailedEPKcS1_j","s+":"F7"}}}
{"t":{"$date":"2022-07-14T09:38:51.092Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn15034","msg":"Frame","attr":{"frame":{"a":"563BEB8EF5A2","b":"563BEA5DD000","o":"13125A2","s":"_ZN5mongo5OpMsg5parseERKNS_7MessageE.cold.429","s+":"644"}}}
{"t":{"$date":"2022-07-14T09:38:51.092Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn15034","msg":"Frame","attr":{"frame":{"a":"563BEBBAFCE6","b":"563BEA5DD000","o":"15D2CE6","s":"_ZN5mongo9transport19ServiceStateMachine4Impl23cleanupExhaustResourcesEv","s+":"66"}}}
{"t":{"$date":"2022-07-14T09:38:51.092Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn15034","msg":"Frame","attr":{"frame":{"a":"563BEBBB0545","b":"563BEA5DD000","o":"15D3545","s":"_ZN5mongo9transport19ServiceStateMachine4Impl14cleanupSessionERKNS_6StatusE","s+":"95"}}}
{"t":{"$date":"2022-07-14T09:38:51.092Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn15034","msg":"Frame","attr":{"frame":{"a":"563BEB48A908","b":"563BEA5DD000","o":"EAD908","s":"_ZN5mongo9transport19ServiceStateMachine4Impl15scheduleNewLoopENS_6StatusE.cold.860","s+":"150"}}}
{"t":{"$date":"2022-07-14T09:38:51.092Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn15034","msg":"Frame","attr":{"frame":{"a":"563BEBBB0AB4","b":"563BEA5DD000","o":"15D3AB4","s":"_ZN5mongo14future_details4callIRZNS_9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusEEUlS5_E2_EEDaOT_NS_10StatusWithINS0_8FakeVoidEEE","s+":"34"}}}
{"t":{"$date":"2022-07-14T09:38:51.092Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn15034","msg":"Frame","attr":{"frame":{"a":"563BEBBB0E03","b":"563BEA5DD000","o":"15D3E03","s":"_ZN5mongo9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusE","s+":"283"}}}
{"t":{"$date":"2022-07-14T09:38:51.092Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn15034","msg":"Frame","attr":{"frame":{"a":"563BEBBB122F","b":"563BEA5DD000","o":"15D422F","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_9transport19ServiceStateMachine4Impl15scheduleNewLoopES1_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1_","s+":"7F"}}}
{"t":{"$date":"2022-07-14T09:38:51.092Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn15034","msg":"Frame","attr":{"frame":{"a":"563BEDC4C9C1","b":"563BEA5DD000","o":"366F9C1","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_9transport26ServiceExecutorSynchronous18runOnDataAvailableERKSt10shared_ptrINS5_7SessionEES3_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1_","s+":"41"}}}
{"t":{"$date":"2022-07-14T09:38:51.092Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn15034","msg":"Frame","attr":{"frame":{"a":"563BEC290ED3","b":"563BEA5DD000","o":"1CB3ED3","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport15ServiceExecutor8scheduleENS0_IFvNS_6StatusEEEEEUlvE_EEDaOT_EN12SpecificImpl4callEv","s+":"33"}}}
{"t":{"$date":"2022-07-14T09:38:51.092Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn15034","msg":"Frame","attr":{"frame":{"a":"563BEDC4CB7B","b":"563BEA5DD000","o":"366FB7B","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport26ServiceExecutorSynchronous12scheduleTaskES2_NS4_15ServiceExecutor13ScheduleFlagsEEUlvE0_EEDaOT_EN12SpecificImpl4callEv","s+":"BB"}}}
{"t":{"$date":"2022-07-14T09:38:51.092Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn15034","msg":"Frame","attr":{"frame":{"a":"563BEDC50A2C","b":"563BEA5DD000","o":"3673A2C","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_25launchServiceWorkerThreadES2_EUlvE2_EEDaOT_EN12SpecificImpl4callEv","s+":"5C"}}}
{"t":{"$date":"2022-07-14T09:38:51.092Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn15034","msg":"Frame","attr":{"frame":{"a":"563BEDC50A9C","b":"563BEA5DD000","o":"3673A9C","s":"_ZN5mongo12_GLOBAL__N_17runFuncEPv","s+":"1C"}}}
{"t":{"$date":"2022-07-14T09:38:51.092Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn15034","msg":"Frame","attr":{"frame":{"a":"7F09B36A1609","b":"7F09B3699000","o":"8609","s":"start_thread","s+":"D9"}}}
{"t":{"$date":"2022-07-14T09:38:51.092Z"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"conn15034","msg":"Frame","attr":{"frame":{"a":"7F09B35C6133","b":"7F09B34A7000","o":"11F133","s":"clone","s+":"43"}}}

Generated at Thu Feb 08 06:09:42 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.