[SERVER-21983] 3.2.0 mongos crash - Invariant failure iter != pool.end() src/mongo/executor/connection_pool.cpp 572 Created: 22/Dec/15  Updated: 22/Dec/15  Resolved: 22/Dec/15

Status: Closed
Project: Core Server
Component/s: Networking
Affects Version/s: 3.2.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Tomasz Chmielewski Assignee: Adam Midvidy
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-21536 Invariant failure in executor/connect... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

I had the following crash in mongos 3.2.0:

2015-12-21T18:38:09.940+0000 I -        [NetworkInterfaceASIO-TaskExecutorPool-0-0] Invariant failure iter != pool.end() src/mongo/executor/connection_pool.cpp 572
2015-12-21T18:38:09.940+0000 I -        [NetworkInterfaceASIO-TaskExecutorPool-0-0] 
 
***aborting after invariant() failure
 
 
2015-12-21T18:38:09.945+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-4-0] Successfully connected to mongo-repl02a-int.sys.timedoctor.com:27017
2015-12-21T18:38:10.175+0000 F -        [NetworkInterfaceASIO-TaskExecutorPool-0-0] Got signal: 6 (Aborted).
 
 0xc308a2 0xc2fa09 0xc30212 0x7f7084bda340 0x7f708483bcc9 0x7f708483f0d8 0xbb9a1b 0x9d2ba0 0x9d4601 0x9d790c 0x9d8487 0x9d8bf1 0xc4c541 0xc4c761 0xc508ff 0x9e1c95 0x7f70853b5a40 0x7f7084bd2182 0x7f70848ff47d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"8308A2"},{"b":"400000","o":"82FA09"},{"b":"400000","o":"830212"},{"b":"7F7084BCA000","o":"10340"},{"b":"7F7084805000","o":"36CC9"},{"b":"7F7084805000","o":"3A0D8"},{"b":"400000","o":"7B9A1B"},{"b":"400000","o":"5D2BA0"},{"b":"400000","o":"5D4601"},{"b":"400000","o":"5D790C"},{"b":"400000","o":"5D8487"},{"b":"400000","o":"5D8BF1"},{"b":"400000","o":"84C541"},{"b":"400000","o":"84C761"},{"b":"400000","o":"8508FF"},{"b":"400000","o":"5E1C95"},{"b":"7F7085304000","o":"B1A40"},{"b":"7F7084BCA000","o":"8182"},{"b":"7F7084805000","o":"FA47D"}],"processInfo":{ "mongodbVersion" : "3.2.0", "gitVersion" : "45d947729a0315accb6d4f15a6b06be6d9c19fe7", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.13.0-68-generic", "version" : "#111-Ubuntu SMP Fri Nov 6 18:17:06 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "A18BC687493736D021F38FBBEBCFB4A7158CD3FF" }, { "b" : "7FFE50802000", "elfType" : 3, "buildId" : "03ABCC13B5E582D807C343784BC60CEAE03D36D1" }, { "b" : "7F7085DEF000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "D08DD65F97859C71BB2CBBF1043BD968EFE18AAD" }, { "b" : "7F7085A14000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "F86FA9FB4ECEB4E06B40DBDF761A4172B70A4229" }, { "b" : "7F708580C000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "92FCF41EFE012D6186E31A59AD05BDBB487769AB" }, { "b" : "7F7085608000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C1AE4CB7195D337A77A3C689051DABAA3980CA0C" }, { "b" : "7F7085304000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "4BF6F7ADD8244AD86008E6BF40D90F8873892197" }, { "b" : "7F7084FFE000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "1D76B71E905CB867B27CEF230FCB20F01A3178F5" }, { "b" : "7F7084DE8000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "8D0AA71411580EE6C08809695C3984769F25725B" }, { "b" : "7F7084BCA000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9318E8AF0BFBE444731BB0461202EF57F7C39542" }, { "b" : "7F7084805000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "30C94DC66A1FE95180C3D68D2B89E576D5AE213C" }, { "b" : "7F708604E000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9F00581AB3C73E3AEA35995A0C50D24D59A01D47" } ] }}
 mongos(_ZN5mongo15printStackTraceERSo+0x32) [0xc308a2]
 mongos(+0x82FA09) [0xc2fa09]
 mongos(+0x830212) [0xc30212]
 libpthread.so.0(+0x10340) [0x7f7084bda340]
 libc.so.6(gsignal+0x39) [0x7f708483bcc9]
 libc.so.6(abort+0x148) [0x7f708483f0d8]
 mongos(_ZN5mongo15invariantFailedEPKcS1_j+0xCB) [0xbb9a1b]
 mongos(_ZN5mongo8executor14ConnectionPool12SpecificPool22takeFromProcessingPoolEPNS1_19ConnectionInterfaceE+0x0) [0x9d2ba0]
 mongos(+0x5D4601) [0x9d4601]
 mongos(+0x5D790C) [0x9d790c]
 mongos(+0x5D8487) [0x9d8487]
 mongos(+0x5D8BF1) [0x9d8bf1]
 mongos(_ZN4asio6detail9scheduler10do_run_oneERNS0_11scoped_lockINS0_11posix_mutexEEERNS0_21scheduler_thread_infoERKSt10error_code+0x2F1) [0xc4c541]
 mongos(_ZN4asio6detail9scheduler3runERSt10error_code+0xC1) [0xc4c761]
 mongos(_ZN4asio10io_service3runEv+0x2F) [0xc508ff]
 mongos(+0x5E1C95) [0x9e1c95]
 libstdc++.so.6(+0xB1A40) [0x7f70853b5a40]
 libpthread.so.0(+0x8182) [0x7f7084bd2182]
 libc.so.6(clone+0x6D) [0x7f70848ff47d]
-----  END BACKTRACE  -----

That's on Ubuntu 14.04, 64 bit, with the following mongo packages installed:

# dpkg -l|grep mongo
ii  mongodb-org-mongos                       3.2.0                                amd64        MongoDB sharded cluster query router
ii  mongodb-org-shell                        3.2.0                                amd64        MongoDB shell client
ii  mongodb-org-tools                        3.2.0                                amd64        MongoDB tools



 Comments   
Comment by Adam Midvidy [ 22/Dec/15 ]

Hi Tomasz,

A fix for this issue has been backported, and will be shipping as part of the 3.2.1 release. I will close this ticket as a duplicate of the root cause issue, please respond there if you have further questions.

Thanks,
Adam

Comment by Adam Midvidy [ 22/Dec/15 ]

Hi Tomasz,

Thank you for reporting this issue and for including the log output.

This looks like an instance of SERVER-21536, which we have recently found and fixed in the development branch of MongoDB. We are working on backporting the fix to an upcoming 3.2 point release. In the mean time the best workaround is simply to restart mongos.

Adam

Generated at Thu Feb 08 03:59:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.