[SERVER-46121] mongos crashes with invariant error after changing taskExecutorPoolSize Created: 13/Feb/20  Updated: 29/Oct/23  Resolved: 27/Feb/20

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 4.2.3
Fix Version/s: 4.2.4, 4.3.4

Type: Bug Priority: Critical - P2
Reporter: Anton Papp Assignee: Benjamin Caimano (Inactive)
Resolution: Fixed Votes: 0
Labels: KP42
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File mongos.conf     File tmp.log    
Issue Links:
Backports
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.2
Steps To Reproduce:

OS: current Ubuntu 18.04

MongoDB - Version: 4.2.3

fcv: 4.0

number of cfg-Servers: 3

number of shards: 4 (PSA - Replicasets)

number of connected mongos: 3

Sprint: Service Arch 2020-02-24, Service Arch 2020-03-09
Participants:
Case:

 Description   

Hi, we are running a sharded cluster on Ubuntu 18.04. (amd64) it was recently upgraded from version 4.0.x to 4.2.3. The cluster's fcv is still set to 4.0.

After facing some performance issues we increased the taskExecutorPoolSize on the mongos to match the number of cores in the servers. This gave us a huge boost in query performance using 4.0.x. After the Upgrade the mongos process crashes after a few seconds using the same configuration. 

mongos.conf tmp.log

Removing the taskExecutorPoolSize entry solves the issue, but gives us poor query performance. (4 times slower on a quad core system).

We also tried a clean install in a test-environment on the same OS and DB version, with an empty Database and got the same error.



 Comments   
Comment by Githook User [ 02/Mar/20 ]

Author:

{'name': 'Ben Caimano', 'email': 'ben.caimano@10gen.com'}

Message: SERVER-46121 Make separate Controllers with taskExecutorPoolSize > 1

This commit also introduces a new variant to verify that
taskExecutorPoolSize=4 works correctly.

(cherry picked from commit ac7c4b8079b8927bab3fbabf179a5be125c9c4bf)
Branch: v4.2
https://github.com/mongodb/mongo/commit/0f89b28128bce591388b20e38fd6c675eaac2b15

Comment by Githook User [ 27/Feb/20 ]

Author:

{'name': 'Ben Caimano', 'email': 'ben.caimano@10gen.com'}

Message: SERVER-46121 Make separate Controllers with taskExecutorPoolSize > 1

This commit also introduces a new variant to verify that
taskExecutorPoolSize=4 works correctly.
Branch: master
https://github.com/mongodb/mongo/commit/ac7c4b8079b8927bab3fbabf179a5be125c9c4bf

Comment by Benjamin Caimano (Inactive) [ 26/Feb/20 ]

toni.papp@gmail.com, dejun.xia@qq.com - I've confirmed the issue exists and persists to our current master branch. It appears to have slipped through a hole in our test coverage. I have a fix in progress that I expect to be on our master and v4.2 branches soon. The fix should be released with r4.2.4.

Comment by Dejun Xia [ 24/Feb/20 ]

Hi, we got exactly the same error when set `taskExecutorPoolSize` to 6 in the mongos configuration file.

2020-02-24T22:25:57.206+0800 F  -        [conn14] Invariant failure !groupAndId.maybeId src/mongo/s/sharding_task_executor_pool_controller.cpp 189
2020-02-24T22:25:57.206+0800 F  -        [conn14] 
 
***aborting after invariant() failure
 
 
2020-02-24T22:25:57.225+0800 F  -        [conn14] Got signal: 6 (Aborted).
 

with mongos version: 4.2.1 .

Comment by Kelsey Schubert [ 19/Feb/20 ]

Thanks for the report, toni.papp@gmail.com. We're investigating and will update this ticket when we know more.

Generated at Thu Feb 08 05:10:33 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.