[SERVER-71826] M60-like-replica.2022-10 3-Node ReplSet is regularly failing Created: 18/Nov/22  Updated: 29/Oct/23  Resolved: 06/Dec/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 6.3.0-rc0

Type: Bug Priority: Major - P3
Reporter: David Daly Assignee: Amy Rogan
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Problem/Incident
Related
Backwards Compatibility: Fully Compatible
Participants:
Linked BF Score: 35

 Description   

Sometimes failing silently, such as here The test is failing to insert docs and should be getting much higher results.



 Comments   
Comment by Githook User [ 06/Dec/22 ]

Author:

{'name': 'roganamy', 'email': '93335130+roganamy@users.noreply.github.com', 'username': 'roganamy'}

Message: SERVER-71826: M60-like-replica.2022-10 3-Node ReplSet is regularly failing
Branch: master
https://github.com/mongodb/mongo/commit/3dd74ed335a29554b47a1188d9bfddb11d7b46ae

Comment by Amy Rogan [ 05/Dec/22 ]

Standup: updated m60-like-replica.2022-11 in system_perf.yml to use new workload setup and infrastructure_provisioning release which gave expected results. Created a PR with the changes

Comment by David Daly [ 29/Nov/22 ]

amy.rogan@mongodb.com maybe there are two issues, because I opened this ticket before my change went in. 

Comment by Amy Rogan [ 29/Nov/22 ]

Standup: David pointed out that I was using the wrong workload_setup file when testing the revert of his change. I reverted the Java change David in the correct workload_setup file and the issue doesn't occur.

Question: Can this change be reverted? Is there another path we can follow to achieve same result if reverting isn't an option? 

Comment by Amy Rogan [ 28/Nov/22 ]

Standup: patch running that reverts davids changes to Java installation as per conversations on Friday

Update: reverting the change did not fix the issue

Comment by Amy Rogan [ 25/Nov/22 ]

Standup: going through ideas for solving based on ss/tls misconfiguration I found other BFs that address misconfiguration of ssl and the error messages are completely different. A lot more indicative of misconfiguration (i.e. NET::ERR_CERT_COMMON_NAME_INVALID) I also ran locally with changes listed in previous comment and didn't led me anywhere/allowed me to prove it was ssl/tls misconfig. 

looking into other possibilities.. in the analysis run it fails to load class "org.slf4j.impl.StaticLoggerBinder" which points you to here for a solution of 'Placing one (and only one) of slf4j-nop.jar slf4j-simple.jarslf4j-log4j12.jarslf4j-jdk14.jar or logback-classic.jar on the class path'.

Question: Not sure where to go from here? been looking through ycsb code to see if I can find something

Comment by James O'Leary [ 24/Nov/22 ]

https://jira.mongodb.org/browse/BF-27004 probably depends on this.

Comment by Amy Rogan [ 24/Nov/22 ]

Standup: no reply from Evergreen ticket, Question - does this need to be escalated?
Looking at more recent runs and the problem is still persistent see here. Looks like it's still a mix up between ssl and tls - see here.

{"t":

{"$date":"2022-11-24T01:26:09.928Z"}

,"s":"W", "c":"CONTROL", "id":23321, "ctx":"main","msg":"Option: This name is deprecated. Please use the preferred name instead.","attr":{"deprecatedName":"ssl","preferredName":"tls"}}

This is seen at the beginning of test_control during cpu_noise workload. Testing turning canaries off to see if this produces the same error or not. And also testing removing upload ssl keys (this fixed something similar before) and changing it to tls to see if it still happens with these changes. Will update here with results

Comment by James O'Leary [ 22/Nov/22 ]

EVG-18089 would have caught this issue but it was closed.

Comment by Amy Rogan [ 22/Nov/22 ]

Standup: Thanks Jim for your suggestions/help (due to mixing ssl and tls). Going through them today to try and prove them. Will give an update in the comments accordingly

Generated at Thu Feb 08 06:20:04 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.