[SERVER-71402] Hang analyzer continues to modify global logging configuration of resmoke Created: 16/Nov/22  Updated: 29/Oct/23  Resolved: 10/May/23

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 7.1.0-rc0

Type: Bug Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Juan Gu
Resolution: Fixed Votes: 0
Labels: auto-reverted
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Problem/Incident
Related
is related to SERVER-61432 symbolizer should not change global l... Closed
Assigned Teams:
Server Development Platform
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:
Linked BF Score: 35

 Description   

SERVER-61432 didn't fully address how the hang analyzer running causes mongod output to be written to the task logs. The same issue also causes log messages from resmoke to be written multiple times to the task logs.

[2022/11/15 19:09:20.153] Found 6 interesting processes [Pinfo(name='mongo', pidv=[25728]), Pinfo(name='mongod', pidv=[22861, 22864, 22867, 22870, 22873])]
[2022/11/15 19:09:20.153] [resmoke] 19:09:20.153Z Found 6 interesting processes [Pinfo(name='mongo', pidv=[25728]), Pinfo(name='mongod', pidv=[22861, 22864, 22867, 22870, 22873])]
[2022/11/15 19:09:20.153] Suspending process mongo with PID 25728
[2022/11/15 19:09:20.153] [resmoke] 19:09:20.153Z Suspending process mongo with PID 25728
[2022/11/15 19:09:20.154] Suspending process mongod with PID 22861
...
[2022/11/15 19:15:26.967] [2022-11-15 19:15:26,966 - j0:sec0 - INFO] | 2022-11-15T19:15:26.966Z I  CONTROL  5760901 [main] "Applied --setParameter options","attr":{"serverParameters":{"backtraceLogFile":{"default":"","value":"/data/db/job0/resmoke/node1/5a0a0573e1074098958720d5428b4328.stacktrace"},"coordinateCommitReturnImmediatelyAfterPersistingDecision":{"default":false,"value":false},"disableLogicalSessionCacheRefresh":{"default":false,"value":true},"enableTestCommands":{"default":false,"value":true},"failpoint.forceSyncSourceCandidate":{"default":{"mode":0,"data":{},"timesEntered":0},"value":{"mode":1,"data":{"hostAndPort":"localhost:20000"},"timesEntered":0}},"featureFlagAdditionalParticipants":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagAnalyzeShardKey":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagAsyncRemoteCommandAPI":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagCatalogShard":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagClusterWideToaster":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagColumnstoreIndexes":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagConcurrencyInChunkMigration":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagConnHealthMetrics":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagDeprioritizeLowPriorityOperations":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagDerivedMetadata":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagDowngradingToUpgrading":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagEnableExecutionControl":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagFLE2Range":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagGlobalIndexes":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagGlobalIndexesShardingCatalog":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagHistoricalPlacementShardingCatalog":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagMongoStore":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagOIDC":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagPointInTimeCatalogLookups":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagSbeFull":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagServerlessChangeStreams":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagShardMerge":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagTimeseriesScalabilityImprovements":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagToaster":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagUpdateDocumentShardKeyUsingTransactionApi":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"featureFlagUpdateOneWithoutShardKey":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"logComponentVerbosity":{"default":{"verbosity":0,"accessControl":{"verbosity":-1},"assert":{"verbosity":-1},"command":{"verbosity":-1},"control":{"verbosity":-1},"executor":{"verbosity":-1},"geo":{"verbosity":-1},"globalIndex":{"verbosity":-1},"index":{"verbosity":-1},"network":{"verbosity":-1,"asio":{"verbosity":-1},"bridge":{"verbosity":-1},"connectionPool":{"verbosity":-1}},"processHealth":{"verbosity":-1},"query":{"verbosity":-1},"replication":{"verbosity":-1,"election":{"verbosity":-1},"heartbeats":{"verbosity":-1},"initialSync":{"verbosity":-1},"rollback":{"verbosity":-1}},"sharding":{"verbosity":-1,"rangeDeleter":{"verbosity":-1},"shardingCatalogRefresh":{"verbosity":-1},"migration":{"verbosity":-1},"reshard":{"verbosity":-1},"migrationPerf":{"verbosity":-1}},"storage":{"verbosity":-1,"recovery":{"verbosity":-1},"journal":{"verbosity":-1},"wt":{"verbosity":-1,"wtBackup":{"verbosity":-1},"wtCheckpoint":{"verbosity":-1},"wtCompact":{"verbosity":-1},"wtEviction":{"verbosity":-1},"wtHS":{"verbosity":-1},"wtRecovery":{"verbosity":-1},"wtRTS":{"verbosity":-1},"wtSalvage":{"verbosity":-1},"wtTiered":{"verbosity":-1},"wtTimestamp":{"verbosity":-1},"wtTransaction":{"verbosity":-1},"wtVerify":{"verbosity":-1},"wtWriteLog":{"verbosity":-1}}},"write":{"verbosity":-1},"ftdc":{"verbosity":-1},"tracking":{"verbosity":-1},"transaction":{"verbosity":-1},"tenantMigration":{"verbosity":-1},"test":{"verbosity":-1},"resourceConsumption":{"verbosity":-1}},"value":{"verbosity":0,"accessControl":{"verbosity":-1},"assert":{"verbosity":-1},"command":{"verbosity":-1},"control":{"verbosity":-1},"executor":{"verbosity":-1},"geo":{"verbosity":-1},"globalIndex":{"verbosity":-1},"index":{"verbosity":-1},"network":{"verbosity":-1,"asio":{"verbosity":-1},"bridge":{"verbosity":-1},"connectionPool":{"verbosity":-1}},"processHealth":{"verbosity":-1},"query":{"verbosity":-1},"replication":{"verbosity":-1,"election":{"verbosity":4},"heartbeats":{"verbosity":2},"initialSync":{"verbosity":2},"rollback":{"verbosity":2}},"sharding":{"verbosity":-1,"rangeDeleter":{"verbosity":-1},"shardingCatalogRefresh":{"verbosity":-1},"migration":{"verbosity":2},"reshard":{"verbosity":-1},"migrationPerf":{"verbosity":-1}},"storage":{"verbosity":-1,"recovery":{"verbosity":2},"journal":{"verbosity":-1},"wt":{"verbosity":-1,"wtBackup":{"verbosity":-1},"wtCheckpoint":{"verbosity":-1},"wtCompact":{"verbosity":-1},"wtEviction":{"verbosity":-1},"wtHS":{"verbosity":-1},"wtRecovery":{"verbosity":-1},"wtRTS":{"verbosity":-1},"wtSalvage":{"verbosity":-1},"wtTiered":{"verbosity":-1},"wtTimestamp":{"verbosity":-1},"wtTransaction":{"verbosity":-1},"wtVerify":{"verbosity":-1},"wtWriteLog":{"verbosity":-1}}},"write":{"verbosity":-1},"ftdc":{"verbosity":-1},"tracking":{"verbosity":-1},"transaction":{"verbosity":4},"tenantMigration":{"verbosity":4},"test":{"verbosity":-1},"resourceConsumption":{"verbosity":-1}}},"maxIndexBuildDrainBatchSize":{"default":1000,"value":10},"numInitialSyncAttempts":{"default":10,"value":10000000},"roleGraphInvalidationIsFatal":{"default":false,"value":true},"shutdownTimeoutMillisForSignaledShutdown":{"default":15000,"value":100},"suppressNetworkInterfaceTransportLayerExceptions":{"default":{"value":false},"value":{"value":true,"version":"6.2"}},"testingDiagnosticsEnabled":{"default":false,"value":true},"transactionLifetimeLimitSeconds":{"default":60,"value":86400},"writePeriodicNoops":{"default":true,"value":false}}}
[2022/11/15 19:15:26.968] [2022-11-15 19:15:26,967 - j0:sec0 - INFO] | 2022-11-15T19:15:26.967Z I  -        4672602 [main] "Testing behaviors are enabled. This has serious implications for both performance and security."
[2022/11/15 19:15:26.969] [2022-11-15 19:15:26,969 - j0:sec0 - INFO] | 2022-11-15T19:15:26.969+00:00 I  NETWORK  4915701 [main] "Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":19},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":19},"outgoing":{"minWireVersion":6,"maxWireVersion":19},"isInternalClient":true}}



 Comments   
Comment by Githook User [ 10/May/23 ]

Author:

{'name': 'Juan Gu', 'email': 'juan.gu@mongodb.com', 'username': 'juangugit'}

Message: SERVER-71402 Hang analyzer continues to modify global logging configuration of resmoke
Branch: master
https://github.com/mongodb/mongo/commit/2acbd4ef153c773d25b6646e64c92a3be293017b

Comment by xgen-buildbaron-user [ 05/May/23 ]

Ticket re-opened due to revert. buildscripts_test began a consistent failure of buildscripts\tests\resmokelib\setup_multiversion\test_setup_multiversion.py

Comment by Githook User [ 05/May/23 ]

Author:

{'name': 'auto-revert-processor', 'email': 'dev-prod-dag@mongodb.com', 'username': ''}

Message: Revert "SERVER-71402 Hang analyzer continues to modify global logging configuration of resmoke"

This reverts commit 56597c49633af87107fd714b48aa0f3be29d04c0.
Branch: master
https://github.com/mongodb/mongo/commit/58694ac0c13585644fa1a9d97a3b8fdd94bd52ee

Comment by Githook User [ 05/May/23 ]

Author:

{'name': 'Juan Gu', 'email': 'juan.gu@mongodb.com', 'username': 'juangugit'}

Message: SERVER-71402 Hang analyzer continues to modify global logging configuration of resmoke
Branch: master
https://github.com/mongodb/mongo/commit/56597c49633af87107fd714b48aa0f3be29d04c0

Comment by Juan Gu [ 14/Dec/22 ]

janna.golden@mongodb.com , Got it. No worries.

Comment by Janna Golden [ 14/Dec/22 ]

Sorry, juan.gu@mongodb.com I swapped two numbers in my PR commit which moved this ticket to code review. Removing the PR link!

Generated at Thu Feb 08 06:18:54 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.