[SERVER-72866] resmoke symbolizer service raises Python error when stacktrace file isn't found Created: 15/Jan/23  Updated: 29/Oct/23  Resolved: 30/Jan/23

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: 6.2.0-rc6
Fix Version/s: 6.3.0-rc0

Type: Bug Priority: Critical - P2
Reporter: Max Hirschhorn Assignee: Jasur Nurboev (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-68632 Lock threads when collecting stacktra... Closed
is related to SERVER-71548 Avoid deleting files after symbolization Closed
is related to SERVER-65781 Have shell fixtures be symbolized Closed
Assigned Teams:
Decision Automation Group
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: DAG 2023-02-06
Participants:
Story Points: 1

 Description   

[2023/01/15 04:10:59.378] [executor:fsm_workload_test:job16] 04:10:59.268Z Encountered an error during test execution.
[2023/01/15 04:10:59.378] Traceback (most recent call last):
[2023/01/15 04:10:59.378]   File "/data/mci/ba0aa3721b36a52ba02e20a2b82cc724/src/buildscripts/resmokelib/testing/job.py", line 83, in __call__
[2023/01/15 04:10:59.378]     self._run(queue, interrupt_flag, teardown_flag, hook_failure_flag)
[2023/01/15 04:10:59.378]   File "/data/mci/ba0aa3721b36a52ba02e20a2b82cc724/src/buildscripts/resmokelib/testing/job.py", line 129, in _run
[2023/01/15 04:10:59.378]     self._execute_test(test, hook_failure_flag)
[2023/01/15 04:10:59.378]   File "/data/mci/ba0aa3721b36a52ba02e20a2b82cc724/src/buildscripts/resmokelib/testing/job.py", line 172, in _execute_test
[2023/01/15 04:10:59.378]     test(self.report)
[2023/01/15 04:10:59.378]   File "/opt/mongodbtoolchain/revisions/ecdb6f23a2cc2e822b8dc2eeb3f04c31f25c66ce/stow/python3-v4.sGn/lib/python3.10/unittest/case.py", line 650, in __call__
[2023/01/15 04:10:59.378]     return self.run(*args, **kwds)
[2023/01/15 04:10:59.378]   File "/opt/mongodbtoolchain/revisions/ecdb6f23a2cc2e822b8dc2eeb3f04c31f25c66ce/stow/python3-v4.sGn/lib/python3.10/unittest/case.py", line 620, in run
[2023/01/15 04:10:59.378]     result.stopTest(self)
[2023/01/15 04:10:59.378]   File "/data/mci/ba0aa3721b36a52ba02e20a2b82cc724/src/buildscripts/resmokelib/testing/report.py", line 149, in stopTest
[2023/01/15 04:10:59.378]     symbolizer.symbolize_test_logs(test)
[2023/01/15 04:10:59.378]   File "/data/mci/ba0aa3721b36a52ba02e20a2b82cc724/src/buildscripts/resmokelib/testing/symbolizer_service.py", line 93, in symbolize_test_logs
[2023/01/15 04:10:59.378]     files = self.collect_stacktrace_files(dbpath)
[2023/01/15 04:10:59.378]   File "/data/mci/ba0aa3721b36a52ba02e20a2b82cc724/src/buildscripts/resmokelib/testing/symbolizer_service.py", line 168, in collect_stacktrace_files
[2023/01/15 04:10:59.378]     files = self.file_service.filter_out_empty_files(files)
[2023/01/15 04:10:59.378]   File "/data/mci/ba0aa3721b36a52ba02e20a2b82cc724/src/buildscripts/resmokelib/testing/symbolizer_service.py", line 274, in filter_out_empty_files
[2023/01/15 04:10:59.378]     return [f for f in files if not os.stat(f).st_size == 0]
[2023/01/15 04:10:59.378]   File "/data/mci/ba0aa3721b36a52ba02e20a2b82cc724/src/buildscripts/resmokelib/testing/symbolizer_service.py", line 274, in <listcomp>
[2023/01/15 04:10:59.378]     return [f for f in files if not os.stat(f).st_size == 0]
[2023/01/15 04:10:59.378] FileNotFoundError: [Errno 2] No such file or directory: '/data/db/job16/resmoke/node0/simulateCrashes/0/8ea6d6b4ccd44df2a02e87bbed143cf9.stacktrace'

https://parsley.mongodb.com/evergreen/mongodb_mongo_master_enterprise_rhel_80_64_bit_dynamic_all_feature_flags_required_query_patch_only_config_fuzzer_simulate_crash_concurrency_replication_a21e26ea38cfcc3ef90a32a8ab9db2e391af51de_23_01_14_15_35_49/0/task?bookmarks=0,1588,1610,1998&selectedLine=1588



 Comments   
Comment by Githook User [ 26/Jan/23 ]

Author:

{'name': 'jasurbeknurboyev', 'email': '998946972365', 'username': ''}

Message: SERVER-72866 resmoke symbolizer service raises Python error when stacktrace file isn't found
Branch: master
https://github.com/mongodb/mongo/commit/c18ffc704499fd2946fbf57fe97b8abc107c27e2

Comment by Max Hirschhorn [ 15/Jan/23 ]

I'm filing this as a P2 because similar to SERVER-72860 the FileNotFoundError exception can lead to resmoke to incorrectly mark Evergreen tasks as setup failures.

Setup failures are intentionally ignored by the Build Barons so this can lead to delays in the timeliness of identifying true failures. (Setup failures are ignored because Logkeeper instability has been generally accepted and accommodated within the testing infrastructure, see SERVER-35472. The concept of setup failures may be worth revisiting now that Logkeeper has moved to S3 but I'm considering that outside the scope of this issue here.)

python buildscripts/resmoke.py run --suite=core --log=buildlogger jstests/core/query/all/all.js

diff --git a/buildscripts/resmokelib/logging/buildlogger.py b/buildscripts/resmokelib/logging/buildlogger.py
index 1ff2689ea64..d1e89db4278 100644
--- a/buildscripts/resmokelib/logging/buildlogger.py
+++ b/buildscripts/resmokelib/logging/buildlogger.py
@@ -266,10 +266,7 @@ class BuildloggerServer(object):
     def __init__(self):
         """Initialize BuildloggerServer."""
         tmp_globals = {}
-        self.config = {}
-        exec(
-            compile(open(_BUILDLOGGER_CONFIG, "rb").read(), _BUILDLOGGER_CONFIG, 'exec'),
-            tmp_globals, self.config)
+        self.config = dict(username="u", password="p", builder="b", build_num="1")
 
         # Rename "slavename" to "username" if present.
         if "slavename" in self.config and "username" not in self.config:
diff --git a/buildscripts/resmokelib/logging/flush.py b/buildscripts/resmokelib/logging/flush.py
index 16335ef44ab..a2b32d5e9a2 100644
--- a/buildscripts/resmokelib/logging/flush.py
+++ b/buildscripts/resmokelib/logging/flush.py
@@ -35,7 +35,7 @@ def stop_thread():
     _FLUSH_THREAD.signal_shutdown()
     # Wait for 1min instead of _FLUSH_THREAD.await_shutdown() because we can
     # sometimes wait indefinitely for a response, causing a task timeout.
-    _FLUSH_THREAD.join(60)
+    _FLUSH_THREAD.join(5)
 
     success = not _FLUSH_THREAD.is_alive()
     return success
diff --git a/buildscripts/resmokelib/logging/handlers.py b/buildscripts/resmokelib/logging/handlers.py
index 29292a3bdef..5ff2d068762 100644
--- a/buildscripts/resmokelib/logging/handlers.py
+++ b/buildscripts/resmokelib/logging/handlers.py
@@ -192,6 +192,8 @@ class HTTPHandler(object):
         on the content type.
         """
 
+        return dict(id="fake_id")
+
         data = utils.default_if_none(data, [])
         data = json.dumps(data)
 
diff --git a/buildscripts/resmokelib/testing/report.py b/buildscripts/resmokelib/testing/report.py
index 5ac5145cdee..d2278ec9abc 100644
--- a/buildscripts/resmokelib/testing/report.py
+++ b/buildscripts/resmokelib/testing/report.py
@@ -145,6 +145,9 @@ class TestReport(unittest.TestResult):
         # Take a lock to download the debug symbols if it hasn't already been downloaded.
         # log symbolized output to test.logger.info()
 
+        if "fixture_setup" not in test.basename() and "fixture_teardown" not in test.basename():
+            raise FileNotFoundError("Intentionally raised")
+
         symbolizer = ResmokeSymbolizer()
         symbolizer.symbolize_test_logs(test)
         # symbolization completed
diff --git a/jstests/core/query/all/all.js b/jstests/core/query/all/all.js
index a718e2615e8..8d8a6dfac1f 100644
--- a/jstests/core/query/all/all.js
+++ b/jstests/core/query/all/all.js
@@ -1,3 +1,5 @@
+throw new Error("Intentionally thrown");
+
 t = db.jstests_all;
 t.drop();

Generated at Thu Feb 08 06:23:00 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.