[SERVER-73941] Silent failures for Linux Shard Lite Cluster Audit 2022-11 Created: 13/Feb/23  Updated: 17/Feb/23

Status: Needs Scheduling
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Simon Eismann Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Participants:
Linked BF Score: 35

 Description   

It seems that the tests on the Linux Shard Lite Cluster Audit 2022-11 variant have been silently failing for a while.

The mongos logs terminate with the following error msg:

{"t":{"$date":"2023-02-05T01:18:25.191+00:00"},"s":"W",  "c":"ACCESS",   "id":24244,   "ctx":"conn123","msg":"Failure writing to audit log: {status}","attr":{"status":{"code":39,"codeName":"FileStreamFailed","errmsg":"File \"/data/auditlogs/auditLog.bson\" in failed state"}}}
{"t":{"$date":"2023-02-05T01:18:25.192+00:00"},"s":"W",  "c":"ACCESS",   "id":24243,   "ctx":"conn44","msg":"Failure acquiring audit logger","attr":{"error":{"code":39,"codeName":"FileStreamFailed","errmsg":"File \"/data/auditlogs/auditLog.bson\" in failed state"}}}
{"t":{"$date":"2023-02-05T01:18:25.192+00:00"},"s":"F",  "c":"CONTROL",  "id":6384300, "ctx":"conn123","msg":"Writing fatal message","attr":{"message":"\n"}}
{"t":{"$date":"2023-02-05T01:18:25.192+00:00"},"s":"W",  "c":"ACCESS",   "id":24244,   "ctx":"conn44","msg":"Failure writing to audit log: {status}","attr":{"status":{"code":39,"codeName":"FileStreamFailed","errmsg":"File \"/data/auditlogs/auditLog.bson\" in failed state"}}}
{"t":{"$date":"2023-02-05T01:18:25.192+00:00"},"s":"F",  "c":"CONTROL",  "id":6384300, "ctx":"conn123","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
{"t":{"$date":"2023-02-05T01:18:25.192+00:00"},"s":"W",  "c":"ACCESS",   "id":24243,   "ctx":"conn36","msg":"Failure acquiring audit logger","attr":{"error":{"code":39,"codeName":"FileStreamFailed","errmsg":"File \"/data/auditlogs/auditLog.bson\" in failed state"}}}
{"t":{"$date":"2023-02-05T01:18:25.192+00:00"},"s":"W",  "c":"ACCESS",   "id":24244,   "ctx":"conn36","msg":"Failure writing to audit log: {status}","attr":{"status":{"code":39,"codeName":"FileStreamFailed","errmsg":"File \"/data/auditlogs/auditLog.bson\" in failed state"}}}
{"t":{"$date":"2023-02-05T01:18:25.192+00:00"},"s":"W",  "c":"ACCESS",   "id":24243,   "ctx":"conn25","msg":"Failure acquiring audit logger","attr":{"error":{"code":39,"codeName":"FileStreamFailed","errmsg":"File \"/data/auditlogs/auditLog.bson\" in failed state"}}}
{"t":{"$date":"2023-02-05T01:18:25.192+00:00"},"s":"W",  "c":"ACCESS",   "id":24244,   "ctx":"conn25","msg":"Failure writing to audit log: {status}","attr":{"status":{"code":39,"codeName":"FileStreamFailed","errmsg":"File \"/data/auditlogs/auditLog.bson\" in failed state"}}}
{"t":{"$date":"2023-02-05T01:18:25.192+00:00"},"s":"W",  "c":"ACCESS",   "id":24243,   "ctx":"conn122","msg":"Failure acquiring audit logger","attr":{"error":{"code":39,"codeName":"FileStreamFailed","errmsg":"File \"/data/auditlogs/auditLog.bson\" in failed state"}}}
{"t":{"$date":"2023-02-

We should check for these error messages and flag the test as failed moving forward.

Example run: https://evergreen.mongodb.com/task/sys_perf_linux_shard_lite_audit.2022_11_industry_benchmarks_450ffbb63d5b265d2e17c6ec4556f030099b9f60_23_02_04_12_30_28


Generated at Thu Feb 08 06:26:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.