Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-73941

Silent failures for Linux Shard Lite Cluster Audit 2022-11

    • Type: Icon: Improvement Improvement
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • 35

      It seems that the tests on the Linux Shard Lite Cluster Audit 2022-11 variant have been silently failing for a while.

      The mongos logs terminate with the following error msg:

      {"t":{"$date":"2023-02-05T01:18:25.191+00:00"},"s":"W",  "c":"ACCESS",   "id":24244,   "ctx":"conn123","msg":"Failure writing to audit log: {status}","attr":{"status":{"code":39,"codeName":"FileStreamFailed","errmsg":"File \"/data/auditlogs/auditLog.bson\" in failed state"}}}
      {"t":{"$date":"2023-02-05T01:18:25.192+00:00"},"s":"W",  "c":"ACCESS",   "id":24243,   "ctx":"conn44","msg":"Failure acquiring audit logger","attr":{"error":{"code":39,"codeName":"FileStreamFailed","errmsg":"File \"/data/auditlogs/auditLog.bson\" in failed state"}}}
      {"t":{"$date":"2023-02-05T01:18:25.192+00:00"},"s":"F",  "c":"CONTROL",  "id":6384300, "ctx":"conn123","msg":"Writing fatal message","attr":{"message":"\n"}}
      {"t":{"$date":"2023-02-05T01:18:25.192+00:00"},"s":"W",  "c":"ACCESS",   "id":24244,   "ctx":"conn44","msg":"Failure writing to audit log: {status}","attr":{"status":{"code":39,"codeName":"FileStreamFailed","errmsg":"File \"/data/auditlogs/auditLog.bson\" in failed state"}}}
      {"t":{"$date":"2023-02-05T01:18:25.192+00:00"},"s":"F",  "c":"CONTROL",  "id":6384300, "ctx":"conn123","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
      {"t":{"$date":"2023-02-05T01:18:25.192+00:00"},"s":"W",  "c":"ACCESS",   "id":24243,   "ctx":"conn36","msg":"Failure acquiring audit logger","attr":{"error":{"code":39,"codeName":"FileStreamFailed","errmsg":"File \"/data/auditlogs/auditLog.bson\" in failed state"}}}
      {"t":{"$date":"2023-02-05T01:18:25.192+00:00"},"s":"W",  "c":"ACCESS",   "id":24244,   "ctx":"conn36","msg":"Failure writing to audit log: {status}","attr":{"status":{"code":39,"codeName":"FileStreamFailed","errmsg":"File \"/data/auditlogs/auditLog.bson\" in failed state"}}}
      {"t":{"$date":"2023-02-05T01:18:25.192+00:00"},"s":"W",  "c":"ACCESS",   "id":24243,   "ctx":"conn25","msg":"Failure acquiring audit logger","attr":{"error":{"code":39,"codeName":"FileStreamFailed","errmsg":"File \"/data/auditlogs/auditLog.bson\" in failed state"}}}
      {"t":{"$date":"2023-02-05T01:18:25.192+00:00"},"s":"W",  "c":"ACCESS",   "id":24244,   "ctx":"conn25","msg":"Failure writing to audit log: {status}","attr":{"status":{"code":39,"codeName":"FileStreamFailed","errmsg":"File \"/data/auditlogs/auditLog.bson\" in failed state"}}}
      {"t":{"$date":"2023-02-05T01:18:25.192+00:00"},"s":"W",  "c":"ACCESS",   "id":24243,   "ctx":"conn122","msg":"Failure acquiring audit logger","attr":{"error":{"code":39,"codeName":"FileStreamFailed","errmsg":"File \"/data/auditlogs/auditLog.bson\" in failed state"}}}
      {"t":{"$date":"2023-02-

      We should check for these error messages and flag the test as failed moving forward.

      Example run: https://evergreen.mongodb.com/task/sys_perf_linux_shard_lite_audit.2022_11_industry_benchmarks_450ffbb63d5b265d2e17c6ec4556f030099b9f60_23_02_04_12_30_28

            Assignee:
            Unassigned Unassigned
            Reporter:
            simon.eismann@mongodb.com Simon Eismann
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: