-
Type:
Build Failure
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: None
-
49
-
Storage Engines - 2022-09-19
-
v6.1
We've had mulltiple Evergreen failures of the verify-datafile-big-endian test on zseries. They go back to at least August 24.
Here's a typical error log:
[2022/09/06 13:35:35.017] Running command 'shell.exec' in "verify wt datafiles" (step 3 of 3)[2022/09/06 13:35:35.237] [dirs_include_datafile]:[2022/09/06 13:35:35.237] ./WT_TEST.1/OPS.TRACE[2022/09/06 13:35:35.237] ./WT_TEST.1[2022/09/06 13:35:35.237] ./WT_TEST.10/OPS.TRACE[2022/09/06 13:35:35.237] ./WT_TEST.10[2022/09/06 13:35:35.237] ./WT_TEST.2/OPS.TRACE[2022/09/06 13:35:35.237] ./WT_TEST.2[2022/09/06 13:35:35.237] ./WT_TEST.3/OPS.TRACE[2022/09/06 13:35:35.237] ./WT_TEST.3[2022/09/06 13:35:35.237] ./WT_TEST.4/OPS.TRACE[2022/09/06 13:35:35.237] ./WT_TEST.4[2022/09/06 13:35:35.237] ./WT_TEST.4/BACKUP[2022/09/06 13:35:35.237] ./WT_TEST.5/OPS.TRACE[2022/09/06 13:35:35.237] ./WT_TEST.5[2022/09/06 13:35:35.237] ./WT_TEST.5/SALVAGE.copy[2022/09/06 13:35:35.237] ./WT_TEST.6/OPS.TRACE[2022/09/06 13:35:35.237] ./WT_TEST.6[2022/09/06 13:35:35.237] ./WT_TEST.6/BACKUP[2022/09/06 13:35:35.237] ./WT_TEST.7/OPS.TRACE[2022/09/06 13:35:35.237] ./WT_TEST.7[2022/09/06 13:35:35.237] ./WT_TEST.8/OPS.TRACE[2022/09/06 13:35:35.237] ./WT_TEST.8[2022/09/06 13:35:35.237] ./WT_TEST.9/OPS.TRACE[2022/09/06 13:35:35.237] ./WT_TEST.9[2022/09/06 13:36:50.272] ./WT_TEST.1/OPS.TRACE[2022/09/06 13:36:50.272] ./WT_TEST.1[2022/09/06 13:36:50.272] [1662471410:272021][27842:0x3ffa94f8720], wt: [WT_VERB_DEFAULT][ERROR]: __wt_log_scan, 2088: no log files found: Operation not supported[2022/09/06 13:36:50.278] wt: printlog: Operation not supported
Prior to these errors we had multiple occurrences of an error storing the artifacts used by this test. For example:
[2022/08/17 02:32:46.688] Running command 's3.put' in "upload endian format artifacts" (step 5 of 5) [2022/08/17 02:32:47.102] Putting wiredtiger/cmake_build/test/format/WT_TEST.tgz into build_external/wiredtiger/big-endian/7ece8460dcfb63fd49bf068cff27d0ada04fc943/artifacts/WT_TEST-big-endian.tgz (https://build_external.s3.amazonaws.com/wiredtiger/big-endian/7ece8460dcfb63fd49bf068cff27d0ada04fc943/artifacts/WT_TEST-big-endian.tgz) [2022/08/17 02:32:47.110] performing s3 put to build_external of wiredtiger/big-endian/7ece8460dcfb63fd49bf068cff27d0ada04fc943/artifacts/WT_TEST-big-endian.tgz [1 of 5] [2022/08/17 02:33:03.006] problem putting s3 file: copying data to file: uploading part: RequestError: send request failed [2022/08/17 02:33:03.006] caused by: Put "https://s3.amazonaws.com/build_external/wiredtiger/big-endian/7ece8460dcfb63fd49bf068cff27d0ada04fc943/artifacts/WT_TEST-big-endian.tgz?partNumber=8&uploadId=yEC4R2_avlkgw66Xt_S8qrCCjGGbf_ULT8VVYjpI2NM4V9aUBwupnMdE_M2GfdD1E58u8v3VN8RqU2ofDlinZomcDLytmB2kbEViXGGuCP2dQ9rzdfbJ2QATmZ.yfais": net/http: TLS handshake timeout [2022/08/17 02:33:03.006] performing s3 put to build_external of wiredtiger/big-endian/7ece8460dcfb63fd49bf068cff27d0ada04fc943/artifacts/WT_TEST-big-endian.tgz [2 of 5] [2022/08/17 02:41:19.261] problem putting s3 file: copying data to file: uploading part: RequestError: send request failed [2022/08/17 02:41:19.261] caused by: Put "https://s3.amazonaws.com/build_external/wiredtiger/big-endian/7ece8460dcfb63fd49bf068cff27d0ada04fc943/artifacts/WT_TEST-big-endian.tgz?partNumber=793&uploadId=yjtFd9ntA4zzdihuPhW8YnbGgf6DK4VOiTMwvPllSy4l7rr4oCfB3W2J2CAv94ZddsQpuTKk8JI4Ybf2uPjOW5U358Az9ZxatCOCKZL6M_TCQKn8e1HTtOYux51ve8r2": net/http: TLS handshake timeout [2022/08/17 02:41:19.261] performing s3 put to build_external of wiredtiger/big-endian/7ece8460dcfb63fd49bf068cff27d0ada04fc943/artifacts/WT_TEST-big-endian.tgz [3 of 5] [2022/08/17 02:41:34.186] problem putting s3 file: copying data to file: uploading part: RequestError: send request failed [2022/08/17 02:41:34.186] caused by: Put "https://s3.amazonaws.com/build_external/wiredtiger/big-endian/7ece8460dcfb63fd49bf068cff27d0ada04fc943/artifacts/WT_TEST-big-endian.tgz?partNumber=4&uploadId=BwPfL79KcRAJ2nuaF64TPHyoVUgZ8Eu.2CAwHvsSgOZC2FzNR3JzKd5RBBNhbQA875EaCvv_xRuDwF5tqrWQi4Zi1nz4CvEmv8bwd1PZBFUUUz2UyqZmTNZW6bNC6aa2": net/http: TLS handshake timeout [2022/08/17 02:41:34.186] performing s3 put to build_external of wiredtiger/big-endian/7ece8460dcfb63fd49bf068cff27d0ada04fc943/artifacts/WT_TEST-big-endian.tgz [4 of 5] [2022/08/17 02:42:37.812] problem putting s3 file: copying data to file: uploading part: RequestError: send request failed [2022/08/17 02:42:37.812] caused by: Put "https://s3.amazonaws.com/build_external/wiredtiger/big-endian/7ece8460dcfb63fd49bf068cff27d0ada04fc943/artifacts/WT_TEST-big-endian.tgz?partNumber=87&uploadId=0j00H3C13IRvDTO6N9b3neJMulgNV9Ay.P0jpn2pFXVsm6.pec8Rs9oVmXpp.XXezbE1uB_pph7UBvoVctj7u2FeBcq6UILzrXhR.mO3ExGtfor8ePcbigxZCx5g4CrM": net/http: TLS handshake timeout [2022/08/17 02:42:37.812] performing s3 put to build_external of wiredtiger/big-endian/7ece8460dcfb63fd49bf068cff27d0ada04fc943/artifacts/WT_TEST-big-endian.tgz [5 of 5] [2022/08/17 02:43:28.247] problem putting s3 file: copying data to file: uploading part: RequestError: send request failed [2022/08/17 02:43:28.247] caused by: Put "https://s3.amazonaws.com/build_external/wiredtiger/big-endian/7ece8460dcfb63fd49bf068cff27d0ada04fc943/artifacts/WT_TEST-big-endian.tgz?partNumber=46&uploadId=s9fW98iyFWiUFGzk_5wtUptvdZoSn2csfhmS1SMH0EeBZqi7D2QBWTzbT2Uq7CJ6TOL5XFJ93wL28UI7UryVx4TqtZU6myC5pl7u6QamA.07nmfamFg60ry_B9a.2Em9": net/http: TLS handshake timeout [2022/08/17 02:43:28.278] attempted to upload 1 files, 0 successfully uploaded [2022/08/17 02:43:28.278] Command failed: uploaded 0 files of 1 requested [2022/08/17 02:43:28.324] Task completed - FAILURE. [2022/08/17 02:43:28.452] Running post-task commands.
