Implement a retry mechanism for transactions that may rollback

Test failure

[2022/03/24 06:33:07.211] ERROR: test_hs20.test_hs20.test_hs20(column) (subunit.RemotedTestCase)

[2022/03/24 06:33:07.211] test_hs20.test_hs20.test_hs20(column)

[2022/03/24 06:33:07.211] ----------------------------------------------------------------------

[2022/03/24 06:33:07.211] testtools.testresult.real._StringException: Traceback (most recent call last):

[2022/03/24 06:33:07.211]   File "/data/mci/72c01000a379bf4ce02ccd35a95d2289/wiredtiger/test/suite/test_hs20.py", line 102, in test_hs20

[2022/03/24 06:33:07.211]     self.assertEqual(cursor[self.make_key(i)], value1 + "B")

[2022/03/24 06:33:07.211]   File "/data/mci/72c01000a379bf4ce02ccd35a95d2289/wiredtiger/cmake_build/lang/python/wiredtiger/swig_wiredtiger.py", line 341, in __getitem__

[2022/03/24 06:33:07.211]     if self.search() != 0:

[2022/03/24 06:33:07.211] _wiredtiger.WiredTigerError: WT_ROLLBACK: conflict between concurrent operations

[2022/03/24 06:33:07.211] ----------------------------------------------------------------------

[2022/03/24 06:33:07.211] Ran 8001 tests in 7698.122s

[2022/03/24 06:33:07.211] FAILED (errors=1, skipped=576)

[2022/03/24 06:33:07.211] failure: test_hs20.test_hs20.test_hs20(column) [

[2022/03/24 06:33:07.211] Traceback (most recent call last):

[2022/03/24 06:33:07.211]   File "/data/mci/72c01000a379bf4ce02ccd35a95d2289/wiredtiger/test/suite/wttest.py", line 508, in tearDown

[2022/03/24 06:33:07.211]     self.captureout.check(self)

[2022/03/24 06:33:07.211]   File "/data/mci/72c01000a379bf4ce02ccd35a95d2289/wiredtiger/test/suite/wttest.py", line 116, in check

[2022/03/24 06:33:07.211]     testcase.fail('unexpected ' + self.desc + ', contains: "' +

[2022/03/24 06:33:07.211]   File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/case.py", line 670, in fail

[2022/03/24 06:33:07.211]     raise self.failureException(msg)

[2022/03/24 06:33:07.211] AssertionError: unexpected standard output, contains: "[1648098744:872560][74426:0x1101a25c0], test_hs20.test_hs20.test_hs20(column), file:test_hs20.wt, WT_CURSOR.search: [WT_VERB_TRANSACTION][NOTICE]: oldest pinned transaction ID rolled back for eviction

[2022/03/24 06:33:07.211] "

[2022/03/24 06:33:07.211] ]

[2022/03/24 06:33:07.252] sysctl -n hw.logicalcpu

[2022/03/24 06:33:07.252] git rev-parse --show-toplevel

[2022/03/24 06:33:07.252] Command failed: command encountered problem: error waiting on process '2ef9226a-b91e-43f8-8c96-f88ee953ac96': exit status 1

[2022/03/24 06:33:07.253] Task completed - FAILURE.

[2022/03/24 06:33:07.271] Running post-task commands.

[2022/03/24 06:33:07.271] Running command 'shell.exec' in "print python stacktrace" (step 1 of 5)

[2022/03/24 06:33:07.276] if [ -d "WT_TEST" ]; then

[2022/03/24 06:33:07.277]   python3 ../test/evergreen/print_python_stack_trace.py -e python3 -c WT_TEST -l .

[2022/03/24 06:33:07.317] fi

[2022/03/24 06:33:07.317] Finished 'shell.exec' in "print python stacktrace" in 45.528919ms

[2022/03/24 06:33:07.317] Running command 'shell.exec' in "dump stderr/stdout" (step 2 of 5)

[2022/03/24 06:33:07.323] if [ -d "WT_TEST" ]; then

[2022/03/24 06:33:07.323]   # Dump stderr/stdout contents generated by the C libraries onto console for Python tests

[2022/03/24 06:33:07.323]   find "WT_TEST" -name "std*.txt" ! -empty -exec sh -c "echo 'Contents from {}:'; cat '{}'" \;

[2022/03/24 06:33:07.331] Contents from WT_TEST/test_hs20.test_hs20.test_hs20_column/stdout.txt:

[2022/03/24 06:33:07.331] [1648098744:872560][74426:0x1101a25c0], test_hs20.test_hs20.test_hs20(column), file:test_hs20.wt, WT_CURSOR.search: [WT_VERB_TRANSACTION][NOTICE]: oldest pinned transaction ID rolled back for eviction

[2022/03/24 06:33:07.332] fi

[2022/03/24 06:33:07.332] Finished 'shell.exec' in "dump stderr/stdout" in 14.7893ms

[2022/03/24 06:33:07.332] Running command 'archive.targz_pack' in "upload artifact" (step 3.1 of 5)

[2022/03/24 06:33:18.883] Finished 'archive.targz_pack' in "upload artifact" in 11.550832481s

[2022/03/24 06:33:18.883] Running command 's3.put' in "upload artifact" (step 3.2 of 5)

[2022/03/24 06:33:18.957] Putting wiredtiger.tgz into build_external/wiredtiger/macos-1014/959ce89863aba5c29048680be8659ee6af8af407/artifacts/unit-test_wiredtiger_macos_1014_959ce89863aba5c29048680be8659ee6af8af407_22_03_24_04_22_06-0.tgz (https://build_external.s3.amazonaws.com/wiredtiger/macos-1014/959ce89863aba5c29048680be8659ee6af8af407/artifacts/unit-test_wiredtiger_macos_1014_959ce89863aba5c29048680be8659ee6af8af407_22_03_24_04_22_06-0.tgz)

[2022/03/24 06:33:18.957] performing s3 put to build_external of wiredtiger/macos-1014/959ce89863aba5c29048680be8659ee6af8af407/artifacts/unit-test_wiredtiger_macos_1014_959ce89863aba5c29048680be8659ee6af8af407_22_03_24_04_22_06-0.tgz [1 of 5]

[2022/03/24 06:33:28.815] Finished 's3.put' in "upload artifact" in 9.932432913s

[2022/03/24 06:33:28.815] Running command 'archive.targz_pack' in "save wt hang analyzer core/debugger files" (step 4.1 of 5)

[2022/03/24 06:33:28.846] Finished 'archive.targz_pack' in "save wt hang analyzer core/debugger files" in 31.172331ms

[2022/03/24 06:33:28.846] Running command 's3.put' in "save wt hang analyzer core/debugger files" (step 4.2 of 5)

[2022/03/24 06:33:28.922] Putting wt-hang-analyzer.tgz into build_external/wiredtiger/macos-1014/959ce89863aba5c29048680be8659ee6af8af407/wt_hang_analyzer/wt-hang-analyzer_unit-test_wiredtiger_macos_1014_959ce89863aba5c29048680be8659ee6af8af407_22_03_24_04_22_06-0.tgz (https://build_external.s3.amazonaws.com/wiredtiger/macos-1014/959ce89863aba5c29048680be8659ee6af8af407/wt_hang_analyzer/wt-hang-analyzer_unit-test_wiredtiger_macos_1014_959ce89863aba5c29048680be8659ee6af8af407_22_03_24_04_22_06-0.tgz)

[2022/03/24 06:33:28.922] performing s3 put to build_external of wiredtiger/macos-1014/959ce89863aba5c29048680be8659ee6af8af407/wt_hang_analyzer/wt-hang-analyzer_unit-test_wiredtiger_macos_1014_959ce89863aba5c29048680be8659ee6af8af407_22_03_24_04_22_06-0.tgz [1 of 5]

[2022/03/24 06:33:28.922] file '/data/mci/72c01000a379bf4ce02ccd35a95d2289/wt-hang-analyzer.tgz' not found but skip missing true

[2022/03/24 06:33:28.923] Finished 's3.put' in "save wt hang analyzer core/debugger files" in 76.11516ms

[2022/03/24 06:33:28.923] Running command 'shell.exec' in "cleanup" (step 5 of 5)

[2022/03/24 06:33:29.221] Finished 'shell.exec' in "cleanup" in 298.336351ms

[2022/03/24 06:33:29.221] [message='Finished running post-task commands.' total_time='21.968226725s']

Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_macos_1014_unit_test_959ce89863aba5c29048680be8659ee6af8af407_22_03_24_04_22_06/0?type=T#L29760