Python3: fix modify related tests

WT-4695 covers running most of the test suite under Python3.  Tests for modify (test_cursor12.py) or calc_modify noticeably do not work with those initial changes, as the API is a little tricky, allowing both 'S' and 'u' types to be used in places.