Right now for WiredTiger.backup there's assumption that we can fprintf the keys and values, and that they won’t contain embedded newlines (i.e., we can read back the key and value in a single line). Particularly with app_metadata, which MongoDB uses, and which is complicated for indexes, this assumption is questionable.
It would be helpful to add a test case to test_alter03.py such as test_alter03_complex_metadata. The test should set up a table, and then set up a complex JSON metadata string, then alter the app_metadata with that JSON and verify it. An example of complex JSON strings can be found in test_jsondump02.py. Perhaps the code in there using bin_unicode and mixed_unicode can be used to test embedded special characters in the app_metadata.