Create a hierarchy for incompatible settings in test/format

Summary
test/format verifies incompatible settings when generating/parsing a configuration. The hierarchy was partially removed through WT-9771.

As part of this ticket, one should document this hierarchy which is defined in the config_transaction function present in the test/format/config.c file. Once this hierarchy is defined, this may impact the changes done in WT-9771.

Motivation

• Does this affect any team outside of WT?
  No

• Is this issue urgent?
  Yes to improve our testing coverage.

Acceptance Criteria (Definition of Done)

• Testing
  Make sure test/format does not generate configurations with incompatible settings.
• Documentation update
  Add a comment to the code explaining the chosen hierarchy.