The WT_SESSION::compact command operates on a single table, but causes database-wide checkpoints. These repeat while compaction is making progress reducing the on-disk size of the table.
While this process does not hold any lock that directly interrupts other operations, taking database-wide checkpoints is a heavyweight operation. If multiple tables require compaction, working through all of them can be inefficient.
Instead, consider having a mode where WT_SESSION::compact marks a table as requiring compaction and returns immediately. The next time checkpoint visits the table, it can do some (preferably time-bounded) amount of compaction work. If no progress is made, the "requires compaction" flag is cleared and subsequent checkpoints will not attempt further compaction. If further compaction is required, the table should remain dirty so it is visited by subsequent checkpoints.
Some design is required to (a) determine whether the "requires compaction" flag persists in the metadata; and (b) decide how much work to do during each checkpoint to avoid impacting the performance of concurrent operations.