When running in in-memory mode, logging is turned off. However, rollback_to_stable uses logging knowledge to avoid rolling back tables that should not be, like the oplog and local/private tables. WT uses wt_btree_immediately_durable (returning true) to decide to skip rolling back a btree like the oplog.

That function only returns true if logging on the database is turned on, and logging on the table is turned on. This function needs updating (as the comment in that function suggests) to support not rolling back the oplog and other local/private tables when database-logging is not enabled.