Please refer to the discussion here: https://www.mongodb.com/community/forums/t/addtoset-slower-hanging-on-large-collections-since-mongodb-6/192395/3
In short, there appear to be two issues with the SBE:
- It too aggressively resorts to disk for $group processing. In my example above, disk isn't used with the classic engine on 6.0, or on 4.0.
- When resorting to disk, it is incredibly slow.
The only solution was to force the use of the classic query engine, which doesn't seem like a desirable approach.