Currently each node in the tree, including leaf pointers, has 256 shared pointers. Each shared pointer is 16 bytes, for a total of 4kB for the pointers. As the first allocation size > 4kB is 6 kB, and each document results in 2 nodes (one for index, one for record store), each document has 12kB overhead (18 kB for replicasets due to the oplog).
This causes each Node to be > 4kB resulting in excessive memory usage during testing and OOMs.
Change the Node representation to avoid shared pointers in leaf nodes.