$vectorSearch::optimizeAt removes consecutive following sorts on "vectorSearchScore" on the assumption it is redundant. It does not check whether the $sort has already absorbed a trailing $limit into $_internalLimit.  Thus, if it removes  the $sort then the limit behavior won't apply, affecting performance.

Instead of indiscriminately removing the redundant $sort with limit, the $sort's limit should be pushed down and then on a second pass in the RBR engine, the redundant $sort stage can be removed.

A related ticket  will add a generic optimization to remove redundant $sorts that live on the $sort stage, then the $sort stage optimization itself can inspect its internals to determine if a $limit stage has been absorbed in its precondition. If so, the stage won't remove itself.