We run into a cases when serializer finder is trying to deduce serializers based on the IUnknowableSerializer or IIgnoreNodeSerializer, because IsKnown method returns true in a node has such serializers assigned. Also SerializerFinder uses IsKnown method to determine if the node was visited or not.
I've patched this logic in CSHARP-5929, but we'd be better to implement separate methods on SerializerMap so the SerializerFinder can clearly distinguish when the node was visited and if it has meaningful serializer.