Should "inverse_of: nil" be restricted to HABTM?

See https://github.com/mongoid/mongoid/issues/2513

Right now, Mongoid silently allows other types of associations, such as belongs_to, to use "inverse_of: nil". This apparent misuse has been creeping into a code base I work on for some time (see also https://github.com/mongoid/mongoid/issues/1434). However, I've not noticed any negative effects. I even tried applying it to some of the associations in the models used by Mongoid's specs, and it didn't seem to cause any failures. Therefore: is it really restricted to HABTM? What is the effect when used on a belongs_to?

If it really should be restricted, should Mongoid raise an error if it sees inverse_of being used inappropriately? That would be backwards incompatible, of course.