From the MongoDB Limits and Thresholds docs page:

Nested Depth for BSON Documents

MongoDB supports no more than 100 levels of nesting for BSON documents.

The server should fail insertion of these documents with an error returned to the user. This would make the server's behavior consistent with the documentation.

Furthermore, if code in the server can assume that the nesting limit is fairly small, and the server can assert that the stack is of some known minimum size at time of process start, then recursive algorithms in the server can executed safely on these documents without concern of stack overflow.

To reproduce, run the following shell snippet.

function makeNestObj(depth){
    toret = {s : 1};
    for(i = 1; i < depth; i++){
        toret = {s : toret};
    }
    return toret;
}
db.foo.drop();
assert.writeError(db.foo.insert(makeNestObj(101)));  // Currently fails, should pass when this ticket is resolved.