Looks like one of the upsert syntaxes is misinterpreted on the server
(seems to be a server bug rather than a shell bug, I get the same
behavior in the Java driver). Before I enter this in JIRA, since I'm
not sure of the terminology, maybe someone could let me know if it's a
known bug. My comments after the //

> db.testcoll.drop();

{"nIndexesWas" : 1 , "msg" : "all indexes deleted for collection" , "ns" : "mfs.testcoll" , "ok" : 1}

> db.testcoll.update({foo: {bar: 123}}, {"$inc": {"baz": 1}},

{upsert: true}

)
> db.testcoll.find()

{"_id" : ObjectId( "4ae725fd4666000000000f21") , "foo" :

{"bar" : 123}

, "baz" : 1}
// we upsert an object with a foo field matching exactly the value

{bar: 123}

and containing no other fields. It doesn't exist so we
create it and increment "baz".

> db.testcoll.update(

{"foo.bar": 123}

, {"$inc": {"baz": 1}},

{upsert: true}

)
> db.testcoll.find()

{"_id" : ObjectId( "4ae725fd4666000000000f21") , "foo" :

{"bar" : 123}

, "baz" : 2}
// we upsert an object with a foo field containing a bar field whose
value is 123. It exists already so "baz" is incremented.

> db.testcoll.drop()

{"nIndexesWas" : 1 , "msg" : "all indexes deleted for collection" , "ns" : "mfs.testcoll" , "ok" : 1}

> db.testcoll.update(

{"foo.bar": 123}

, {"$inc": {"baz": 1}},

{upsert: true}

)
> db.testcoll.find()

{"_id" : ObjectId( "4ae726234666000000000f22") , "foo.bar" : 123 , "baz" : 1}

// Again we upsert an object with a foo field containing a bar field
whose value is 123. It doesn't exist already. But this time the server
seems to misinterpret the query

{"foo.bar" : 123}

and creates a new
record with the field "foo.bar", rather than a new record with the
field "foo" containing a subfield "bar".