[SERVER-5674] Updates with $ sign in key name fail Created: 21/Apr/12  Updated: 09/Jul/16  Resolved: 27/Jul/13

Status: Closed
Project: Core Server
Component/s: Usability, Write Ops
Affects Version/s: 2.0.1, 2.0.4, 2.1.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Peter Uhrig Assignee: Scott Hernandez (Inactive)
Resolution: Duplicate Votes: 2
Labels: update, validation
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Tested on 2.0.1 on 64bit Linux and on 2.0.4 on 32bit Windows XP


Issue Links:
Duplicate
duplicates SERVER-7557 Update doesn't support "$"s in the fi... Closed
Operating System: ALL
Participants:

 Description   

db.test.insert(

{"_id": "someid", "attribute": "somevalue"}

);
db.test.update(

{"_id" : "someid"}

,{$set: {"pos_list":

{"PRP$": "PRP$ - Possessive pronoun"}

}});

The second line returns the following error:
not okForStorage

Interestingly, inserting the same attribute-value pair works:
db.test.insert(

{"_id": "someotherid", "PRP$": "PRP$ - Possessive pronoun"}

);



 Comments   
Comment by Jullian Chavez [ 26/Aug/12 ]

Ok thanks for the help. It makes sense that it's the driver causing my problem. And I agree, this should be a server-level enforcement. Oh well, guess it's time to write something to recursively rename the youtube field names as there are a TON of them. Thanks again.

Comment by Glenn Maynard [ 26/Aug/12 ]

You should never be able to insert keys that start with a $. It seems like this is enforced at the driver level, which is a bit strange; it should be enforced at the server level. Doing it in drivers just duplicates work and creates inconsistent behavior.

This bug is about not being able to perform updates when keys contain a non-leading $, which is valid. For example, while {$PRP: 1} is illegal,

{PRP$: 1}

is perfectly legal. You may be hitting a similar bug in the other direction, with your driver allowing you to insert "$t" when it shouldn't. (This is all fallout from the original problem, IMO: this should all be enforced uniformly at the server, not in each driver.)

Also note that there are other cases of JSON that you can't store directly in Mongo. In particular, Mongo keys can't contain ".", which is valid in JSON. If you want to store external data that comes from JSON, you'll need to scan through and escape $ and . in keys. Otherwise, even if your driver accidentally allows it, you're liable to get bitten later on if Mongo adds (say) a "$t" operator.

Comment by Jullian Chavez [ 26/Aug/12 ]

Let me clarify. I'm using the Codeigniter framework (PHP) to build my apps. One of the apps connects to Youtube's API to collect information about a user's videos.

The response from Youtube looks something along the lines of: {"title":{"$t":"My Video Title"}} (obviously the whole response has a lot more data, but this bit is all that matters). I then turn this json-encoded string into a PHP array and add the video's id:

Array(
[videoid]=>725fuiZTVOA
[title]=>Array(
[$t] => My Video Title
)
)

I then store the entire thing in mongodb using a library (PHP class) written for codeigniter:
$mongo_db->insert('videos',$yt_data); //Where 'videos' is the collection, and $yt_data is the array shown above

The equivalent of this in the mongo shell would be:
db.videos.insert({videoid:"725fuiZTVOA",title:{$t:"My Video Title"}});

Interestingly enough, using the PHP class to insert this data works perfectly fine, while running the shell command above gives the error: field names cannot start with $ [$t]. Not sure how the PHP class is getting by this, but mongo seems to have no issue adding fields containing a "$".

Now, on to why this is relevant to this thread:

When I use the same PHP class to UPDATE the document i just made, i get the error "not okForStorage" by using this command:
$mongo_db->where(array("videoid"=>"725fuiZTVOA"))>set(array("title"=>array("$t"=>"My New Video Title")))>update('videos');

The equivalent of which in the mongo shell would be:
db.videos.update(

{videoid:"725fuiZTVOA"}

,{$set:{title:{$t:"My New Video Title"}}});

In the end, my entire question is why would I be able to insert fields whose names contain a "$", but not update them? The only reason i am trying to do this is because i'm trying to store the ENTIRE Youtube API response, many field of which contain a "$" for whatever reason. Again, for some reason i can insert this data, but not update it.

Comment by Glenn Maynard [ 26/Aug/12 ]

Jullian: Are you storing "{'title':{'$t':'My Youtube Video'}}" as a JSON-encoded string, or are you decoding it and attempting to store it as a Mongo structure directly? It looks like you're decoding it, because JSON uses double-quotes for strings, not single-quotes. You can't use keys starting with $, eg. "$t"; they need to be escaped manually. That's not a bug, but I'm not quite sure what you're doing, since Mongo won't let you insert it, either.

(That would be related to this bug if you were using the literal string "{'title':{'$t':'My Youtube Video'}}" as a long document key, which is possible but seems unlikely...)

In any case, remember that you can always use escaping to avoid inserting '$' in documents in any place they're causing issues.

Comment by Jullian Chavez [ 25/Aug/12 ]

This is also causing a HUGE project-stopping issue. Youtube's API returns a JSON string with many keys containing a "$" in them. Storing this data into Mongo is not a problem the first time you insert the data, but when you try to update it, it returns "not okForStorage".
Example Youtube JSON string: {'title':{'$t':'My Youtube Video'}}

Comment by Antonio Lobato [ 22/Aug/12 ]

I've just run into this bug as well and is a major show stopper. Using the Ruby driver, the following works:

@data.update(

{'_id' => id}, data, {:upsert => true, :safe => true})

While this returns not okForStorage:

data.each do |name, mdata|
@data.update({'_id' => id}

, {'$set' => {name => mdata}}, {:upsert => true, :safe => true})
end

Comment by Peter Uhrig [ 21/Apr/12 ]

It also causes no problems within a list:
db.pipeline.update(

{"_id" : "someid"}

,{$set: {"pos_list": [

{"PRP": "PRP - Personal pronoun"}

,

{"PRP$": "PRP$ - Possessive pronoun"}

,

{"RB": "RB - Adverb"}

]}});

Comment by Scott Hernandez (Inactive) [ 21/Apr/12 ]

full trace

Sat Apr 21 06:00:06 [conn2] User Assertion: 12527:not okForStorage
Sat Apr 21 06:00:06 [conn2] warning: DBException thrown :: caused by :: 12527 not okForStorage
0x10034e96d 0x10034eeda 0x1000ceaad 0x1000cfb17 0x1000df856 0x1000dd508 0x1000bf577 0x1000c2d4b 0x1000c6ce5 0x10035acbe 0x100360519 0x10001fdd4 0x100458a73 0x100671b84 0x7fff81de9fd6 0x7fff81de9e89
 0   mongod                              0x000000010034e96d _ZN5mongo11DBException13traceIfNeededERKS0_ + 333
 1   mongod                              0x000000010034eeda _ZN5mongo9uassertedEiPKc + 170
 2   mongod                              0x00000001000ceaad _ZNK5mongo3Mod18_checkForAppendingERKNS_11BSONElementE + 157
 3   mongod                              0x00000001000cfb17 _ZN5mongo11ModSetState16appendNewFromModINS_14BSONObjBuilderEEEvRNS_8ModStateERT_ + 695
 4   mongod                              0x00000001000df856 _ZN5mongo11ModSetState18_appendNewFromModsINS_14BSONObjBuilderEEEvRKSsRNS_8ModStateERT_RSt3setISsSt4lessISsESaISsEE + 1222
 5   mongod                              0x00000001000dd508 _ZN5mongo11ModSetState17createNewFromModsINS_14BSONObjBuilderEEEvRKSsRT_RKNS_7BSONObjE + 1752
 6   mongod                              0x00000001000bf577 _ZN5mongo11ModSetState17createNewFromModsEv + 295
 7   mongod                              0x00000001000c2d4b _ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES2_bbbRNS_7OpDebugEPNS_11RemoveSaverE + 3819
 8   mongod                              0x00000001000c6ce5 _ZN5mongo13updateObjectsEPKcRKNS_7BSONObjES2_bbbRNS_7OpDebugE + 309
 9   mongod                              0x000000010035acbe _ZN5mongo14receivedUpdateERNS_7MessageERNS_5CurOpE + 1038
 10  mongod                              0x0000000100360519 _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 3993
 11  mongod                              0x000000010001fdd4 _ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE + 148
 12  mongod                              0x0000000100458a73 _ZN5mongo3pms9threadRunEPNS_13MessagingPortE + 691
 13  mongod                              0x0000000100671b84 thread_proxy + 132
 14  libSystem.B.dylib                   0x00007fff81de9fd6 _pthread_start + 331
 15  libSystem.B.dylib                   0x00007fff81de9e89 thread_start + 13
Sat Apr 21 06:00:06 [conn2] update test.a update: { $set: { c: { some$: "val$" } } } nscanned:1 keyUpdates:0 exception: not okForStorage code:12527 1ms
Sat Apr 21 06:00:06 [conn2] runQuery called test.$cmd { getlasterror: 1.0, w: 1.0 }

Generated at Thu Feb 08 03:09:35 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.