[SERVER-30575] Please add escaping convention for dot and dollar signs! Created: 09/Aug/17  Updated: 26/Jan/24

Status: Backlog
Project: Core Server
Component/s: Querying
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Dmitry Kravchenko Assignee: Backlog - Query Optimization
Resolution: Unresolved Votes: 45
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-61982 $merge changing structure of fields c... Backlog
is depended on by DRIVERS-618 Add escaping convention for dot and d... Closed
is depended on by DRIVERS-308 BSON Key Validation in Drivers Closed
is depended on by JAVA-2810 Allow dots and $ in field names Closed
Duplicate
is duplicated by SERVER-3943 $quote operator Closed
Related
related to SERVER-31493 JSON Schema incorrectly treats dotted... Backlog
related to SERVER-8436 Aggregation pipeline cannot process s... Backlog
related to SERVER-6768 MongoDB Profiler creates illegal docu... Closed
related to SERVER-30417 add expression to get value by keynam... Closed
is related to SERVER-10987 Disallow inserting documents with inv... Backlog
is related to NODE-1645 Bson serialisation throw error on inn... Closed
is related to NODE-1650 Node driver does not allow nested key... Closed
is related to GODRIVER-998 Driver should not allow insertion of ... Closed
is related to JAVA-151 Dot allowed in key names Closed
is related to JAVA-3101 wrong dots validation in field-name m... Closed
is related to JAVA-3452 java.lang.IllegalArgumentException: I... Closed
is related to PYTHON-1522 bson.errors.InvalidDocument key must ... Closed
is related to PYTHON-1809 pymongo insert_many ordered not working Closed
is related to PYTHON-2127 Can't insert a dict with dot in field... Closed
is related to SERVER-56462 Field starting with dollar in an embe... Closed
is related to CSHARP-2346 Allow options in driver for escaping ... Closed
is related to NODE-1649 Node driver does not allow keys conta... Closed
is related to CSHARP-435 Add a mechanism to add filters to (de... Closed
is related to CDRIVER-2045 Cannot insert keys with dots Closed
is related to SERVER-30287 Add $literal to the matcher Closed
is related to PYTHON-1982 pass invalid document to bson.errors.... Backlog
is related to SERVER-14255 Allow storing aggregation pipelines a... Backlog
is related to JAVA-3273 Ability to configure replacement for ... Closed
is related to PYTHON-31 Dot not allowed in key names Closed
is related to SERVER-1991 error message for '.' in key name is ... Closed
Assigned Teams:
Query Optimization
Participants:
Case:

 Description   

Please add escaping convention for dot and dollar sign! You can't just prohibit some characters, especially if you claim you store JSON which allows these characters! That is incredibly bad design!

Look: every other servers have escape conventions. For example, SQL has wildcard characters, but you can also use them by escaping. Imagine if they just prohibited usage of them!

How did you get such an idea at all???



 Comments   
Comment by Daniel Pasette (Inactive) [ 06/Jul/21 ]

This will be available on Atlas as soon as the release becomes generally available. It's not currently possible to test a release candidate on Atlas though.

Comment by Santosh Kumar Aitha [ 06/Jul/21 ]

Thanks @Daniel, we are using Atlas enterprise version of MongoDB. May I know will this fix available on Atlas version already? or do we have RC for Atlas too?

Comment by Daniel Pasette (Inactive) [ 02/Jul/21 ]

The 5.0 release is scheduled for release within the next few weeks – still working to confirm the exact date. You can try out the latest release candidate (5.0.0-rc7) by downloading from here: https://www.mongodb.com/try/download/community. We'd love for you to give it a try!

Comment by Santosh Kumar Aitha [ 01/Jul/21 ]

Thanks Katya, When is 5.0 scheduled to be released for production?

Comment by Katya Kamenieva [ 01/Jul/21 ]

The 5.0 release will include few new aggregation expressions: $getField (SERVER-30417), $setField (SERVER-41611), and $unsetField. You will be able to reference the field that contains "." or start with "$" using those expressions.

Adding escaping convention is not currently scheduled.

 

 

Comment by Santosh Kumar Aitha [ 01/Jul/21 ]

is there any update on the ticket. i see it is in open state since long time. we have use-case to support it and now it impacting the customer which is forcing us to update our application to remove $ prefix from the field.

 

It would be great if you bump up the priority and support in coming release. 

Comment by Bernard Grymonpon [ 06/Apr/21 ]

I think we're hitting this issue when trying to work with the Change stream functionality. We want to retrieve only the entries in the oplog which have a certain change in a field set. The field is an embedded field.

 

As an example, the oplog entry comes out like this:

...
"op" : "u",
"o": {
  "$v" : 1,
  "$set" :
{{    { "some.embedded.field" : 3302, }}
{{...   }}

We want to filter on this, we're only interested in entries which set the value to 0. This is however impossible.

A simple match doesn't work:

{{{"$match": { "o.$set":

{"some.embedded.field": 1}

}}}}

or

{"$match": { "o.$set.some.embedded.field": 1}}

 Does not give an error, but yields no results (where there should be).

Trying to use `$set`/`$addFields` to get rid of the "o.$set" part by storing it on some other key yields errors ("FieldPath field names may not start with $").  

This makes it impossible to filter these events server-side. We cannot use the fullDocument, as that isn't representing the state of the document when the change was made, but when the oplog is read.  

Comment by Katya Kamenieva [ 10/Dec/20 ]

sekhar.atmakuri@gmail.com the description in the docs indicates that you can have keys with dots or nested keys starting with $ stored in the database.

"Top-level field names cannot start with the dollar sign ($) character. Otherwise, starting in MongoDB 3.6, the server permits storage of field names that contain dots (i.e. .) and dollar signs (i.e. $)."

For example, you can insert them using shell insert() method or by issuing insert command (see comment from Asya above). But helpers in drivers do not allow this, as well as query language does not have convenient syntax to query these fields.

We will post update to this ticket as soon as the solution gets scheduled for the specific release. 

Comment by Shekhar Atmakuri [ 09/Dec/20 ]

any ETA on the fix? The docs and manual says it should be fixed Mondo DB 3.6+

https://docs.mongodb.com/manual/reference/limits/#Restrictions-on-Field-Names

We are on 4.0.12 but still having the issue

Comment by deep sengupta [ 12/Nov/20 ]

Form 2017 to 2020 still this defect is open. No wonder why no one buys commercial license of Mongo. Great support !!!!

Comment by Bruce Lucas (Inactive) [ 21/Apr/20 ]

Lack of this feature also causes problems for tooling that wants to take advantage of our new JSON logs by loading them into a db for fast querying. Many of those log statements have fields that start with "$" that can't be queried unless the logs are "sanitized" by removing or replacing "$" before loading the logs.

Comment by Scott L'Hommedieu (Inactive) [ 11/Sep/19 ]

fox.bob@gmail.com There doesn't appear to be any security implication related to this behavior.  AFAICT, there haven't been any security tickets open related to this behavior.  If you have details that suggest this is a security issue it would be helpful to understand that.  If that is the case please open a security ticket here.

Comment by Robert Fox [ 03/Sep/19 ]

Does the fact that this isn't fixed also, by definition, mean that there are big gaping security holes in Mongo drivers?  Agree with everyone on this thread... the fact that very popular characters are not allowed isn't great.  I've also had to work around this.

Comment by Satyaprakash Reddy [ 23/Apr/19 ]

Would be good to get this feature enabled as soon as possible. To overcome this, we had to encode/decode special characters in json which is hacky.

Comment by Eoin [ 05/Apr/18 ]

Are there still no plans for fixing this issue? Consumers of JSON do not always have control of how it is generated. Search replace prior to import and then search replace for queries and search replace on display of matching results is a bit cumbersome.

Comment by alain florencie [ 03/Apr/18 ]

My customer data having dot in names (they aware of the query issue and won't query on those)
i overcome the problem by overriding the driver class which is checking for dot .
It depends on how and which data you insert, in my case the driver uses class.com.mongodb.internal.validator.CollectibleDocumentFieldNameValidator
I added that class to my classpath and commented out the dot check.

This is not the right way but until we get a fix .... it prevents the driver from generating an exception and data is inserted.

Comment by Jan S. [ 23/Mar/18 ]

I can confirm this result. The following Java sample code fails using the latest driver (3.6.3)

	try (MongoClient mongoClient = new MongoClient("localhost")) {
			System.out.println(mongoClient.getDatabase("admin").runCommand(new Document("buildinfo", 1)).get("version")); // prints 3.6.3
			
			MongoDatabase database = mongoClient.getDatabase("mydb");
			MongoCollection<Document> collection = database.getCollection("collection1");
			Document doc = new Document("key", "value").append("subdocument",
					new Document("key.with.dots", "value").append("$dollar", "value"));
			
			collection.insertOne(doc);
			// Exception in thread "main" java.lang.IllegalArgumentException: Invalid BSON field name key.with.dots
			// 	at org.bson.AbstractBsonWriter.writeName(AbstractBsonWriter.java:532)
			// 	at org.bson.codecs.DocumentCodec.writeMap(DocumentCodec.java:198)
		}

Comment by alain florencie [ 19/Mar/18 ]

Hi

i'm using the latest java driver 3.6.3 with mongodb 3.6.3
i can insert key names with dots from the mongo shell like you did
> db.mytable.find()
{ "_id" : ObjectId("5aafd807fa689ef1183b2459"), "context" :

{ "name.sub" : "abc" }

}

but the driver generates a an IllegalArgumentException in the BSON library

java.lang.IllegalArgumentException: Invalid BSON field name name.sub
at org.bson.AbstractBsonWriter.writeName(AbstractBsonWriter.java:532)
at org.bson.codecs.DocumentCodec.writeMap(DocumentCodec.java:198)
at org.bson.codecs.DocumentCodec.writeValue(DocumentCodec.java:182)
at org.bson.codecs.DocumentCodec.writeMap(DocumentCodec.java:199)
at org.bson.codecs.DocumentCodec.writeValue(DocumentCodec.java:182)
at org.bson.codecs.DocumentCodec.writeMap(DocumentCodec.java:199)
at org.bson.codecs.DocumentCodec.encode(DocumentCodec.java:141)
at org.bson.codecs.DocumentCodec.encode(DocumentCodec.java:45)

using com.mongodb.MongoCollectionImpl.insertOne

Comment by Asya Kamsky [ 05/Mar/18 ]

The server does not forbid this document (as of 3.6):

(3.6.3) > db.dottedanddollars.insert([{
...     "_id" : "2bb2c149-4100-4043-9672-ef1b0ef46af3",
...     "@type" : "MyClass",
...     "indicatorName" : "Info",
...     "value" : {
...         "key1" : NumberLong(9895315),
...         "key2" : { "$binary" : "rYKpB/D7rxH5I+csvSfJSQ==", "$type" : "00" },
...         "$anotherkey" : "-",
...         "org.apache.commons" : "bla"
...     }
... }])
BulkWriteResult({
	"writeErrors" : [ ],
	"writeConcernErrors" : [ ],
	"nInserted" : 1,
	"nUpserted" : 0,
	"nMatched" : 0,
	"nModified" : 0,
	"nRemoved" : 0,
	"upserted" : [ ]
})
(3.6.3) > db.dottedanddollars.findOne()
{
	"_id" : "2bb2c149-4100-4043-9672-ef1b0ef46af3",
	"@type" : "MyClass",
	"indicatorName" : "Info",
	"value" : {
		"key1" : NumberLong(9895315),
		"key2" : {
			"$binary" : "rYKpB/D7rxH5I+csvSfJSQ==",
			"$type" : "00"
		},
		"$anotherkey" : "-",
		"org.apache.commons" : "bla"
	}
}

Using insert command:

(3.6.3) > db.runCommand({insert:"dottedanddollars", documents:[ {     "_id" : "1bb2c149-4100-4043-9672-ef1b0ef46af3",     "@type" : "MyClass",     "indicatorName" : "Info",     "value" : {         "key1" : NumberLong(9895315),         "key2" : { "$binary" : "rYKpB/D7rxH5I+csvSfJSQ==", "$type" : "00" },         "$anotherkey" : "-",         "org.apache.commons" : "bla"     } }]})
{ "n" : 1, "ok" : 1 }
(3.6.3) > db.dottedanddollars.find()
{ "_id" : "2bb2c149-4100-4043-9672-ef1b0ef46af3", "@type" : "MyClass", "indicatorName" : "Info", "value" : { "key1" : NumberLong(9895315), "key2" : { "$binary" : "rYKpB/D7rxH5I+csvSfJSQ==", "$type" : "00" }, "$anotherkey" : "-", "org.apache.commons" : "bla" } }
{ "_id" : "1bb2c149-4100-4043-9672-ef1b0ef46af3", "@type" : "MyClass", "indicatorName" : "Info", "value" : { "key1" : NumberLong(9895315), "key2" : { "$binary" : "rYKpB/D7rxH5I+csvSfJSQ==", "$type" : "00" }, "$anotherkey" : "-", "org.apache.commons" : "bla" } }

I believe you should be able to use insert command with any driver to bypass validation.

Comment by Jan S. [ 05/Mar/18 ]

Hi Asya,
I have a collection of documents with a basic fixed structure and one variable element (see example below).
The variable element - the "value" key can hold arbitrary JSON documents and therefore JSON documents starting with" $" or contain dots always make problems.
I assume "value" is in this case what the documentation calls a "nested field", right?

{
    "_id" : "2bb2c149-4100-4043-9672-ef1b0ef46af3",
    "@type" : "MyClass",
    "indicatorName" : "Info",
    "value" : {
        "key1" : NumberLong(9895315),
        "key2" : { "$binary" : "rYKpB/D7rxH5I+csvSfJSQ==", "$type" : "00" },
        "$anotherkey" : "-",
        "org.apache.commons" : "bla"
    }
}

Comment by Asya Kamsky [ 02/Mar/18 ]

mango

The documentation mentions that:

Although valid, avoid using . or $ in field names since querying on such fields is not yet functional.

I believe the way to write such fields from drivers is via the server insert or update commands directly.

While technically 3.6 server allows such writes, we still want to prevent them since querying by them will not work as expected in 3.6.

Can you describe your use case so I can confirm that there is a way to handle it in 3.6 with the current driver?

Asya

Comment by Jan S. [ 28/Feb/18 ]

According to the 3.6 documentation the key restrictions do no longer apply to the server. However the different clients (e.g. mongo-java-driver 3.6.3) is still checking for the restrictions, making it impossible to write data to the database containing dollar signs or dots (tested using the mentioned java driver and Mongo Server 3.6.3 with featureCompatibilityVersion set to 3.6).

Is there a plan to adapt them (I assume all drivers are affected) or is there an option to disable the check?

Comment by David Storch [ 12/Oct/17 ]

A quick followup: We ended up deciding not to implement the $literal operator in the match language as I mentioned above. This will not be available as a partial solution in the upcoming 3.6 stable release series. But adding this feature properly is definitely still on our minds as future work!

Comment by David Storch [ 15/Aug/17 ]

Hi dims,

Thanks for filing this feature request, and for your interest in MongoDB! This is actually something that has been on our mind lately, although its fixVersion of "Backlog" means that we are not planning to work on it immediately for an upcoming stable release.

Please note that we are working on adding a $literal operator to the match language under SERVER-30287. This will provide a partial solution for expressing queries over documents that have field names with "." or "$". (We are interested in this partial solution since it is necessary for some unrelated enhancements we are making to the $lookup aggregation stage.) It also functions similarly to the $literal aggregation operator, which has existed for some time.

Best,
Dave

Comment by Mark Agarunov [ 10/Aug/17 ]

Hello dims,

Thank you for the report. Note that MongoDB has restrictions on the use of . and $ in the naming of fields, databases, and collections, however these characters may be used in the data.

I've set the fixVersion for this ticket to "Needs Triage" for this new feature to be scheduled against our currently planned work. Updates will be posted on this ticket as they happen.

Thanks,
Mark

Generated at Thu Feb 08 04:24:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.