[SERVER-30575] Please add escaping convention for dot and dollar signs! Created: 09/Aug/17 Updated: 26/Jan/24 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | Querying |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Dmitry Kravchenko | Assignee: | Backlog - Query Optimization |
| Resolution: | Unresolved | Votes: | 45 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Assigned Teams: |
Query Optimization
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
Please add escaping convention for dot and dollar sign! You can't just prohibit some characters, especially if you claim you store JSON which allows these characters! That is incredibly bad design! Look: every other servers have escape conventions. For example, SQL has wildcard characters, but you can also use them by escaping. Imagine if they just prohibited usage of them! How did you get such an idea at all??? |
| Comments |
| Comment by Daniel Pasette (Inactive) [ 06/Jul/21 ] | |||||||||||||||||||||||||||||||||||||||||
|
This will be available on Atlas as soon as the release becomes generally available. It's not currently possible to test a release candidate on Atlas though. | |||||||||||||||||||||||||||||||||||||||||
| Comment by Santosh Kumar Aitha [ 06/Jul/21 ] | |||||||||||||||||||||||||||||||||||||||||
|
Thanks @Daniel, we are using Atlas enterprise version of MongoDB. May I know will this fix available on Atlas version already? or do we have RC for Atlas too? | |||||||||||||||||||||||||||||||||||||||||
| Comment by Daniel Pasette (Inactive) [ 02/Jul/21 ] | |||||||||||||||||||||||||||||||||||||||||
|
The 5.0 release is scheduled for release within the next few weeks – still working to confirm the exact date. You can try out the latest release candidate (5.0.0-rc7) by downloading from here: https://www.mongodb.com/try/download/community. We'd love for you to give it a try! | |||||||||||||||||||||||||||||||||||||||||
| Comment by Santosh Kumar Aitha [ 01/Jul/21 ] | |||||||||||||||||||||||||||||||||||||||||
|
Thanks Katya, When is 5.0 scheduled to be released for production? | |||||||||||||||||||||||||||||||||||||||||
| Comment by Katya Kamenieva [ 01/Jul/21 ] | |||||||||||||||||||||||||||||||||||||||||
|
The 5.0 release will include few new aggregation expressions: $getField ( Adding escaping convention is not currently scheduled.
| |||||||||||||||||||||||||||||||||||||||||
| Comment by Santosh Kumar Aitha [ 01/Jul/21 ] | |||||||||||||||||||||||||||||||||||||||||
|
is there any update on the ticket. i see it is in open state since long time. we have use-case to support it and now it impacting the customer which is forcing us to update our application to remove $ prefix from the field.
It would be great if you bump up the priority and support in coming release. | |||||||||||||||||||||||||||||||||||||||||
| Comment by Bernard Grymonpon [ 06/Apr/21 ] | |||||||||||||||||||||||||||||||||||||||||
|
I think we're hitting this issue when trying to work with the Change stream functionality. We want to retrieve only the entries in the oplog which have a certain change in a field set. The field is an embedded field.
As an example, the oplog entry comes out like this: ... We want to filter on this, we're only interested in entries which set the value to 0. This is however impossible. A simple match doesn't work: {{{"$match": { "o.$set": {"some.embedded.field": 1}}}}} or {"$match": { "o.$set.some.embedded.field": 1}} Does not give an error, but yields no results (where there should be). Trying to use `$set`/`$addFields` to get rid of the "o.$set" part by storing it on some other key yields errors ("FieldPath field names may not start with $"). This makes it impossible to filter these events server-side. We cannot use the fullDocument, as that isn't representing the state of the document when the change was made, but when the oplog is read. | |||||||||||||||||||||||||||||||||||||||||
| Comment by Katya Kamenieva [ 10/Dec/20 ] | |||||||||||||||||||||||||||||||||||||||||
|
sekhar.atmakuri@gmail.com the description in the docs indicates that you can have keys with dots or nested keys starting with $ stored in the database. "Top-level field names cannot start with the dollar sign ($) character. Otherwise, starting in MongoDB 3.6, the server permits storage of field names that contain dots (i.e. .) and dollar signs (i.e. $)." For example, you can insert them using shell insert() method or by issuing insert command (see comment from Asya above). But helpers in drivers do not allow this, as well as query language does not have convenient syntax to query these fields. We will post update to this ticket as soon as the solution gets scheduled for the specific release. | |||||||||||||||||||||||||||||||||||||||||
| Comment by Shekhar Atmakuri [ 09/Dec/20 ] | |||||||||||||||||||||||||||||||||||||||||
|
any ETA on the fix? The docs and manual says it should be fixed Mondo DB 3.6+ https://docs.mongodb.com/manual/reference/limits/#Restrictions-on-Field-Names We are on 4.0.12 but still having the issue | |||||||||||||||||||||||||||||||||||||||||
| Comment by deep sengupta [ 12/Nov/20 ] | |||||||||||||||||||||||||||||||||||||||||
|
Form 2017 to 2020 still this defect is open. No wonder why no one buys commercial license of Mongo. Great support !!!! | |||||||||||||||||||||||||||||||||||||||||
| Comment by Bruce Lucas (Inactive) [ 21/Apr/20 ] | |||||||||||||||||||||||||||||||||||||||||
|
Lack of this feature also causes problems for tooling that wants to take advantage of our new JSON logs by loading them into a db for fast querying. Many of those log statements have fields that start with "$" that can't be queried unless the logs are "sanitized" by removing or replacing "$" before loading the logs. | |||||||||||||||||||||||||||||||||||||||||
| Comment by Scott L'Hommedieu (Inactive) [ 11/Sep/19 ] | |||||||||||||||||||||||||||||||||||||||||
|
fox.bob@gmail.com There doesn't appear to be any security implication related to this behavior. AFAICT, there haven't been any security tickets open related to this behavior. If you have details that suggest this is a security issue it would be helpful to understand that. If that is the case please open a security ticket here. | |||||||||||||||||||||||||||||||||||||||||
| Comment by Robert Fox [ 03/Sep/19 ] | |||||||||||||||||||||||||||||||||||||||||
|
Does the fact that this isn't fixed also, by definition, mean that there are big gaping security holes in Mongo drivers? Agree with everyone on this thread... the fact that very popular characters are not allowed isn't great. I've also had to work around this. | |||||||||||||||||||||||||||||||||||||||||
| Comment by Satyaprakash Reddy [ 23/Apr/19 ] | |||||||||||||||||||||||||||||||||||||||||
|
Would be good to get this feature enabled as soon as possible. To overcome this, we had to encode/decode special characters in json which is hacky. | |||||||||||||||||||||||||||||||||||||||||
| Comment by Eoin [ 05/Apr/18 ] | |||||||||||||||||||||||||||||||||||||||||
|
Are there still no plans for fixing this issue? Consumers of JSON do not always have control of how it is generated. Search replace prior to import and then search replace for queries and search replace on display of matching results is a bit cumbersome. | |||||||||||||||||||||||||||||||||||||||||
| Comment by alain florencie [ 03/Apr/18 ] | |||||||||||||||||||||||||||||||||||||||||
|
My customer data having dot in names (they aware of the query issue and won't query on those) This is not the right way but until we get a fix .... it prevents the driver from generating an exception and data is inserted. | |||||||||||||||||||||||||||||||||||||||||
| Comment by Jan S. [ 23/Mar/18 ] | |||||||||||||||||||||||||||||||||||||||||
|
I can confirm this result. The following Java sample code fails using the latest driver (3.6.3)
| |||||||||||||||||||||||||||||||||||||||||
| Comment by alain florencie [ 19/Mar/18 ] | |||||||||||||||||||||||||||||||||||||||||
|
Hi i'm using the latest java driver 3.6.3 with mongodb 3.6.3 } but the driver generates a an IllegalArgumentException in the BSON library java.lang.IllegalArgumentException: Invalid BSON field name name.sub using com.mongodb.MongoCollectionImpl.insertOne | |||||||||||||||||||||||||||||||||||||||||
| Comment by Asya Kamsky [ 05/Mar/18 ] | |||||||||||||||||||||||||||||||||||||||||
|
The server does not forbid this document (as of 3.6):
Using insert command:
I believe you should be able to use insert command with any driver to bypass validation. | |||||||||||||||||||||||||||||||||||||||||
| Comment by Jan S. [ 05/Mar/18 ] | |||||||||||||||||||||||||||||||||||||||||
|
Hi Asya,
| |||||||||||||||||||||||||||||||||||||||||
| Comment by Asya Kamsky [ 02/Mar/18 ] | |||||||||||||||||||||||||||||||||||||||||
|
The documentation mentions that:
I believe the way to write such fields from drivers is via the server insert or update commands directly. While technically 3.6 server allows such writes, we still want to prevent them since querying by them will not work as expected in 3.6. Can you describe your use case so I can confirm that there is a way to handle it in 3.6 with the current driver? Asya | |||||||||||||||||||||||||||||||||||||||||
| Comment by Jan S. [ 28/Feb/18 ] | |||||||||||||||||||||||||||||||||||||||||
|
According to the 3.6 documentation the key restrictions do no longer apply to the server. However the different clients (e.g. mongo-java-driver 3.6.3) is still checking for the restrictions, making it impossible to write data to the database containing dollar signs or dots (tested using the mentioned java driver and Mongo Server 3.6.3 with featureCompatibilityVersion set to 3.6). Is there a plan to adapt them (I assume all drivers are affected) or is there an option to disable the check? | |||||||||||||||||||||||||||||||||||||||||
| Comment by David Storch [ 12/Oct/17 ] | |||||||||||||||||||||||||||||||||||||||||
|
A quick followup: We ended up deciding not to implement the $literal operator in the match language as I mentioned above. This will not be available as a partial solution in the upcoming 3.6 stable release series. But adding this feature properly is definitely still on our minds as future work! | |||||||||||||||||||||||||||||||||||||||||
| Comment by David Storch [ 15/Aug/17 ] | |||||||||||||||||||||||||||||||||||||||||
|
Hi dims, Thanks for filing this feature request, and for your interest in MongoDB! This is actually something that has been on our mind lately, although its fixVersion of "Backlog" means that we are not planning to work on it immediately for an upcoming stable release. Please note that we are working on adding a $literal operator to the match language under Best, | |||||||||||||||||||||||||||||||||||||||||
| Comment by Mark Agarunov [ 10/Aug/17 ] | |||||||||||||||||||||||||||||||||||||||||
|
Hello dims, Thank you for the report. Note that MongoDB has restrictions on the use of . and $ in the naming of fields, databases, and collections, however these characters may be used in the data. I've set the fixVersion for this ticket to "Needs Triage" for this new feature to be scheduled against our currently planned work. Updates will be posted on this ticket as they happen. Thanks, |