[JAVA-2810] Allow dots and $ in field names Created: 14/Mar/18  Updated: 27/Oct/19  Resolved: 12/Mar/19

Status: Closed
Project: Java Driver
Component/s: Command Operations
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Alex Vayda Assignee: Scott L'Hommedieu (Inactive)
Resolution: Won't Fix Votes: 7
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-30575 Please add escaping convention for do... Backlog
Duplicate
duplicates JAVA-3273 Ability to configure replacement for ... Closed
is duplicated by JAVA-3101 wrong dots validation in field-name m... Closed
Related
Case:

 Description   

MongoDB 3.6 allows using dots and dollar signs in the field names.
https://docs.mongodb.com/manual/reference/limits/#Restrictions-on-Field-Names

Please remove this restriction from CollectibleDocumentFieldNameValidator



 Comments   
Comment by Oz Madar [ 13/Mar/19 ]

Thank you scott.lhommedieu for the workaround

Comment by Scott L'Hommedieu (Inactive) [ 12/Mar/19 ]

We'll deal with this in DRIVERS-618.  For the time being we won't do anything in individual drivers.

Comment by Scott L'Hommedieu (Inactive) [ 12/Mar/19 ]

Created DRIVERS-618 to track how we handle this at a spec/drivers level.

Comment by Scott L'Hommedieu (Inactive) [ 12/Mar/19 ]

Hi Oz,

There is a way to way to bypass these limitations on the write by using run command syntax from drivers.  Here for Java.
There are also some ways, noted here, to query this data with aggregations.

Until the whole of the MongoDB platform is geared toward supporting this we do not want to encourage this in Drivers.

Comment by Oz Madar [ 17/Jan/19 ]

Thanks Scott, yet I don't understand why not allowing "power-users" that choose to skip this validation? 

Our data-model contains dots in fields names, the result of protecting us is that I have to modify the data model.

By the way, I cloned driver-core git repo, deleted the validation, compiled it and it works fine for me. 

I would be very happy if I could cancel that validation with some simple configuration (env, java_opts etc.) and it will not have any effect on other parts 

(I not asking for queries capabilities, just not preventing an allowed insert behavior)

Comment by Scott L'Hommedieu (Inactive) [ 17/Jan/19 ]

My apologies, I should have been more clear and accurate when I stated "usable way". 

The server has not adequately defined behavior regarding reading and writing . and $ in field names such that drivers can make a clear and lasting decision on an API to expose that behavior.  If we make these decisions in drivers now the drivers would provide an interface that would likely be up for deprecation as soon as the server does implement this behavior and even worse would be susceptible to breaking changes from server changes.

Also, there are currently options available to write and read values with keys that contain . and $.  These options aren't as simple to use as we would like (i.e. no helper methods).  But, this is also because the server support for these characters in field names is not complete.  See: SERVER-30575 and SERVER-30417

Rather than add driver functionality that may be invalidated by the results of those server changes we opt to continue to disallow those in drivers, wait for the changes in server and advocate that users try the options detailed in the server tickets.

Comment by Alex Vayda [ 09/Jan/19 ]

Not sure I'm following. What is your definition of a usable way? Lacking ability to query on fields with dots doesn't mean storing those fields isn't usable. If I get it right the main concern here is to be able just to store and get. Isn't is a valid case in your opinion? To me it looks like a unilateral decision for forbidding this functionality in the driver has already been made in the first place.

Comment by Oz Madar [ 09/Jan/19 ]

@Scott L'Hommedieu, The server supports storing such documents, the problem is only in querying. Preventing insert from on the driver side is changing the declared behaviour of MongoDB.

What about making it configurable, e.g by adding some JVM_OPTS that will allow the user disabling this validation? 

Comment by Scott L'Hommedieu (Inactive) [ 09/Jan/19 ]

We don't want to make a unilateral decision in any drivers for behavior of all query functionality.  Server does not support this in a usable way.  This is blocked on SERVER-30575

Comment by Bastien Arata [ 31/Aug/18 ]

@Michael Everett During my debugging, I found the flag in the Node.js driver, but it'is not documented. Flag is checkKeys, you can see an exemple on this comment. Maybe this hidden flag exists in the Java driver too ?

Comment by Michael Everett [ 23/Aug/18 ]

@BHackett This seems like an overly restrictive approach to solving this problem. The reality is, there are many very valid reasons to want to store fields that contains these characters without ever wanting to query against them (storing JSON that you haven't generated, storing JsonSchema are two that come to mind). I'd imagine that these are the most common use cases for wanting fields containing these characters.

If the server is supporting these characters, at the very least it seems the driver should include a flag to disable this validation. 

 

Comment by Terence Bodola [ 07/Aug/18 ]

I'm sticking data into Mongo using Java in one environment and then pulling whole docs from collections by an indexed root property (rocket fast) and managing grouping/aggregation in JavaScript code in Node.JS in another environment, using bracket instead of dot notation (also rocket fast ...didn't do in Mongo because a bit complex of an operation for agg pipeline and dead simple in custom code).

I get what you are saying, but there is definitely a use case for just being able to get the data in using Java at least.

 

Comment by Bernie Hackett [ 07/Aug/18 ]

Though dots and dollar signs are now allowed in keys by the server, there is still no way to unambiguously query on such fields. You can put the data in, but you may have a very hard time getting it back out. Until that problem is solved in the server we'll still be rejecting these keys in drivers.

Comment by Terence Bodola [ 07/Aug/18 ]

https://github.com/mongodb/mongo-java-driver/pull/480

Generated at Thu Feb 08 08:58:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.