[SERVER-6768] MongoDB Profiler creates illegal documents Created: 14/Aug/12  Updated: 06/Dec/22  Resolved: 19/Oct/20

Status: Closed
Project: Core Server
Component/s: Diagnostics
Affects Version/s: 2.0.4, 2.6.4, 2.7.6
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Sean Coates Assignee: Backlog - Query Team (Inactive)
Resolution: Done Votes: 1
Labels: profiling
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux (Debian). N/A, really.


Issue Links:
Depends
Duplicate
is duplicated by SERVER-7349 Queries with dotted notation create i... Closed
Related
is related to SERVER-13245 system.profile query format inconsistent Backlog
is related to SERVER-30575 Please add escaping convention for do... Backlog
is related to SERVER-6767 Interpret $query as special so you ca... Closed
Assigned Teams:
Query
Participants:

 Description   

The DB profiler creates illegal documents. These documents contain:
1) keys with $ in the name
2) keys with . in the name

This problem makes working with these documents incredibly difficult.

Here's an example:

PRIMARY> test.record = db.system.profile.findOne({"ts" : ISODate("2012-08-12T14:26:18.806Z")})
{
	"ts" : ISODate("2012-08-12T14:26:18.806Z"),
	"op" : "query",
	"ns" : "gimmebar.collectionassets",
	"query" : {
		"$query" : {
			"collection_id" : "redacted"
		},
		"$orderby" : {
			"date" : -1
		}
	},
	"ntoreturn" : 5,
	"nscanned" : 169,
	"scanAndOrder" : true,
	"nreturned" : 5,
	"responseLength" : 325,
	"millis" : 227,
	"client" : "redacted",
	"user" : ""
}
 
PRIMARY> db.tmp_profiler_test.insert(testrecord)
Tue Aug 14 18:39:40 uncaught exception: field names cannot start with $ [$query]

The same thing happens for $orderby, $or, $set, etc., as well as for keys with "." in the name.

I think the profiler should not create documents that can't be inserted. (I had to insert them into another collection because I wanted to filter the profiler data set, but I can't delete from a capped collection, which db.system.profile is.)

I realize this changes the data format of the profiler, but it seems like the right move to me.

As a workaround, for those developers who might stumble upon this, I managed to get the following transposition to work, but it's far from ideal:

// recursion is fun
function rename$(r) {
    for (k in r) {
        var changed = false;
        var newk = k;
        if (newk.indexOf(".") != -1) {
            changed = true;
            newk = newk.replace(".", "_DOT_", "g");
        }
        if (newk.indexOf("$") != -1) {
            changed = true;
            newk = newk.replace("$", "_DOLLAR_", "g");
        }
        if (changed) {
            r[newk] = r[k];
            delete r[k];
        }
        if (typeof r[newk] == "object") {
            r[newk] = rename$(r[newk]);
        }
    }
    return r;
}

db.tmp_profiler_test.drop();
db.system.profile.find(query_parameters).forEach(function(r) { r = rename$(r); db.tmp_profiler_test.insert(r);})

(I also couldn't get this box to respect wiki markup, so editor: feel free to fix.) [Done.]



 Comments   
Comment by Katya Kamenieva [ 26/Oct/20 ]

Thanks for more details, david.storch, I linked SERVER-30575 to this ticket. As you said, querying fields with "." and "$" should be improved, and there is no plan to stop using "." and "$" in the fields of system.profile.

Comment by David Storch [ 26/Oct/20 ]

kateryna.kamenieva, one clarification: I believe that the server still rejects the "." and "$" characters in some scenarios. We have open tickets which would remove those guardrails in more cases (see SERVER-49473, SERVER-49474, and SERVER-49117). However, I think in the simple case of inserts, we currently will only reject insertion of top-level $-prefixed fields. Dollar-prefixed nested fields are allowed, and the "." character is always allowed:

MongoDB Enterprise > db.c.insert({$dollarPrefixed: 1})
WriteResult({
	"nInserted" : 0,
	"writeError" : {
		"code" : 2,
		"errmsg" : "Document can't have $ prefixed field names: $dollarPrefixed"
	}
})
MongoDB Enterprise > db.c.insert({obj: {$dollarPrefixed: 1}})
WriteResult({ "nInserted" : 1 })
MongoDB Enterprise > db.c.insert({"dotted.name": 1})
WriteResult({ "nInserted" : 1 })
MongoDB Enterprise > db.c.insert({obj: {"dotted.name": 1}})
WriteResult({ "nInserted" : 1 })
MongoDB Enterprise > db.c.find()
{ "_id" : ObjectId("5f975341fe41dc9c72f347d9"), "obj" : { "$dollarPrefixed" : 1 } }
{ "_id" : ObjectId("5f97534dfe41dc9c72f347da"), "dotted.name" : 1 }
{ "_id" : ObjectId("5f975358fe41dc9c72f347db"), "obj" : { "dotted.name" : 1 } }

Is the reasoning for closing this ticket that, due the schema of the system.profile collection, it is impossible to create a document in the profile collection which has a $-prefixed top-level field?

Regardless of the answer to this question, I do agree that it makes sense to close this ticket and instead improve the query language so that it can express queries over documents with "$" and "." in the keyname (tracked by SERVER-30575).

Comment by Katya Kamenieva [ 19/Oct/20 ]

Starting 3.6 field names with "." and "$" are allowed. Here is section in docs.

Generated at Thu Feb 08 03:12:39 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.