[SERVER-13952] Cache partial BSONObj if possible (instead of {$msg: "query not recording (too large)"}) Created: 15/May/14  Updated: 10/Dec/14  Resolved: 04/Nov/14

Status: Closed
Project: Core Server
Component/s: Logging, Querying
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Kevin Pulo Assignee: Unassigned
Resolution: Duplicate Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-13935 Allow specifying profile entry and cu... Closed
related to SERVER-1794 make CurOp query's lifespan same as t... Closed
related to SERVER-7677 Limit CurOp output to fixed size Closed
related to SERVER-5605 db.currentOp() - "query" : {"$msg" : ... Closed
is related to SERVER-16324 Command execution log line displays "... Closed
Participants:

 Description   

When a CachedBSONObj is set to a BSONObj that is larger than the CachedBSONObj's fixed size buffer, the BSONObj is not copied and is instead flagged as being "tooBig". This causes the cached object to return the fixed value of

{ $msg : "query not recording (too large)" }

This is what shows up for large queries in currentOp(), profiling, and in 2.6, the logfiles.

Unfortunately, this is not particularly useful when large queries are causing performance problems. It also wastes space, because the CachedBSONObj's fixed size buffer is still present but unused.

Much better would be to copy as much of the BSONObj as possible into the CachedBSONObj's buffer.

For example, even a naive algorithm which walked the BSONObj's fields and memcpyed them one at a time until there's not enough space left in the buffer (at which point, copy in a suitable "$msg: "query truncated" field or similar). Even better would be something that can do this recursively, diving inside arrays and sub-documents and copying them (partially if necessary) until the buffer has been exhausted. Neither of these ought to add very much in the way of overheads. The pathological case would be an object with very many tiny fields, and this could be dealt with by capping the number of fields copied to, say, 100.



 Comments   
Comment by Richard Kreuter (Inactive) [ 04/Nov/14 ]

Sometimes (often?) what's relevant about a query isn't the verbatim arguments the mongod received, but the query shape that the optimizer uses. (This is useful, e.g., to classify what queries are missing indexes, for example. Cf. mloginfo query reporting feature in mtools.) Maybe printing the query shape (and a hash of the query, so it'd be possible to tell whether the app is issuing exactly the same query repeatedly) would be an improvement over this truncation. (This might be useful even when the query isn't too large, since it'd simplify log file/profiler analysis.)

Comment by Eliot Horowitz (Inactive) [ 15/May/14 ]

Full fix is in SERVER-1794

Generated at Thu Feb 08 03:33:23 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.