[SERVER-6218] Profile documents should include partial query object instead of "too large" error Created: 27/Jun/12  Updated: 11/Jul/16  Resolved: 25/Nov/14

Status: Closed
Project: Core Server
Component/s: Diagnostics
Affects Version/s: 2.0.5
Fix Version/s: 2.8.0-rc2

Type: Improvement Priority: Major - P3
Reporter: Kay Agahd Assignee: J Rassi
Resolution: Done Votes: 4
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux 64 Bit


Issue Links:
Depends
Related
related to SERVER-14990 Do not omit the information about the... Closed
is related to SERVER-16324 Command execution log line displays "... Closed
Tested
Sprint: Server 2.7.3, Server 2.7.4, Query 2.7.8
Participants:

 Description   

When a profile line becomes too large, essential fields as for example "millis" are omitted:

PRIMARY> db.system.profile.findOne({err:{$exists:true},abbreviated:{$exists:false}})
{
    "ts" : ISODate("2012-06-26T04:37:06.372Z"),
    "client" : "x.x.x.x",
    "user" : "",
    "err" : "profile line too large (max is 100KB)"
}

However, in some cases there is a field "abbreviated" in which we can find the missing "millis" field:

PRIMARY> db.system.profile.findOne({err:{$exists:true},abbreviated:{$exists:true}})
{
    "ts" : ISODate("2012-06-24T19:02:46.054Z"),
    "client" : "x.x.x.x",
    "user" : "",
    "err" : "profile line too large (max is 100KB)",
    "abbreviated" : "{ ts: new Date(1340564566053), op: \"getmore\", ns: \"offerStore.offer\", query: { _id: { $in: [ 258406435, ...cut out... ] } }, cursorid: 532897853037598938, nreturned: 2708, responseLength: 4195498, millis: 5620, client: \"x.x.x.x\", user: \"\" }"
}

As the field "abbreviated" is a String, not an Object, we can't even use dot notation to reach into the object to retrieve millis. Thus, we need to use eval or a regexp to do so.

Essential information like millis, cursorid, nreturned, responseLength, client and user should ALWAYS be present in a profiling document. When the profile line is too large, then it's certainly not to large because of these fields. It's too large because of the query! So cut out or shorten the query but not the other fields!

Last but not least: What are the rules to have an "abbreviated" field or not? Btw. what has been abbreviated - it seems to be complete though?



 Comments   
Comment by Githook User [ 25/Nov/14 ]

Author:

{u'username': u'jrassi', u'name': u'Jason Rassi', u'email': u'rassi@10gen.com'}

Message: SERVER-6218 Profiler should only abbreviate query/updateobj
Branch: master
https://github.com/mongodb/mongo/commit/cf8bb50e5a8a22205e344e0536646f1ca6bc81e4

Comment by A. Jesse Jiryu Davis [ 24/Jun/14 ]

Can we ensure that $comment is also included as an essential field? Spoke today with a user who uses comments to track the source of queries. Probably similar to the technique here:

http://emptysqua.re/blog/mongo-profiling-hacks/

In this case it's important not to lose the $comment, if the $comment is short but the query is very long.

Comment by AHL Linux Support [ 26/Jul/13 ]

I have just hit the issues described here and would like to make some suggestions:

Since this is currently undocumented behaviour (http://docs.mongodb.org/manual/reference/database-profiler/) you're relatively free to change it without raising legitimate complaints.

I suggest that you either make the 'abbreviated' field contain the (abbreviated form of the) nested object rather than a string representation of it, or (better, IMHO), you simply include a boolean 'abbreviated' field in the original (but abbreviated!) profile record.

The latter would make it easier to process both abbreviated and non-abbreviated entries.

Could you also document which fields might be missing, and the circumstances under which they might be omitted: I've seen missing 'millis', 'ns', 'responseLength', even in non-abbreviated records.

Whatever you do, please document it.

Thanks,

Bob Gautier

Comment by Alex [ 13/Feb/13 ]

if (p.objsize() > 100*1024) {
    string small = p.toString(/*isArray*/false, /*full*/false);
 
    warning() << "can't add full line to system.profile: " << small;
 
    // rebuild with limited info
    BSONObjBuilder b(profileBufBuilder);
    b.appendDate("ts", jsTime());
    b.append("client", c.clientAddress() );
    if ( c.getAuthenticationInfo() )
        b.append( "user" , c.getAuthenticationInfo()->getUser( nsToDatabase( ns ) ) );
 
    b.append("err", "profile line too large (max is 100KB)");
    if (small.size() < 100*1024){ // should be much smaller but if not don't break anything
       b.append("abbreviated", small);
    }
 
    p = b.done();
}

So abbreviated field should be less than 100 KB too to be displayed

Generated at Thu Feb 08 03:11:04 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.