[SERVER-15017] benchRun might return incorrect stats values Created: 25/Aug/14  Updated: 19/Sep/15  Resolved: 17/Jun/15

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: 2.6.4, 2.7.6
Fix Version/s: 3.0.5, 3.1.5

Type: Bug Priority: Major - P3
Reporter: Davide Italiano Assignee: David Daly
Resolution: Done Votes: 0
Labels: 28qa
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Backwards Compatibility: Minor Change
Operating System: ALL
Backport Completed:
Sprint: CAP - 3.1.5
Participants:

 Description   

benchRun results might be slightly incorrect as long as the tool relies on serverStatus() calls rather than collecting stats client side. For the same reason (stats collected server side) two parallel executions of benchRun won't return per-client results (but cumulative results).

As an added bonus, scotthernandez noticed that if the test lasts < 1 second, the results returned are completely broken, as showed in this testcase (provided by him).

$ mongo --eval 'printjson(benchRun( { ops : [{op:"command", ns:"test",
command:{ping:1}}] , parallel : 1 , seconds : .001 , host : db.getMongo().host }
))'
MongoDB shell version: 2.6.1
connecting to: test
{
...	"command" : 9000
}



 Comments   
Comment by Githook User [ 02/Jul/15 ]

Author:

{u'username': u'dalyd', u'name': u'dalyd', u'email': u'david.daly@mongodb.com'}

Message: Changes to handle benchrun result changes from SERVER-15017

(cherry picked from commit a446b28f19dad6fb8a2c7690946b6a187bd9e8bb)
Branch: develop
https://github.com/mongodb/mongo-perf/commit/b4be1f8638890fe4318bea5473ebd323e86f2253

Comment by Githook User [ 18/Jun/15 ]

Author:

{u'username': u'dalyd', u'name': u'dalyd', u'email': u'david.daly@mongodb.com'}

Message: SERVER-15017: benchRun might return incorrect stats values

(cherry picked from commit 532571b5bc3db0560f41ad5e60a748ddd1782c85)

Conflicts:
src/mongo/shell/bench.cpp
Branch: v3.0
https://github.com/mongodb/mongo/commit/42ad90b1d4b555e25ca19210bbf60a8037d80e10

Comment by David Daly [ 17/Jun/15 ]

Changes the results returned by benchrun. Mongo-perf updated to handle the change.

Comment by Githook User [ 17/Jun/15 ]

Author:

{u'username': u'dalyd', u'name': u'dalyd', u'email': u'david.daly@mongodb.com'}

Message: SERVER-15017: benchRun might return incorrect stats values
Branch: master
https://github.com/mongodb/mongo/commit/532571b5bc3db0560f41ad5e60a748ddd1782c85

Comment by Githook User [ 17/Jun/15 ]

Author:

{u'username': u'dalyd', u'name': u'dalyd', u'email': u'david.daly@mongodb.com'}

Message: Changes to handle benchrun result changes from SERVER-15017

(cherry picked from commit a446b28f19dad6fb8a2c7690946b6a187bd9e8bb)
Branch: evergreen
https://github.com/mongodb/mongo-perf/commit/527eb3a9c22d4ef244cd7b729c75d53d41487255

Comment by Githook User [ 17/Jun/15 ]

Author:

{u'username': u'dalyd', u'name': u'dalyd', u'email': u'david.daly@mongodb.com'}

Message: Changes to handle benchrun result changes from SERVER-15017

(cherry picked from commit a446b28f19dad6fb8a2c7690946b6a187bd9e8bb)
Branch: master
https://github.com/mongodb/mongo-perf/commit/b4be1f8638890fe4318bea5473ebd323e86f2253

Comment by Githook User [ 17/Jun/15 ]

Author:

{u'username': u'dalyd', u'name': u'dalyd', u'email': u'david.daly@mongodb.com'}

Message: Changes to handle benchrun result changes from SERVER-15017
Branch: develop
https://github.com/mongodb/mongo-perf/commit/a446b28f19dad6fb8a2c7690946b6a187bd9e8bb

Generated at Thu Feb 08 03:36:41 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.