[SERVER-18902] Retrieval of documents larger that 1MB slower on WiredTiger than MMAPv1 Created: 09/Jun/15  Updated: 19/Sep/15  Resolved: 12/Jun/15

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.0.3
Fix Version/s: 3.0.5, 3.1.5

Type: Bug Priority: Critical - P2
Reporter: James Wahlin Assignee: Daniel Pasette (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
related to SERVER-20197 find() on collection with large docum... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Steps To Reproduce:

1) Setup a 2 member MongoDB 3.0.3 replica set with:

  • One member running WiredTiger
  • One member running MMAPv1

2) Load a data set with large documents:

use test;
var hugeString="";
for (var i = 0; i < (1024 * 1100); i++) { hugeString = hugeString + "12345"; }
for (var j = 0; j < 1000; ++j) { db.test.insert({a: hugeString}); }

3) Run the following explain 2 times against both the WiredTiger and the MMAPv1 members:

use test;
db.test.find().explain(verbose='executionStats');

Sprint: Quint Iteration 5
Participants:

 Description   
Issue Status as of Jul 14, 2015

ISSUE SUMMARY
WiredTiger does not store documents larger than its default leaf_value_max in the in-memory cache. The leaf_value_max value is 1 megabyte. As a result, operations that read or modify larger documents are significantly less efficient than with smaller documents.

Starting with MongoDB 3.0.5 all documents are stored in the in-memory WritedTiger cache, but this change only impacts collections created after upgrading to 3.0.5 or later. Existing collections are unaffected by this change.

USER IMPACT
Users running with the WiredTiger storage engine may observe slower performance when retrieving documents larger than 1 megabyte than when retrieving smaller documents.

WORKAROUNDS
Users with documents larger than 1 megabyte may use the --wiredTigerCollectionConfigString 'leaf_value_max=<bytes>' configuration option to increase the value of leaf_value_max beyond 1 megabyte to improve performance when accessing large documents.

For example, to ask WiredTiger to put in its cache documents of up to 16 megabytes in size, use:

--wiredTigerCollectionConfigString 'leaf_value_max=16000000'

Note that this setting only impacts collections created after the change. Existing collections are not be affected by this change, so users with existing collections containing large documents may want to move them to a newly created collection, replicate them to new instances running 3.0.5, or perform a dump + restore cycle on these collections.

AFFECTED VERSIONS
MongoDB 3.0.0 through 3.0.4.

FIX VERSION
The fix is included in the 3.0.5 production release.

Original description

Retrieval of large documents under MongoDB 3.0.3 and WiredTiger is significantly slower than 3.0.3 running MMAPv1. With the following reproduction on my macbook, I saw the following execution times, running the find() operation twice to account for cold data:

Engine First find Second find
MMAPv1 10ms 0ms
WiredTiger 5950ms 5921ms

Increasing the leaf_value_max setting to 64000000 on mongod prior to collection creation improves performance:

Engine First find Second find
WiredTiger 1803ms 590ms


 Comments   
Comment by Githook User [ 16/Jun/15 ]

Author:

{u'username': u'monkey101', u'name': u'Dan Pasette', u'email': u'dan@10gen.com'}

Message: SERVER-18902 Bump leaf_value_max to 64MB to keep large docs in WiredTiger cache

(cherry picked from commit f32226bc1c8489a7cfff2ba2242c98b3f2efb16b)
Branch: v3.0
https://github.com/mongodb/mongo/commit/17678f50406d674fb2b3a6bcdcd00f72ae8eacc1

Comment by Githook User [ 12/Jun/15 ]

Author:

{u'username': u'monkey101', u'name': u'Dan Pasette', u'email': u'dan@10gen.com'}

Message: SERVER-18902 Bump leaf_value_max to 64MB to keep large docs in WiredTiger cache
Branch: master
https://github.com/mongodb/mongo/commit/f32226bc1c8489a7cfff2ba2242c98b3f2efb16b

Generated at Thu Feb 08 03:49:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.