[SERVER-22078] Remove term list limits for text index in FCV 4.2 Created: 06/Jan/16  Updated: 06/Jul/21  Resolved: 13/Aug/18

Status: Closed
Project: Core Server
Component/s: MMAPv1, Text Search
Affects Version/s: 3.0.1, 3.1.0
Fix Version/s: 4.1.2

Type: Task Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Xiangyu Yao (Inactive)
Resolution: Done Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Documented
is documented by DOCS-11991 Docs for SERVER-22078: Remove term li... Closed
Related
is related to SERVER-8431 Large update to text index can cause ... Closed
is related to SERVER-36440 Completely remove code and tests abou... Closed
is related to SERVER-17501 Increase journalling capacity limits Closed
is related to SERVER-3372 Allow indexing fields of arbitrary le... Closed
Backwards Compatibility: Fully Compatible
Sprint: Storage NYC 2018-08-13, Storage NYC 2018-08-27
Participants:
Case:

 Description   

SERVER-8431 restricted inserting a document into a collection with a text index under the following circumstances:

  • The number of unique terms in the fields indexed by the text index is greater than 400,000.
  • The total size of those terms is greater than 4MB.

This was done to avoid causing the server to shut down from generating a group commit larger than 512MB. However, we have since increased the journaling capacity limit to 2000MB as part of SERVER-17501. We should considering raising these limits to allows users to store more text data in a single document.


Steps to reproduce

db.foo.drop();
db.foo.createIndex({a: 'text'});
 
var termLength = 16;
 
var str = '';
for (var i = 0; i < (5 * 1024 * 1024 / termLength); i++) {
    if (i > 0) {
        str += ' ';
    }
    str += i.zeroPad(termLength);
}
 
assert.writeOK(db.foo.insert({a: str}));

assert: write failed with error: {
    "nInserted" : 0,
    "writeError" : {
        "code" : 16733,
        "errmsg" : "trying to index text where term list is too big, max is 4mb _id: ObjectId('568d68d5dd2e24af1002e247')"
    }
}
_getErrorWithCode@src/mongo/shell/utils.js:23:13
doassert@src/mongo/shell/assert.js:13:14
assert.writeOK@src/mongo/shell/assert.js:414:9



 Comments   
Comment by Githook User [ 13/Aug/18 ]

Author:

{'username': 'xy24', 'email': 'xiangyu.yao@mongodb.com', 'name': 'Xiangyu Yao'}

Message: SERVER-22078 Remove term list limits for text index in FCV 4.2
Branch: master
https://github.com/mongodb/mongo/commit/ab5cf1f214277536f1d4d2d1dc1fa319af911bbe

Comment by Daniel Pasette (Inactive) [ 06/Jan/16 ]

I think this seems like a good idea for 3.4. This journal limit only applies to MMAPv1.

Generated at Thu Feb 08 03:59:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.