[SERVER-16632] Change WiredTiger index key format to no longer use BSON Created: 22/Dec/14  Updated: 27/Oct/15  Resolved: 13/Jan/15

Status: Closed
Project: Core Server
Component/s: Index Maintenance, Storage
Affects Version/s: None
Fix Version/s: 2.8.0-rc5

Type: Improvement Priority: Critical - P2
Reporter: Daniel Pasette (Inactive) Assignee: Mathias Stearn
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-16604 Improve integration with rocks Closed
Documented
is documented by DOCS-4664 Document user-facing changes related ... Closed
Duplicate
is duplicated by SERVER-16762 Loss of type information for covered ... Closed
is duplicated by SERVER-16975 UnsupportedFormat fatal assertion in ... Closed
Related
related to SERVER-16916 RecordId might be inserted more than ... Closed
Tested
Backwards Compatibility: Major Change
Participants:

 Description   

Change index key format to no longer use BSON in WiredTiger.
Benefits include:

  • much faster key comparison
  • reduced storage size for indexes
  • take much better advantage of WT key prefix compression
  • no longer require custom collator, which means stock WT tools like repair work with no modification

As this is an on-disk format change, indexes created before this change with WT will no longer be usable without running:
mongod --storageEngine wiredTiger --repair



 Comments   
Comment by Ramon Fernandez Marina [ 17/Jan/15 ]

redbeard0531, is the above code review for this ticket? It would be very confusing to have a commit in a ticket after it has been resolved

Comment by Mathias Stearn [ 13/Jan/15 ]

@Docs: Two things that need mention: upgrading WT data from older RCs will require --repair. Also, when doing a covered index query, if the result is a double with the value NaN, we no longer guarantee which NaN you get back.

Comment by Githook User [ 12/Jan/15 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-16632 Encode TypeBits along with KeyStrings in WT indexes

This allows covered queries to recover the original data, without collapsing
cases that compare equally (such as 1 and 1.0, or 0.0 and -0.0). However, no
attempt is made to preserve specific versions of NaN, they all become
quiet_NaN().

This resolves this issue discussed in SERVER-16762.
Branch: master
https://github.com/mongodb/mongo/commit/a9cacdf8762ed197b1de1fd5d4ff2b47692e5a92

Comment by Githook User [ 12/Jan/15 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-16632 Improve WT index cursors

Now only fetches data from WT at most once. Also avoids converting back and
forth between KeyString and BSON.
Branch: master
https://github.com/mongodb/mongo/commit/743051d82afc04d9e236a6519c50f7b1526f991d

Comment by Githook User [ 12/Jan/15 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-16632 Split WT index cursor into classes for unique and non-unique

This commit is largely code motion and reorganization. The actual operations
performed haven't changed, other than no longer checking if the index is
unique now that different functions are executed. The next commit will take
advantage of this to improve how these cursors work.
Branch: master
https://github.com/mongodb/mongo/commit/2a84c1955e5f127c0befe9b8851ae7eeaba3f220

Comment by Githook User [ 12/Jan/15 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-16632 Test more interesting numeric cases for KeyString
Branch: master
https://github.com/mongodb/mongo/commit/f436133aa0460fd5fd6cd7f6b91b61208645f867

Comment by Githook User [ 12/Jan/15 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-16632 Support strings with NUL bytes in KeyString

Resolves issue SERVER-16798
Branch: master
https://github.com/mongodb/mongo/commit/ebe9daa834a1b1e59194a2c91d8a6962c62c7770

Comment by Githook User [ 07/Jan/15 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-16632 Remove unneeded kEquals byte after each field in KeyString
Branch: master
https://github.com/mongodb/mongo/commit/331911d3f31cf93c82bfaab3b6db0a77fdb80374

Comment by Githook User [ 07/Jan/15 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-16632 Compress index keys in KeyString

  • Bools are now completely encoded in the type byte
  • BinData size is stored in a single byte if < 255
  • Numeric encoding improvements:
    • Maximum encoded size including type byte is now 9 bytes
    • The "size-class" of each number is now stored in the type byte
    • Integers and the integer part of Doubles use variable numbers of bytes
    • Doubles > 1 with a fractional part implicitly store their exponent in the
      integer part and use a variable number of bytes for the fractional part
      Branch: master
      https://github.com/mongodb/mongo/commit/5d46a3a9f7b7f8b17ae1cbe73dd9f8d056350b81
Comment by Githook User [ 07/Jan/15 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-16632 Compress small RecordIds in KeyString
Branch: master
https://github.com/mongodb/mongo/commit/4075f40df4fcb86ce6c8c6ea9565a4699769e4ed

Comment by Githook User [ 07/Jan/15 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: Make BSONObj::woCompare a total ordering

Changes:

  • Date and Timestamp are now distinct with all Dates before all Timestamps.
  • Numeric comparisons now correctly handle Doubles and Long > 2**53.
  • CodeWScope doesn't strcmp BSONObjs or strings which may contain NULs.

There should be no changes in any of the cases where woCompare defined a
correct total ordering before. The new behavior matches the ordering defined
by KeyString.

Related tickets:

Changes to the comparison function for aggregation will be handled separately
as SERVER-16708.
Branch: master
https://github.com/mongodb/mongo/commit/f58fa5f78a91ce36d8e31d6730ebfdaa6cc1d5ab

Comment by Githook User [ 31/Dec/14 ]

Author:

{u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-16632: fix os x compile with signed/unsigned
Branch: master
https://github.com/mongodb/mongo/commit/d3e04458eb1d59e5e83be86cfaec288e7dda7139

Comment by Githook User [ 31/Dec/14 ]

Author:

{u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-16632: use KeyString for WiredTiger index keys
Branch: master
https://github.com/mongodb/mongo/commit/23dc8ce626613cf9d92cb58dcf877407fd668d5b

Comment by Githook User [ 31/Dec/14 ]

Author:

{u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-16632: introduct KeyString, a memcmpable version of a index key
Branch: master
https://github.com/mongodb/mongo/commit/24706753ff3682714abe6e3468b68379eed9a526

Generated at Thu Feb 08 03:41:44 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.