[SERVER-47652] size of _id index 60% bigger Created: 16/Apr/20  Updated: 16/Jan/24  Resolved: 07/May/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.3.5
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Asya Kamsky Assignee: Vamsi Boyapati
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on WT-6082 Skip writing txn-id to data store whe... Closed
Duplicate
duplicates WT-6082 Skip writing txn-id to data store whe... Closed
Related
related to SERVER-47669 Add on-disk storage size testing Closed
Sprint: Storage - Ra 2020-05-18
Participants:
Story Points: 8

 Description   

I noticed in 4.3.5 that _id index size is about 60% bigger than in 4.3.4



 Comments   
Comment by Sulabh Mahajan [ 07/May/20 ]

I investigated in WT-6132 the reason for durable history to have caused the _id index size increase and concluded that WT-6082 will help us restore the size back once we are storing the validity window only when needed.

Here is my edited comment from WT-6132:

I investigated how much of impact WT-6082 has on MongoDB _id index.

I used the script to generate the dataset and ran test on 4.2.6, the latest MongoDB 4.4, and with the hack mentioned in WT-6082 to disable storing validity window. Here are the results from printCollectionStats after running the script:

MongoDB - 4.2.6
-----------------

        "nindexes" : 1,
        "indexBuilds" : [ ],
        "totalIndexSize" : 110592,
        "indexSizes" : {
                "_id_" : 110592
        },
        "scaleFactor" : 1,
        "ok" : 1

MongoDB - 4.3.5
-----------------

        "nindexes" : 1,
        "indexBuilds" : [ ],
        "totalIndexSize" : 163840,
        "totalSize" : 287096832,
        "indexSizes" : {
                "_id_" : 163840
        },
        "scaleFactor" : 1,
        "ok" : 1

MongoDB - 4.4 latest (7 May)
------------------------------

        "nindexes" : 1,
        "indexBuilds" : [ ],
        "totalIndexSize" : 163840,
        "totalSize" : 287096832,
        "indexSizes" : {
                "_id_" : 163840
        },
        "scaleFactor" : 1,
        "ok" : 1

MongoDB - 4.4 latest (7 May) + WT-6082 hack to disable storing validity window
---------------------------------------------------------------------------------

        "nindexes" : 1,
        "indexBuilds" : [ ],
        "totalIndexSize" : 110592,
        "totalSize" : 287043584,
        "indexSizes" : {
                "_id_" : 110592
        },
        "scaleFactor" : 1,
        "ok" : 1

Clearly the _id index reduces to the size from MongoDB-4.2.6 when validity window is not stored in the data files. WT-6082 is going to find a proper solution for that.

Now that I have confirmed that all of _id index size increment is due to WT-6082, I am going to close this ticket as a duplicate.

Comment by Jocelyn del Prado [ 30/Apr/20 ]

This may be related to WT-6082

Comment by Mark Callaghan (Inactive) [ 29/Apr/20 ]

I see a growth in _id index size in 4.4 before and after durable history was merged, but the problem is worse with durable history. Results are here

Comment by Asya Kamsky [ 20/Apr/20 ]

> One idea: I wonder if prefix compression broke somewhere along the way?

I had secondary index that would have significantly benefited from prefix compression that had the same size in these versions. You can see it in my previous comment stats as ""department_1" : 121303040 in both versions.

Comment by Maria van Keulen [ 20/Apr/20 ]

In case anybody is curious, here are excerpts of printCollectionStats for the offending commit and for its parent commit. storageSize also differs between the two commits.
Excerpt of printCollectionStats output for offending commit:

{
	"ns" : "test.asdf",
	"size" : 5314779000,
	"count" : 9000,
	"avgObjSize" : 590531,
	"storageSize" : 258236416,
...
	"nindexes" : 1,
	"indexBuilds" : [ ],
	"totalIndexSize" : 159744,
	"totalSize" : 258396160,
	"indexSizes" : {
		"_id_" : 159744
	},
	"scaleFactor" : 1,
	"ok" : 1
}

Excerpt of printCollectionStats output for parent commit of offending commit:

{
	"ns" : "test.asdf",
	"size" : 5314779000,
	"count" : 9000,
	"avgObjSize" : 590531,
	"storageSize" : 258207744,
...
	"nindexes" : 1,
	"indexBuilds" : [ ],
	"totalIndexSize" : 98304,
	"totalSize" : 258306048,
	"indexSizes" : {
		"_id_" : 98304
	},
	"scaleFactor" : 1,
	"ok" : 1
}

Comment by Eric Milkie [ 20/Apr/20 ]

One idea: I wonder if prefix compression broke somewhere along the way?

Comment by Eric Milkie [ 20/Apr/20 ]

I'm still puzzling over why only _id indexes seem to be affected. They are mostly the same as secondary index tables (which appear to not be affected).
One difference that comes to mind is that we insert records into _id index tables in key order, whereas in secondary index tables we typically do not. However, we also insert records into data tables in key order, and those don't appear to be affected either, although it would be good to confirm that.

Comment by Maria van Keulen [ 20/Apr/20 ]

Cool, thanks Eric.

I did a git bisect, and this is the commit after which the index size measurement shows the increase.

Passing to Storage Engines to investigate what part of this WT drop could have caused this.

Comment by Eric Milkie [ 20/Apr/20 ]

The sizes are different because a checkpoint is triggered by shutting down. The default checkpoint frequency is 60 seconds, so I suspect if you waited that long (and a little bit longer), you'd see the file sizes unchanging when you shut down. You can also just run the fsync command to trigger a checkpoint manually.

Comment by Maria van Keulen [ 20/Apr/20 ]

In my attempts to reproduce this, I noticed some interesting behavior of the index size measurement.
alexander.gorrod milkie Is there any reason why we would expect the measure of index sizes to differ across restarts?

I created a large collection on a 4.3.4 node, and made the following index size measurement:

> db.printCollectionStats()
asdf
{
	"ns" : "test.asdf",
	"size" : 5314779000,
	"count" : 9000,
	"avgObjSize" : 590531,
	"storageSize" : 200830976,
	"capped" : false,
	"wiredTiger" : {
		"metadata" : {
			"formatVersion" : 1
		},
...
	"nindexes" : 1,
	"indexBuilds" : [ ],
	"totalIndexSize" : 77824,
	"totalSize" : 200908800,
	"indexSizes" : {
		"_id_" : 77824
	},
	"scaleFactor" : 1,
	"ok" : 1
}

I restarted the 4.3.4 node, and observed a different measurement:

	
> db.printCollectionStats()
asdf
{
	"ns" : "test.asdf",
	"size" : 5314779000,
	"count" : 9000,
	"avgObjSize" : 590531,
	"storageSize" : 258224128,
	"freeStorageSize" : 16384,
	"capped" : false,
	"wiredTiger" : {
		"metadata" : {
			"formatVersion" : 1
		},
...
"nindexes" : 1,
	"indexBuilds" : [ ],
	"totalIndexSize" : 122880,
	"totalSize" : 258347008,
	"indexSizes" : {
		"_id_" : 122880
	},
	"scaleFactor" : 1,
	"ok" : 1
}

I mongorestore'd this data on a new 4.3.4 node with a new dbpath, and got a third measurement:

> db.printCollectionStats()
asdf
{
	"ns" : "test.asdf",
	"size" : 5314779000,
	"count" : 9000,
	"avgObjSize" : 590531,
	"storageSize" : 258207744,
	"capped" : false,
	"wiredTiger" : {
		"metadata" : {
			"formatVersion" : 1
		},
...
	"nindexes" : 1,
	"indexBuilds" : [ ],
	"totalIndexSize" : 94208,
	"totalSize" : 258301952,
	"indexSizes" : {
		"_id_" : 94208
	},
	"scaleFactor" : 1,
	"ok" : 1

This measurement remained the same across restarts of the new node.

In my investigation, I'll use the third measurement as my baseline.

Comment by Eric Milkie [ 20/Apr/20 ]

Interestingly, the 4.4 branch and the master branch diverged about halfway between the 4.3.4 and the 4.3.5 tags. That is to say, the 4.3.5 tag only exists in the 4.4 branch.
I quickly skimmed over all the commits between 4.3.4 and 4.3.5 and didn't see anything that would obviously cause such a behavior.

Comment by Alexander Gorrod [ 19/Apr/20 ]

I've moved the ticket from WT to SERVER and taken the storage engines team off as the owner, since it doesn't seem to be due to a durable history change. milkie would you mind helping to find the right owner?

Comment by Asya Kamsky [ 18/Apr/20 ]

mongo --port 43400 --eval 'db.printCollectionStats()' m1 | grep -e "_1" -e Size -e "_id_"
	"avgObjSize" : 46,
	"storageSize" : 73728,
			"Size of the root page" : 0,
	"totalIndexSize" : 53248,
	"totalSize" : 126976,
	"indexSizes" : {
		"_id_" : 53248
	"avgObjSize" : 51,
	"storageSize" : 90112,
			"Size of the root page" : 0,
	"totalIndexSize" : 53248,
	"totalSize" : 143360,
	"indexSizes" : {
		"_id_" : 53248
	"avgObjSize" : 101,
	"storageSize" : 602021888,
	"freeStorageSize" : 229376,
			"Size of the root page" : 0,
	"totalIndexSize" : 202838016,
	"totalSize" : 804859904,
	"indexSizes" : {
		"_id_" : 202838016
	"avgObjSize" : 101,
	"storageSize" : 602554368,
	"freeStorageSize" : 933888,
			"Size of the root page" : 0,
	"totalIndexSize" : 324169728,
	"totalSize" : 926724096,
	"indexSizes" : {
		"_id_" : 202866688,
		"department_1" : 121303040

mongo --port 43500 --eval 'db.printCollectionStats()' m1 | grep -e "_1" -e Size -e "_id_"
	"avgObjSize" : 46,
	"storageSize" : 77824,
			"Size of the root page" : 0,
	"totalIndexSize" : 81920,
	"totalSize" : 159744,
	"indexSizes" : {
		"_id_" : 81920
	"avgObjSize" : 51,
	"storageSize" : 90112,
			"Size of the root page" : 0,
	"totalIndexSize" : 77824,
	"totalSize" : 167936,
	"indexSizes" : {
		"_id_" : 77824
	"avgObjSize" : 101,
	"storageSize" : 637845504,
	"freeStorageSize" : 69632,
			"Size of the root page" : 0,
	"totalIndexSize" : 358305792,
	"totalSize" : 996151296,
	"indexSizes" : {
		"_id_" : 358305792
	"avgObjSize" : 101,
	"storageSize" : 632401920,
	"freeStorageSize" : 45056,
			"Size of the root page" : 0,
	"totalIndexSize" : 455430144,
	"totalSize" : 1087832064,
	"indexSizes" : {
		"_id_" : 334127104,
		"department_1" : 121303040

This is on an AWS machine I can provide access to.

Comment by Asya Kamsky [ 18/Apr/20 ]

Secondary index is identical size.

Data size is identical.

Comment by Eric Milkie [ 18/Apr/20 ]

Are any of the other files bigger? Can you check a secondary non-_id index?

Comment by Asya Kamsky [ 18/Apr/20 ]

To test, load a large collection with ObjectId (automatically generated) or with another field, small documents. I see this in all scenarios.

For collection of 4000 documents with ObjectId, _id index is 53% bigger. For 20M documents it's 76% bigger.

Comment by Asya Kamsky [ 18/Apr/20 ]

Sorry, I found an error in my check on 4.3.6 - I reloaded into 4.3.6 and _id index is the same size as in 4.4.0 (so it's not related to durable history) and it is 60% larger than in 4.2.x

I checked again and the size of _id index changed in version 4.3.5.

4.3.4 was the same size as 4.2.x but 4.3.5 was about 60% bigger.

Comment by Asya Kamsky [ 16/Apr/20 ]

My _id is just an ObjectId() so I'm not sure anything specific is needed to reproduce this...

Comment by Asya Kamsky [ 16/Apr/20 ]

It's synthetically generated data from some benchmarks. I can attach or upload a dump somewhere, it's only about a GBs.

Comment by Daniel Pasette (Inactive) [ 16/Apr/20 ]

likely a duplicate of one of the tickets that sulabh.mahajan is chasing (e.g., WT-6002), but it might be helpful to know what workload you're using to generate the comparison.

Comment by Alexander Gorrod [ 16/Apr/20 ]

asya can you give us some information about how to reproduce this? Can we get access to the data you are loading?

Comment by Asya Kamsky [ 16/Apr/20 ]

Same data was restored into each new DB (no upgrades involved).

Generated at Thu Feb 08 05:14:50 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.