[SERVER-13996] Space freed by deleted documents not reused in collection Created: 20/May/14  Updated: 10/Dec/14  Resolved: 10/Jun/14

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 2.4.9
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Teemu Ikonen Assignee: Thomas Rueckstiess
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Participants:

 Description   

We have messaging inbox collection where documents are inserted in regular rate. Documents are roughly the same size and are not modified after insertion in a way that would change their size. Collection has index by insertion time and we that to clean up documents with following rules.

1. Documents that are over 30 days are deleted
2. Amount of documents per user are capped to 100 most recent documents.

What we are observing:

A. On normal operation mongo does not seem to reuse the properly the space freed by deleted documents and keeps creating new extents and thus constantly increasing storage space in disk.

B. On manual collection compact we can see that older extents are nearly empty and only most recently created have lots of documents.

> db.battleinbox.stats(1024*1024*1024)
{
"ns" : "war.battleinbox",
"count" : 1532581,
"size" : 3,
"avgObjSize" : 0.000001957482182018438,
"storageSize" : 85,
"numExtents" : 63,
"nindexes" : 4,
"lastExtentSize" : 1,
"paddingFactor" : 1.0000000000083054,
"systemFlags" : 1,
"userFlags" : 0,
"totalIndexSize" : 0,
"indexSizes" : {
"id" : 0,
"read_1_uid_1" : 0,
"ts_1" : 0,
"shid_1_uid_1_ts_-1" : 0
},
"ok" : 1
}

Snipped from Mongo log during compaction.

Tue May 20 05:00:51.428 [conn43968] compact war.battleinbox begin
Tue May 20 05:00:51.428 [conn43968] paddingFactor:1 paddingBytes:0
Tue May 20 05:00:51.436 [conn43968] compact 63 extents
Tue May 20 05:00:51.436 [conn43968] compact orphan deleted lists
Tue May 20 05:00:51.438 [conn43968] compact dropping indexes
Tue May 20 05:00:51.464 [conn43968] compact begin extent #0 for namespace war.battleinbox
Tue May 20 05:00:51.464 [conn43968] compact paging in len=0.004096MB
Tue May 20 05:00:51.464 [conn43968] compact copying records
Tue May 20 05:00:51.464 [conn43968] compact finished extent #0 containing 0 documents (0MB) oldPadding: 1 1
Tue May 20 05:00:51.464 [conn43968] compact begin extent #1 for namespace war.battleinbox
Tue May 20 05:00:51.464 [conn43968] compact paging in len=0.032768MB
Tue May 20 05:00:51.464 [conn43968] compact copying records
Tue May 20 05:00:51.464 [conn43968] compact finished extent #1 containing 0 documents (0MB) oldPadding: 1 1
Tue May 20 05:00:51.465 [conn43968] compact begin extent #2 for namespace war.battleinbox
Tue May 20 05:00:51.465 [conn43968] compact paging in len=0.131072MB
Tue May 20 05:00:51.465 [conn43968] compact copying records
Tue May 20 05:00:51.465 [conn43968] compact finished extent #2 containing 0 documents (0MB) oldPadding: 1 1
Tue May 20 05:00:51.465 [conn43968] compact begin extent #3 for namespace war.battleinbox
Tue May 20 05:00:51.465 [conn43968] compact paging in len=0.524288MB
Tue May 20 05:00:51.465 [conn43968] compact copying records
Tue May 20 05:00:51.489 [conn43968] compact finished extent #3 containing 4 documents (0.007676MB) oldPadding: 1.01886 1
Tue May 20 05:00:51.489 [conn43968] compact begin extent #4 for namespace war.battleinbox
Tue May 20 05:00:51.489 [conn43968] compact paging in len=2.09715MB
Tue May 20 05:00:51.490 [conn43968] compact copying records
Tue May 20 05:00:51.499 [conn43968] compact finished extent #4 containing 5 documents (0.010288MB) oldPadding: 1.02043 1
Tue May 20 05:00:51.499 [conn43968] compact begin extent #5 for namespace war.battleinbox
Tue May 20 05:00:51.499 [conn43968] compact paging in len=8.38861MB
Tue May 20 05:00:51.510 [conn43968] compact copying records
Tue May 20 05:00:51.540 [conn43968] compact finished extent #5 containing 16 documents (0.033408MB) oldPadding: 1.04345 1
Tue May 20 05:00:51.540 [conn43968] compact begin extent #6 for namespace war.battleinbox
Tue May 20 05:00:51.540 [conn43968] compact paging in len=11.3254MB
Tue May 20 05:00:51.570 [conn43968] compact copying records
Tue May 20 05:00:51.571 [conn43968] compact finished extent #6 containing 9 documents (0.018416MB) oldPadding: 1.03004 1
Tue May 20 05:00:51.571 [conn43968] compact begin extent #7 for namespace war.battleinbox
Tue May 20 05:00:51.571 [conn43968] compact paging in len=15.2904MB
Tue May 20 05:00:51.758 [conn43968] compact copying records
Tue May 20 05:00:51.758 [conn43968] compact finished extent #7 containing 5 documents (0.010032MB) oldPadding: 1.01827 1
Tue May 20 05:00:51.758 [conn43968] compact begin extent #8 for namespace war.battleinbox
Tue May 20 05:00:51.758 [conn43968] compact paging in len=21.0248MB
Tue May 20 05:00:52.250 [conn43968] compact copying records
Tue May 20 05:00:52.364 [conn43968] compact finished extent #8 containing 23 documents (0.04648MB) oldPadding: 1.03964 1
Tue May 20 05:00:52.364 [conn43968] compact begin extent #9 for namespace war.battleinbox
Tue May 20 05:00:52.364 [conn43968] compact paging in len=28.3853MB
Tue May 20 05:00:52.923 [conn43968] compact copying records
Tue May 20 05:00:53.056 [conn43968] compact finished extent #9 containing 37 documents (0.074752MB) oldPadding: 1.03086 1
Tue May 20 05:00:53.056 [conn43968] compact begin extent #10 for namespace war.battleinbox
Tue May 20 05:00:53.056 [conn43968] compact paging in len=38.3222MB
Tue May 20 05:00:53.815 [conn43968] compact copying records
Tue May 20 05:00:53.907 [conn43968] compact finished extent #10 containing 53 documents (0.109488MB) oldPadding: 1.03551 1
Tue May 20 05:00:53.907 [conn43968] compact begin extent #11 for namespace war.battleinbox
Tue May 20 05:00:53.907 [conn43968] compact paging in len=51.7366MB
Tue May 20 05:00:55.061 [conn43968] compact end paging in 1154ms 0.0448324MB/sec
Tue May 20 05:00:55.061 [conn43968] compact copying records
Tue May 20 05:00:55.217 [conn43968] compact finished extent #11 containing 78 documents (0.158624MB) oldPadding: 1.03201 1
Tue May 20 05:00:55.217 [conn43968] compact begin extent #12 for namespace war.battleinbox
Tue May 20 05:00:55.217 [conn43968] compact paging in len=69.845MB
Tue May 20 05:00:56.769 [conn43968] compact end paging in 1552ms 0.0450032MB/sec
Tue May 20 05:00:56.769 [conn43968] compact copying records
Tue May 20 05:00:56.888 [conn43968] compact finished extent #12 containing 83 documents (0.171216MB) oldPadding: 1.03235 1
Tue May 20 05:00:56.888 [conn43968] compact begin extent #13 for namespace war.battleinbox
Tue May 20 05:00:56.888 [conn43968] compact paging in len=94.294MB
Tue May 20 05:00:58.955 [conn43968] compact end paging in 2067ms 0.0456188MB/sec
Tue May 20 05:00:58.955 [conn43968] compact copying records
Tue May 20 05:00:59.095 [conn43968] compact finished extent #13 containing 144 documents (0.293888MB) oldPadding: 1.03259 1
Tue May 20 05:00:59.095 [conn43968] compact begin extent #14 for namespace war.battleinbox
Tue May 20 05:00:59.095 [conn43968] compact paging in len=127.3MB
Tue May 20 05:01:01.997 [conn43968] compact end paging in 2901ms 0.0438813MB/sec
Tue May 20 05:01:01.997 [conn43968] compact copying records
Tue May 20 05:01:02.144 [conn43968] compact finished extent #14 containing 212 documents (0.4336MB) oldPadding: 1.03178 1
Tue May 20 05:01:02.144 [conn43968] compact begin extent #15 for namespace war.battleinbox
Tue May 20 05:01:02.144 [conn43968] compact paging in len=171.856MB
Tue May 20 05:01:06.019 [conn43968] compact end paging in 3875ms 0.0443499MB/sec
Tue May 20 05:01:06.020 [conn43968] compact copying records
Tue May 20 05:01:06.179 [conn43968] compact finished extent #15 containing 260 documents (0.539968MB) oldPadding: 1.034 1
Tue May 20 05:01:06.179 [conn43968] compact begin extent #16 for namespace war.battleinbox
Tue May 20 05:01:06.179 [conn43968] compact paging in len=235.098MB
Tue May 20 05:01:11.984 [conn43968] compact end paging in 5805ms 0.0404992MB/sec
Tue May 20 05:01:11.984 [conn43968] compact copying records
Tue May 20 05:01:12.204 [conn43968] compact finished extent #16 containing 375 documents (0.777568MB) oldPadding: 1.03569 1
Tue May 20 05:01:12.204 [conn43968] compact begin extent #17 for namespace war.battleinbox
Tue May 20 05:01:12.204 [conn43968] compact paging in len=345.985MB
Tue May 20 05:01:22.110 [conn43968] compact end paging in 9906ms 0.0349268MB/sec
Tue May 20 05:01:22.111 [conn43968] compact copying records
Tue May 20 05:01:22.269 [conn43968] compact finished extent #17 containing 454 documents (0.937632MB) oldPadding: 1.03502 1
Tue May 20 05:01:22.269 [conn43968] compact begin extent #18 for namespace war.battleinbox
Tue May 20 05:01:22.269 [conn43968] compact paging in len=467.083MB
Tue May 20 05:01:33.031 [conn43968] compact end paging in 10761ms 0.0434052MB/sec
Tue May 20 05:01:33.031 [conn43968] compact copying records
Tue May 20 05:01:33.158 [conn43968] compact finished extent #18 containing 373 documents (0.754992MB) oldPadding: 1.0286 1
Tue May 20 05:01:33.158 [conn43968] compact begin extent #19 for namespace war.battleinbox
Tue May 20 05:01:33.158 [conn43968] compact paging in len=630.563MB
Tue May 20 05:01:47.821 [conn43968] compact end paging in 14662ms 0.0430066MB/sec
Tue May 20 05:01:47.821 [conn43968] compact copying records
Tue May 20 05:01:47.884 [conn43968] compact finished extent #19 containing 489 documents (1.00082MB) oldPadding: 1.03251 1
Tue May 20 05:01:47.884 [conn43968] compact begin extent #20 for namespace war.battleinbox
Tue May 20 05:01:47.884 [conn43968] compact paging in len=851.263MB
Tue May 20 05:02:07.721 [conn43968] compact end paging in 19837ms 0.0429129MB/sec
Tue May 20 05:02:07.721 [conn43968] compact copying records
Tue May 20 05:02:07.939 [conn43968] compact finished extent #20 containing 696 documents (1.42874MB) oldPadding: 1.03263 1
Tue May 20 05:02:07.939 [conn43968] compact begin extent #21 for namespace war.battleinbox
Tue May 20 05:02:07.939 [conn43968] compact paging in len=1149.21MB
Tue May 20 05:02:34.656 [conn43968] compact end paging in 26716ms 0.0430157MB/sec
Tue May 20 05:02:34.656 [conn43968] compact copying records
Tue May 20 05:02:34.840 [conn43968] compact finished extent #21 containing 791 documents (1.6269MB) oldPadding: 1.03209 1
Tue May 20 05:02:34.840 [conn43968] compact begin extent #22 for namespace war.battleinbox
Tue May 20 05:02:34.840 [conn43968] compact paging in len=1551.43MB
Tue May 20 05:03:10.939 [conn43968] compact end paging in 36098ms 0.0429783MB/sec
Tue May 20 05:03:10.939 [conn43968] compact copying records
Tue May 20 05:03:11.178 [conn43968] compact finished extent #22 containing 1384 documents (2.86797MB) oldPadding: 1.03559 1
Tue May 20 05:03:11.178 [conn43968] compact begin extent #23 for namespace war.battleinbox
Tue May 20 05:03:11.178 [conn43968] compact paging in len=2146.43MB
Tue May 20 05:03:49.700 [conn43968] compact end paging in 38521ms 0.055721MB/sec
Tue May 20 05:03:49.700 [conn43968] compact copying records
Tue May 20 05:03:50.877 [conn43968] compact finished extent #23 containing 1749 documents (3.64651MB) oldPadding: 1.03661 1
Tue May 20 05:03:50.877 [conn43968] compact begin extent #24 for namespace war.battleinbox
Tue May 20 05:03:50.877 [conn43968] compact paging in len=2146.43MB
Tue May 20 05:04:29.322 [conn43968] compact end paging in 38444ms 0.0558326MB/sec
Tue May 20 05:04:29.322 [conn43968] compact copying records
Tue May 20 05:04:30.037 [conn43968] compact finished extent #24 containing 940 documents (1.99226MB) oldPadding: 1.03773 1
Tue May 20 05:04:30.037 [conn43968] compact begin extent #25 for namespace war.battleinbox
Tue May 20 05:04:30.037 [conn43968] compact paging in len=2146.43MB
Tue May 20 05:05:08.443 [conn43968] compact end paging in 38405ms 0.0558893MB/sec
Tue May 20 05:05:08.443 [conn43968] compact copying records
Tue May 20 05:05:08.780 [conn43968] compact finished extent #25 containing 437 documents (0.929712MB) oldPadding: 1.0404 1
Tue May 20 05:05:08.780 [conn43968] compact begin extent #26 for namespace war.battleinbox
Tue May 20 05:05:08.780 [conn43968] compact paging in len=2146.43MB
Tue May 20 05:05:47.164 [conn43968] compact end paging in 38383ms 0.0559213MB/sec
Tue May 20 05:05:47.164 [conn43968] compact copying records
Tue May 20 05:05:47.562 [conn43968] compact finished extent #26 containing 273 documents (0.59096MB) oldPadding: 1.04322 1
Tue May 20 05:05:47.562 [conn43968] compact begin extent #27 for namespace war.battleinbox
Tue May 20 05:05:47.562 [conn43968] compact paging in len=2146.43MB
Tue May 20 05:06:25.931 [conn43968] compact end paging in 38368ms 0.0559432MB/sec
Tue May 20 05:06:25.931 [conn43968] compact copying records
Tue May 20 05:06:26.524 [conn43968] compact finished extent #27 containing 467 documents (1.00718MB) oldPadding: 1.04159 1
Tue May 20 05:06:26.524 [conn43968] compact begin extent #28 for namespace war.battleinbox
Tue May 20 05:06:26.524 [conn43968] compact paging in len=2146.43MB
Tue May 20 05:07:04.861 [conn43968] compact end paging in 38336ms 0.0559898MB/sec
Tue May 20 05:07:04.861 [conn43968] compact copying records
Tue May 20 05:07:05.507 [conn43968] compact finished extent #28 containing 609 documents (1.32581MB) oldPadding: 1.04206 1
Tue May 20 05:07:05.507 [conn43968] compact begin extent #29 for namespace war.battleinbox
Tue May 20 05:07:05.507 [conn43968] compact paging in len=2146.43MB
Tue May 20 05:07:43.792 [conn43968] compact end paging in 38285ms 0.0560644MB/sec
Tue May 20 05:07:43.792 [conn43968] compact copying records
Tue May 20 05:07:44.608 [conn43968] compact finished extent #29 containing 818 documents (1.7967MB) oldPadding: 1.04566 1
Tue May 20 05:07:44.608 [conn43968] compact begin extent #30 for namespace war.battleinbox
Tue May 20 05:07:44.608 [conn43968] compact paging in len=2146.43MB
Tue May 20 05:08:22.851 [conn43968] compact end paging in 38242ms 0.0561275MB/sec
Tue May 20 05:08:22.851 [conn43968] compact copying records
Tue May 20 05:08:23.527 [conn43968] compact finished extent #30 containing 924 documents (2.03334MB) oldPadding: 1.04598 1
Tue May 20 05:08:23.527 [conn43968] compact begin extent #31 for namespace war.battleinbox
Tue May 20 05:08:23.527 [conn43968] compact paging in len=2146.43MB
Tue May 20 05:09:01.712 [conn43968] compact end paging in 38185ms 0.0562113MB/sec
Tue May 20 05:09:01.712 [conn43968] compact copying records
Tue May 20 05:09:02.664 [conn43968] compact finished extent #31 containing 937 documents (2.06181MB) oldPadding: 1.04446 1
Tue May 20 05:09:02.664 [conn43968] compact begin extent #32 for namespace war.battleinbox
Tue May 20 05:09:02.664 [conn43968] compact paging in len=2146.43MB
Tue May 20 05:09:40.797 [conn43968] compact end paging in 38132ms 0.0562894MB/sec
Tue May 20 05:09:40.797 [conn43968] compact copying records
Tue May 20 05:09:42.087 [conn43968] compact finished extent #32 containing 1256 documents (2.76416MB) oldPadding: 1.04493 1
Tue May 20 05:09:42.087 [conn43968] compact begin extent #33 for namespace war.battleinbox
Tue May 20 05:09:42.087 [conn43968] compact paging in len=2146.43MB
Tue May 20 05:10:20.169 [conn43968] compact end paging in 38081ms 0.0563648MB/sec
Tue May 20 05:10:20.169 [conn43968] compact copying records
Tue May 20 05:10:21.857 [conn43968] compact finished extent #33 containing 1777 documents (3.90667MB) oldPadding: 1.04512 1
Tue May 20 05:10:21.857 [conn43968] compact begin extent #34 for namespace war.battleinbox
Tue May 20 05:10:21.857 [conn43968] compact paging in len=2146.43MB
Tue May 20 05:10:59.852 [conn43968] compact end paging in 37995ms 0.0564924MB/sec
Tue May 20 05:10:59.852 [conn43968] compact copying records
Tue May 20 05:11:01.549 [conn43968] compact finished extent #34 containing 2016 documents (4.46843MB) oldPadding: 1.04702 1
Tue May 20 05:11:01.549 [conn43968] compact begin extent #35 for namespace war.battleinbox
Tue May 20 05:11:01.549 [conn43968] compact paging in len=2146.43MB
Tue May 20 05:11:39.422 [conn43968] compact end paging in 37873ms 0.0566743MB/sec
Tue May 20 05:11:39.422 [conn43968] compact copying records
Tue May 20 05:11:41.351 [conn43968] compact finished extent #35 containing 2370 documents (5.26141MB) oldPadding: 1.04594 1
Tue May 20 05:11:41.351 [conn43968] compact begin extent #36 for namespace war.battleinbox
Tue May 20 05:11:41.351 [conn43968] compact paging in len=2094.43MB
Tue May 20 05:12:19.144 [conn43968] compact end paging in 37792ms 0.05542MB/sec
Tue May 20 05:12:19.144 [conn43968] compact copying records
Tue May 20 05:12:21.563 [conn43968] compact finished extent #36 containing 3085 documents (6.85304MB) oldPadding: 1.04672 1
Tue May 20 05:12:21.563 [conn43968] compact begin extent #37 for namespace war.battleinbox
Tue May 20 05:12:21.563 [conn43968] compact paging in len=1921.29MB
Tue May 20 05:12:59.234 [conn43968] compact end paging in 37671ms 0.0510018MB/sec
Tue May 20 05:12:59.234 [conn43968] compact copying records
Tue May 20 05:13:01.777 [conn43968] compact finished extent #37 containing 4382 documents (9.68259MB) oldPadding: 1.04524 1
Tue May 20 05:13:01.777 [conn43968] compact begin extent #38 for namespace war.battleinbox
Tue May 20 05:13:01.778 [conn43968] compact paging in len=2146.43MB
Tue May 20 05:13:39.289 [conn43968] compact end paging in 37511ms 0.0572213MB/sec
Tue May 20 05:13:39.289 [conn43968] compact copying records
Tue May 20 05:13:42.864 [conn43968] compact finished extent #38 containing 7626 documents (16.8782MB) oldPadding: 1.04552 1
Tue May 20 05:13:42.864 [conn43968] compact begin extent #39 for namespace war.battleinbox
Tue May 20 05:13:42.864 [conn43968] compact paging in len=2146.43MB
Tue May 20 05:14:20.132 [conn43968] compact end paging in 37268ms 0.0575944MB/sec
Tue May 20 05:14:20.132 [conn43968] compact copying records
Tue May 20 05:14:26.157 [conn43968] compact finished extent #39 containing 5139 documents (11.604MB) oldPadding: 1.04755 1
Tue May 20 05:14:26.157 [conn43968] compact begin extent #40 for namespace war.battleinbox
Tue May 20 05:14:26.157 [conn43968] compact paging in len=2146.43MB
Tue May 20 05:14:41.515 [DataFileSync] flushing mmaps took 11811ms for 139 files
Tue May 20 05:15:06.768 [conn43968] compact end paging in 40611ms 0.0528533MB/sec
Tue May 20 05:15:06.768 [conn43968] compact copying records
Tue May 20 05:15:09.439 [conn43968] compact finished extent #40 containing 4555 documents (10.3034MB) oldPadding: 1.04767 1
Tue May 20 05:15:09.439 [conn43968] compact begin extent #41 for namespace war.battleinbox
Tue May 20 05:15:09.439 [conn43968] compact paging in len=2146.43MB
Tue May 20 05:15:47.953 [conn43968] compact end paging in 38513ms 0.0557325MB/sec
Tue May 20 05:15:47.953 [conn43968] compact copying records
Tue May 20 05:15:52.689 [conn43968] compact finished extent #41 containing 6049 documents (13.6823MB) oldPadding: 1.04752 1
Tue May 20 05:15:52.690 [conn43968] compact begin extent #42 for namespace war.battleinbox
Tue May 20 05:15:52.690 [conn43968] compact paging in len=2146.43MB
Tue May 20 05:16:30.982 [conn43968] compact end paging in 38292ms 0.0560542MB/sec
Tue May 20 05:16:30.982 [conn43968] compact copying records
Tue May 20 05:16:37.888 [conn43968] compact finished extent #42 containing 12125 documents (27.1408MB) oldPadding: 1.04679 1
Tue May 20 05:16:37.888 [conn43968] compact begin extent #43 for namespace war.battleinbox
Tue May 20 05:16:37.889 [conn43968] compact paging in len=2146.43MB
...
Tue May 20 05:25:23.830 [conn43968] compact begin extent #54 for namespace war.battleinbox
Tue May 20 05:25:23.830 [conn43968] compact paging in len=2146.43MB
Tue May 20 05:26:04.208 [conn43968] compact end paging in 40378ms 0.0531583MB/sec
Tue May 20 05:26:04.208 [conn43968] compact copying records
Tue May 20 05:26:15.817 [conn43968] compact finished extent #54 containing 33767 documents (88.8476MB) oldPadding: 1.05309 1
Tue May 20 05:26:15.818 [conn43968] compact begin extent #55 for namespace war.battleinbox
Tue May 20 05:26:15.818 [conn43968] compact paging in len=2146.43MB
Tue May 20 05:26:55.963 [conn43968] compact end paging in 40145ms 0.0534669MB/sec
Tue May 20 05:26:55.963 [conn43968] compact copying records
Tue May 20 05:27:08.487 [conn43968] compact finished extent #55 containing 42391 documents (113.912MB) oldPadding: 1.05201 1
...

After compaction:

> db.battleinbox.stats(1024*1024*1024)
{
"ns" : "war.battleinbox",
"count" : 1531300,
"size" : 3,
"avgObjSize" : 0.000001959119702213805,
"storageSize" : 4,
"numExtents" : 22,
"nindexes" : 4,
"lastExtentSize" : 1,
"paddingFactor" : 1.0000000000083054,
"systemFlags" : 1,
"userFlags" : 0,
"totalIndexSize" : 0,
"indexSizes" :

{ "_id_" : 0, "read_1_uid_1" : 0, "ts_1" : 0, "shid_1_uid_1_ts_-1" : 0 }

,
"ok" : 1
}

Is this expected behavior and how to avoid this problem?



 Comments   
Comment by Ramon Fernandez Marina [ 03/Sep/14 ]

Thanks for the updated status tikonen, glad to hear that usePowerOf2Sizes worked out for you.

Regards,
Ramón.

Comment by Teemu Ikonen [ 03/Sep/14 ]

Hi, just updating this with new long term production experiences. System has been running nearly same load, same mongo version, settings (powerof2) and cleanup logic.

Seems that the storage size on disk has stabilized well.

Now stats are:

> db.battleinbox.stats(1024*1024)
{
"ns" : "war.battleinbox",
"count" : 704745,
"size" : 2717,
"avgObjSize" : 0.003855295177688384,
"storageSize" : 26069,
"numExtents" : 33,
"nindexes" : 4,
"lastExtentSize" : 2046,
"paddingFactor" : 1.0000000000164775,
"systemFlags" : 1,
"userFlags" : 1,
"totalIndexSize" : 162,
"indexSizes" :

{ "_id_" : 54, "read_1_uid_1" : 31, "ts_1" : 28, "shid_1_uid_1_ts_-1" : 47 }

,
"ok" : 1
}
> db.inbox.stats(1024*1024)
{
"ns" : "war.inbox",
"count" : 311708,
"size" : 286,
"avgObjSize" : 0.0009175253763137295,
"storageSize" : 1630,
"numExtents" : 20,
"nindexes" : 5,
"lastExtentSize" : 430,
"paddingFactor" : 1.0000000000014622,
"systemFlags" : 1,
"userFlags" : 1,
"totalIndexSize" : 76,
"indexSizes" :

{ "_id_" : 26, "read_1_uid_1" : 12, "ts_1" : 15, "adminid_1" : 2, "shid_1_uid_1_ts_-1" : 19 }

,
"ok" : 1
}

/Teemu

Comment by Teemu Ikonen [ 11/Jun/14 ]

Yes, we enabled the usePowerOf2Sizes option in the production and are now waiting for a week to rotate few tens of millions entries in the problem collections.
I'll report the results when we have next scheduled compact to see how much situation has improved. At least now I don't see MongoDB anymore growing the disk space.

Here are current stats for the problem collections:

> db.battleinbox.stats(1024*1024)
{
"ns" : "war.battleinbox",
"count" : 1640431,
"size" : 5708,
"avgObjSize" : 0.0034795733560265565,
"storageSize" : 21975,
"numExtents" : 31,
"nindexes" : 4,
"lastExtentSize" : 2046,
"paddingFactor" : 1.0000000000151579,
"systemFlags" : 1,
"userFlags" : 1,
"totalIndexSize" : 372,
"indexSizes" :

{ "_id_" : 125, "read_1_uid_1" : 71, "ts_1" : 69, "shid_1_uid_1_ts_-1" : 105 }

,
"ok" : 1
}

> db.inbox.stats(1024*1024)
{
"ns" : "war.inbox",
"count" : 1250322,
"size" : 957,
"avgObjSize" : 0.0007654028322304175,
"storageSize" : 1200,
"numExtents" : 19,
"nindexes" : 5,
"lastExtentSize" : 319,
"paddingFactor" : 1.0000000000014622,
"systemFlags" : 1,
"userFlags" : 1,
"totalIndexSize" : 264,
"indexSizes" :

{ "_id_" : 76, "read_1_uid_1" : 52, "ts_1" : 46, "adminid_1" : 13, "shid_1_uid_1_ts_-1" : 75 }

,
"ok" : 1
}

/Teemu

Comment by Ramon Fernandez Marina [ 10/Jun/14 ]

Hi tikonen,

have you had a chance to try the usePowerOf2Sizes option and see if it has any impact? At this stage I don't believe there's a bug in MongoDB with respect to this ticket, so I'm going to mark it as resolved. Feel free to add additional information on this ticket if the usePowerOf2Sizes option does help you.

Regards,
Ramón.

Comment by Thomas Rueckstiess [ 22/May/14 ]

Hi Teemu, thanks, please let us know if the option has any impact.

Cheers,
Thomas

Comment by Teemu Ikonen [ 21/May/14 ]

Thank you for reply, I will try with the powerof2sizes option next and report here if this does not improve situation.

This was filed as bug as for me it's not expected behavior to see database not to re-use large empty spaces of the data storage, especially when all documents are nearly of same size. Collection compaction does keep the problem in bay, but it's possible to run only on scheduled maintenance and takes very long time to run.

Thanks, Teemu

Comment by Thomas Rueckstiess [ 21/May/14 ]

Hi Teemu,

Sorry, I just realized the "usePowerOf2Sizes" option became the default only in version 2.6, not in 2.4 as I stated above. Regardless, I'll encourage you to try it out and see if the documents can be re-allocated more efficiently.

Regards,
Thomas

Comment by Thomas Rueckstiess [ 20/May/14 ]

Hi Teemu,

It sounds like your use case would benefit from using the usePowerOf2Sizes option. This option ensures that documents are always allocated in increments of powers of 2 (32, 64, 128) bytes, and therefore it is much more likely that a deleted document on the freelist can be re-used for a new allocation.

This option is enabled by default in version 2.4.x, but from your collection stats, I can see that the userFlags value is 0; a sign that the option is not set. It seems that you created this collection before 2.4 was released and the setting is not automatically enabled for existing collections when upgrading.

In general, this SERVER project is for tracking bugs and feature requests. In the future, please ask such questions in our mongodb-user Google Group or on Stack Overflow with the mongodb tag.

Regards,
Thomas

Generated at Thu Feb 08 03:33:32 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.