[SERVER-5285] Either disallow or fix capped collections with more than 2^32 docs Created: 11/Mar/12  Updated: 11/Jul/16  Resolved: 26/Nov/12

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 2.1.0
Fix Version/s: 2.3.2

Type: Bug Priority: Major - P3
Reporter: Richard Kreuter (Inactive) Assignee: Eliot Horowitz (Inactive)
Resolution: Done Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
is duplicated by SERVER-5503 All capped collections are implicitly... Closed
Related
related to SERVER-7780 downgrading 2.3 to 2.0 causes local.o... Closed
Operating System: ALL
Participants:

 Description   

The max documents counter for a capped collection is an int, and it appears as if creating a very large capped collection and then inserting more than 2^31-1 documents into it will give a collection for which the document counters in collStats stop working.



 Comments   
Comment by auto [ 26/Nov/12 ]

Author:

{u'date': u'2012-11-26T02:09:46Z', u'email': u'eliot@10gen.com', u'name': u'Eliot Horowitz'}

Message: SERVER-5285 - fix assert code conflict
Branch: master
https://github.com/mongodb/mongo/commit/8ae059694e049913eaf5f291a6ba1218ea12ca63

Comment by auto [ 26/Nov/12 ]

Author:

{u'date': u'2012-11-20T19:26:09Z', u'email': u'eliot@10gen.com', u'name': u'Eliot Horowitz'}

Message: SERVER-5285 fix log messages and test
Branch: master
https://github.com/mongodb/mongo/commit/55ca3569e8f9c7ace67f20f67cf5ece631e47df8

Comment by auto [ 26/Nov/12 ]

Author:

{u'date': u'2012-11-20T15:29:52Z', u'email': u'eliot@10gen.com', u'name': u'Eliot Horowitz'}

Message: SERVER-5285 - capped collection max should be 2^63
Branch: master
https://github.com/mongodb/mongo/commit/a710a4a341f79af877e12a714317a4c3c5caa017

Comment by Thorn Roby [ 28/Sep/12 ]

SECONDARY> db.log.stats()
{
"ns" : "lfs.log",
"count" : 34839278692,
"size" : NumberLong("18058354354488"),
"avgObjSize" : 518.3331869219975,
"storageSize" : NumberLong("20995000011952"),
"numExtents" : 9783,
"nindexes" : 2,
"lastExtentSize" : 4096,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : NumberLong("4059987315216"),
"indexSizes" :

{ "_id_" : NumberLong("1384552832096"), "t_1_h_1_a_1" : NumberLong("2675434483120") }

,
"capped" : 1,
"max" : 2147483647,
"ok" : 1
}

Comment by Thorn Roby [ 24/Sep/12 ]

We've been successfully running version 2.0.4a for several months with the interim fix for this issue, which as far as I know consists of ignoring the 32bit value (2Bn) when deciding to wrap a collection that exceeds this count.
However, we are something like 3 weeks away from the point where we will actually reach the end of the 20TB of storage allocated for the collection (currently about 32Bn documents). I am concerned that we may hit another bug at that point and potentially lose 8 months of data. I also understand that the long-term fix for this won't be implemented soon. Could someone do some research on what is likely to happen at the point we actually fill the collection, and determine whether it needs any additional interim fixes?

Comment by auto [ 09/Apr/12 ]

Author:

{u'login': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: use long long instead of int as part of capped alloc - part of SERVER-5285
Branch: master
https://github.com/mongodb/mongo/commit/1d605e6a9f610a3113783fcc4351ade26f067950

Comment by auto [ 07/Apr/12 ]

Author:

{u'login': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: make NamespaceDetails::(capped|max) private to work on SERVER-5285
Branch: master
https://github.com/mongodb/mongo/commit/6ca732d038405fd541da068ffa577324a57af715

Comment by Daniel Crosta [ 06/Apr/12 ]

Also note: I exhaustively tested that the documents are no longer findable by doing

> db.bigcapped.find({i: NumberLong(0)}).sort({$natural :1}).explain()
{
	"cursor" : "ForwardCappedCursor",
	"nscanned" : 2064887947,
	"nscannedObjects" : 2064887947,
	"n" : 0,
	"millis" : 1585105,
	"nYields" : 0,
	"nChunkSkips" : 0,
	"isMultiKey" : false,
	"indexOnly" : false,
	"indexBounds" : {
		
	},
	"server" : "dcrosta.local:27017"
}

I ran this query after I began to notice that db.collection.find().sort({$natural: 1}).limit(1).next() was not finding the first element I inserted (where i == 0)

Comment by Daniel Crosta [ 06/Apr/12 ]

This seems to be an actual issue where records are deleted "near" 2^31 - 1 records in a capped collection (I say near because I've not observed the count go to 2^31 - 1 and stay there in my testing). Here's how I've repro'd this:

1. Create a huge capped collection (mine is 100G), and do not specify "max" when creating (just specify size)
2. Start inserting data in a way that you can observe when the oldest element starts advancing (I used an incrementing NumberLong counter)
3. Once the collection count as reported in db.collection.stats() gets "near" (say above 95%) of the max (== 2^31 - 1), begin observing the oldest element with db.collection.find().sort({$natural: 1}).limit(1).next()

Expected:

As long as the "size" as reported in stats() is lower than the "storageSize", the first element should not change regardless of the count

Actual:

The first element begins "advancing" once "count" in stats() is within about 5% of "max" (which is 2^31 - 1).

Comment by Richard Kreuter (Inactive) [ 12/Mar/12 ]

Note: I think this would be an on-disk format change, so might have to be handled carefully.

Generated at Thu Feb 08 03:08:26 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.