[SERVER-525] unique index in capped collection Created: 08/Jan/10  Updated: 12/Jul/16  Resolved: 20/Jan/10

Status: Closed
Project: Core Server
Component/s: Stability
Affects Version/s: None
Fix Version/s: 1.3.2

Type: Bug Priority: Major - P3
Reporter: dustin norlander Assignee: Aaron Staple
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Participants:

 Description   

Ability to have a unique index in a capped collection. Currently results in errors:

Fri Jan 8 23:53:09 userassert:E11000 duplicate key errorindex:
twitter.tweet_cache.$id dup key: { : 7537480484 } : on
addIndex/capped - collection and its index will not match
unindex failed (key too big?) twitter.tweet_cache.$id



 Comments   
Comment by Aaron Staple [ 20/Jan/10 ]

I went ahead and implemented the index checking without the dummy strategy I described above, we can optimize later if necessary. I also committed and then reverted my earlier implementation that allows removing the last record inserted into a capped collection, so there's a record of that work in case we want to do something similar in future.

Comment by auto [ 20/Jan/10 ]

Author:{'login': {'remember_token': None, 'auth_token': '2b15f37eaf67ac3f9c029f79f2339eca', 'last_ip': '99.59.73.209', 'billed_on': None, 'disabled': False, 'remember_token_expires_at': None, 'billingid': None, 'billing_attempts': 0, 'upgrade_ignore': None, 'primary':

{'email': 'aaron@10gen.com', 'user_id': 33134, 'primary': None, 'id': 103147}

, 'id': 33134, 'upgrade_accept': None, 'spammy': False, 'bouncing_email': None, 'billing_extra': None, 'updated_at': '2009/12/15 09:22:47 -0800', 'plan': 'free', 'plan_duration': 'month', 'crypted_password': 'f13fa87b01329b155592c62e6309500b90d38c21', 'wants_email': True, 'gift': None, 'created_at': '2008/11/07 00:52:12 -0800', 'login': 'astaple', 'salt': '9127f885b8aa5dfee0cb5d6dd4066dd597dbf5dd', 'gh_role': None}, 'name': 'Aaron', 'email': 'aaron@10gen.com'}
Message: SERVER-525 check for index conflict before inserting into capped collection
http://github.com/mongodb/mongo/commit/f623739aca557c54dec885c482aacfd5c97f74a3

Comment by auto [ 20/Jan/10 ]

Author:{'login': {'remember_token': None, 'auth_token': '2b15f37eaf67ac3f9c029f79f2339eca', 'last_ip': '99.59.73.209', 'billed_on': None, 'disabled': False, 'remember_token_expires_at': None, 'billingid': None, 'billing_attempts': 0, 'upgrade_ignore': None, 'primary':

{'email': 'aaron@10gen.com', 'user_id': 33134, 'primary': None, 'id': 103147}

, 'id': 33134, 'upgrade_accept': None, 'spammy': False, 'bouncing_email': None, 'billing_extra': None, 'updated_at': '2009/12/15 09:22:47 -0800', 'plan': 'free', 'plan_duration': 'month', 'crypted_password': 'f13fa87b01329b155592c62e6309500b90d38c21', 'wants_email': True, 'gift': None, 'created_at': '2008/11/07 00:52:12 -0800', 'login': 'astaple', 'salt': '9127f885b8aa5dfee0cb5d6dd4066dd597dbf5dd', 'gh_role': None}, 'name': 'Aaron', 'email': 'aaron@10gen.com'}
Message: SERVER-525 revert undoInsert implementation
http://github.com/mongodb/mongo/commit/fdde436041be95bf91819f2712e4c41eae36759a

Comment by auto [ 20/Jan/10 ]

Author:{'login': {'remember_token': None, 'auth_token': '2b15f37eaf67ac3f9c029f79f2339eca', 'last_ip': '99.59.73.209', 'billed_on': None, 'disabled': False, 'remember_token_expires_at': None, 'billingid': None, 'billing_attempts': 0, 'upgrade_ignore': None, 'primary':

{'email': 'aaron@10gen.com', 'user_id': 33134, 'primary': None, 'id': 103147}

, 'id': 33134, 'upgrade_accept': None, 'spammy': False, 'bouncing_email': None, 'billing_extra': None, 'updated_at': '2009/12/15 09:22:47 -0800', 'plan': 'free', 'plan_duration': 'month', 'crypted_password': 'f13fa87b01329b155592c62e6309500b90d38c21', 'wants_email': True, 'gift': None, 'created_at': '2008/11/07 00:52:12 -0800', 'login': 'astaple', 'salt': '9127f885b8aa5dfee0cb5d6dd4066dd597dbf5dd', 'gh_role': None}, 'name': 'Aaron', 'email': 'aaron@10gen.com'}
Message: SERVER-525 add more extensive test for undoInsert
http://github.com/mongodb/mongo/commit/0036246413fd1cd6fd96a60c7512f303572d432f

Comment by auto [ 20/Jan/10 ]

Author:{'login': {'remember_token': None, 'auth_token': '2b15f37eaf67ac3f9c029f79f2339eca', 'last_ip': '99.59.73.209', 'billed_on': None, 'disabled': False, 'remember_token_expires_at': None, 'billingid': None, 'billing_attempts': 0, 'upgrade_ignore': None, 'primary':

{'email': 'aaron@10gen.com', 'user_id': 33134, 'primary': None, 'id': 103147}

, 'id': 33134, 'upgrade_accept': None, 'spammy': False, 'bouncing_email': None, 'billing_extra': None, 'updated_at': '2009/12/15 09:22:47 -0800', 'plan': 'free', 'plan_duration': 'month', 'crypted_password': 'f13fa87b01329b155592c62e6309500b90d38c21', 'wants_email': True, 'gift': None, 'created_at': '2008/11/07 00:52:12 -0800', 'login': 'astaple', 'salt': '9127f885b8aa5dfee0cb5d6dd4066dd597dbf5dd', 'gh_role': None}, 'name': 'Aaron', 'email': 'aaron@10gen.com'}
Message: SERVER-525 only need _deleteRecord when rolling back
http://github.com/mongodb/mongo/commit/2597876acbb883cd3b80d17cbc5cf00396429b81

Comment by auto [ 20/Jan/10 ]

Author:{'login': {'remember_token': None, 'auth_token': '2b15f37eaf67ac3f9c029f79f2339eca', 'last_ip': '99.59.73.209', 'billed_on': None, 'disabled': False, 'remember_token_expires_at': None, 'billingid': None, 'billing_attempts': 0, 'upgrade_ignore': None, 'primary':

{'email': 'aaron@10gen.com', 'user_id': 33134, 'primary': None, 'id': 103147}

, 'id': 33134, 'upgrade_accept': None, 'spammy': False, 'bouncing_email': None, 'billing_extra': None, 'updated_at': '2009/12/15 09:22:47 -0800', 'plan': 'free', 'plan_duration': 'month', 'crypted_password': 'f13fa87b01329b155592c62e6309500b90d38c21', 'wants_email': True, 'gift': None, 'created_at': '2008/11/07 00:52:12 -0800', 'login': 'astaple', 'salt': '9127f885b8aa5dfee0cb5d6dd4066dd597dbf5dd', 'gh_role': None}, 'name': 'Aaron', 'email': 'aaron@10gen.com'}
Message: SERVER-525 update error codes
http://github.com/mongodb/mongo/commit/9f3561944572086d0bb883e3665d482eff72d454

Comment by auto [ 20/Jan/10 ]

Author:{'login': {'remember_token': None, 'auth_token': '2b15f37eaf67ac3f9c029f79f2339eca', 'last_ip': '99.59.73.209', 'billed_on': None, 'disabled': False, 'remember_token_expires_at': None, 'billingid': None, 'billing_attempts': 0, 'upgrade_ignore': None, 'primary':

{'email': 'aaron@10gen.com', 'user_id': 33134, 'primary': None, 'id': 103147}

, 'id': 33134, 'upgrade_accept': None, 'spammy': False, 'bouncing_email': None, 'billing_extra': None, 'updated_at': '2009/12/15 09:22:47 -0800', 'plan': 'free', 'plan_duration': 'month', 'crypted_password': 'f13fa87b01329b155592c62e6309500b90d38c21', 'wants_email': True, 'gift': None, 'created_at': '2008/11/07 00:52:12 -0800', 'login': 'astaple', 'salt': '9127f885b8aa5dfee0cb5d6dd4066dd597dbf5dd', 'gh_role': None}, 'name': 'Aaron', 'email': 'aaron@10gen.com'}
Message: SERVER-525 implement undoInsert for capped collection
http://github.com/mongodb/mongo/commit/e5e39b21da286ddad5fc6c0bbbaa03d32a70aa96

Comment by auto [ 20/Jan/10 ]

Author:{'login': {'remember_token': None, 'auth_token': '2b15f37eaf67ac3f9c029f79f2339eca', 'last_ip': '99.59.73.209', 'billed_on': None, 'disabled': False, 'remember_token_expires_at': None, 'billingid': None, 'billing_attempts': 0, 'upgrade_ignore': None, 'primary':

{'email': 'aaron@10gen.com', 'user_id': 33134, 'primary': None, 'id': 103147}

, 'id': 33134, 'upgrade_accept': None, 'spammy': False, 'bouncing_email': None, 'billing_extra': None, 'updated_at': '2009/12/15 09:22:47 -0800', 'plan': 'free', 'plan_duration': 'month', 'crypted_password': 'f13fa87b01329b155592c62e6309500b90d38c21', 'wants_email': True, 'gift': None, 'created_at': '2008/11/07 00:52:12 -0800', 'login': 'astaple', 'salt': '9127f885b8aa5dfee0cb5d6dd4066dd597dbf5dd', 'gh_role': None}, 'name': 'Aaron', 'email': 'aaron@10gen.com'}
Message: SERVER-525 enable case in capped5.js
http://github.com/mongodb/mongo/commit/9d42a8e04bdc4aa99148122c0e51c3745f87f906

Comment by auto [ 20/Jan/10 ]

Author:{'login': {'remember_token': None, 'auth_token': '2b15f37eaf67ac3f9c029f79f2339eca', 'last_ip': '99.59.73.209', 'billed_on': None, 'disabled': False, 'remember_token_expires_at': None, 'billingid': None, 'billing_attempts': 0, 'upgrade_ignore': None, 'primary':

{'email': 'aaron@10gen.com', 'user_id': 33134, 'primary': None, 'id': 103147}

, 'id': 33134, 'upgrade_accept': None, 'spammy': False, 'bouncing_email': None, 'billing_extra': None, 'updated_at': '2009/12/15 09:22:47 -0800', 'plan': 'free', 'plan_duration': 'month', 'crypted_password': 'f13fa87b01329b155592c62e6309500b90d38c21', 'wants_email': True, 'gift': None, 'created_at': '2008/11/07 00:52:12 -0800', 'login': 'astaple', 'salt': '9127f885b8aa5dfee0cb5d6dd4066dd597dbf5dd', 'gh_role': None}, 'name': 'Aaron', 'email': 'aaron@10gen.com'}
Message: SERVER-525 more capped collection / unique index tests
http://github.com/mongodb/mongo/commit/84ec76b935af3a8a848a351ee24dc28688994d97

Comment by Eliot Horowitz (Inactive) [ 20/Jan/10 ]

Yes - replacing the old value with the new value is correct.
The case I'm a little worried about is if you add something which removes something else, but then you have to roll back, so you removed something you didn't have to.
So maybe we should just do 2 btree scans (or do the dummy trick)

Comment by Aaron Staple [ 20/Jan/10 ]

I've spent some time working on an implementation that will allow us to roll back the most recently inserted record in a capped collection if we discover an index conflict. One question I have is this:

Say I want to insert

{i:1,a:2}

and there already is an

{i:1,a:3}

in the capped collection and the index on i is unique. If the capped collection is full when I attempt to insert

{i:1,a:2}

, then

{i:1,a:3}

may be deleted to make room, in which case inserting

{i:1,a:3}

will be allowed. Is that ok?

If we don't want the db to behave this way, we could check for an index conflict before insert, but that would require two btree searches (one to check conflict, another later to insert in the index once we've inserted a record in the capped collection). If we wanted we could insert a dummy value in the btree and keep a reference to the btree node during the validation step, then update this dummy with the correct value once we've inserted into the capped collection.

Comment by Eliot Horowitz (Inactive) [ 13/Jan/10 ]

Right now unique indexes with capped collections result in data being in the collection but not in indexes, etc...

We can either make the rollback remove the item, or if that's a lot of work for now, just not allow unique indexes on capped collections so there isn't inconsistency. If we do that, then make a new feature case for allowing them and link to this case.

Generated at Thu Feb 08 02:54:23 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.