[SERVER-9347] Allow sparse multi-key entries in a compound index Created: 13/Apr/13  Updated: 04/Nov/15  Resolved: 04/Nov/15

Status: Closed
Project: Core Server
Component/s: Index Maintenance
Affects Version/s: 2.2.3
Fix Version/s: None

Type: Bug Priority: Minor - P4
Reporter: James Blackburn Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
Operating System: ALL
Participants:

 Description   

This is similar to SERVER-2193. But I think the use case is compelling, and the existing semantics unintuitive.

Given the following GridFS like schema:
1) Metadata collection for file metadata
2) Chunks collection for file chunks

Files are large, and we support multiple versions of files. We therefore chunk files and sha the chunks such that we can save disk space when most the file hasn't changed.

The chunks collection has a trivial unique index:

coll.ensure_index([('symbol', 1), ('parents', 1), ('chunk', 1)], unique=True, sparse=True)

'symbol' / file name can be used for sharding (not strictly necessary)
'parents' - array of parent metadata documents representing multiple versions
'chunk' - chunk number for a given version of the file

For every version of a file, ('parent', 'chunk') must be unique.

Now this works great, it's fast, you can easily slice out ranges of the file, provides version control, and space savings when most data stays the same between versions.

However a problem arises when you try to delete. If the parents array becomes empty for more than one chunk-version, the unique constraint is violated as (null, 'chunk') can result in duplicates.

For example:

coll.ensure_index([('symbol', 1), ('parents', 1), ('chunk', 1)], unique=True, sparse=True)
 
# Two versions of file 'a', 'b'. They share chunk 1
coll.insert({'symbol':'sym', 'parents':['a', 'b'], 'chunk': '1', 'sha':1})
coll.insert({'symbol':'sym', 'parents':['a'], 'chunk': '2', 'sha':1})
coll.insert({'symbol':'sym', 'parents':['b'], 'chunk': '2', 'sha':2})
 
# Add version 'c'
coll.insert({'symbol':'sym', 'parents':['c'], 'chunk': '1', 'sha':3})
coll.insert({'symbol':'sym', 'parents':['c'], 'chunk': '2', 'sha':4})
 
# Now delete versions 'a', 'b'
coll.update({}, {'$pullAll': {'parents': ['a', 'b']}}, multi=True)
 
Traceback (most recent call last):
  File "/users/is/jblackburn/pyenvs/research/lib/python2.6/site-packages/ipython-0.11_5-py2.6.egg/IPython/core/interactiveshell.py", line 2400, in run_code
    exec code_obj in self.user_global_ns, self.user_ns
  File "<ipython-input-29-5a1fca9e1f55>", line 1, in <module>
    coll.update({}, {'$pullAll': {'parents': ['a', 'b']}}, multi=True)
  File "/local/home/jblackburn/net/pymongo-2.4.2/pymongo/collection.py", line 481, in update
    check_keys, self.__uuid_subtype), safe)
  File "/local/home/jblackburn/net/pymongo-2.4.2/pymongo/mongo_client.py", line 844, in _send_message
    rv = self.__check_response_to_last_error(response)
  File "/local/home/jblackburn/net/pymongo-2.4.2/pymongo/mongo_client.py", line 785, in __check_response_to_last_error
    raise DuplicateKeyError(details["err"])
DuplicateKeyError: E11000 duplicate key error index: jblackburn_scratch.test3.$symbol_1_parents_1_chunk_1  dup key: { : "sym", : undefined, : "2" }

It's great that arrays work as multi-key indexes. However it's less great that the empty array is given a special 'undefined' value.

I can't see how it's useful for documents in a compound unique index, which contains a multi-key field, to be included when that multi-key field is empty. Certainly sparse could reasonably ignore empty multi-key documents in compound indexes.



 Comments   
Comment by Daniel Pasette (Inactive) [ 04/Nov/15 ]

Partial indexes (coming in 3.2.0) can be used here. Using the following index description, you can ensure that only documents whose parents array is non-empty will be included in the index. There isn't currently a better way to express "non-empty array" in the query language. A possible query language extension would be to add a $size operator.

coll.ensureIndex({'symbol': 1, 'parents': 1, 'chunk': 1}, {unique:true, partialFilterExpression:{'parents.0': {$exists:true}}})

Since this can be expressed with partial indexes and we don't want to make a backwards breaking change to indexing semantics, I'm going to close this as a duplicate of the partial indexes ticket. Please comment if you disagree.

Comment by Matt DeKrey [ 27/Jul/13 ]

I encountered this issue as well while using the C# driver, but wrote a work-around.

Pseudo code:
1. Declare the array to be indexed with a custom serialization method that excludes an empty array from being serialized.
2. Ensure the multi-key index is marked as Unique and Sparse.

I hope this helps anyone else that runs across this issue.

Generated at Thu Feb 08 03:20:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.