[CSHARP-896] GridFS C# driver Unique Index error with sharding Created: 22/Jan/14  Updated: 23/Jan/14  Resolved: 23/Jan/14

Status: Closed
Project: C# Driver
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Hari Dosapati Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File GridFS Error.PNG    

 Description   

When trying to insert a GridFS record into a shard collection and replicateset cluster env using the below code getting error as mentined in the image

db.GridFS.Upload(mongoDocs.m_stream[docItem], mongoDocs.m_fileInfo[docItem].Name) 

We haven't created any index on the fs.chunks collection.

Error is :

can't use unique indexes with sharding  
ns:researchdocs.fs.chunks key: { files_id: 1, n: 1 }



 Comments   
Comment by Hari Dosapati [ 23/Jan/14 ]

I changed shard key and its good now. Thanks for all the support. You can close the ticket.

Comment by Robert Stam [ 23/Jan/14 ]

You can shard your fs.chunks collection on either:

1. { files_id : 1 }
2. { files_id : 1, n : 1 }

I've tested that both shard keys work when using GridFS with the .NET driver.

It is usually not necessary to shard the fs.files collection because it is so small compared to the fs.chunks collection.

Comment by Matt Kalan [ 22/Jan/14 ]

Hari, as I suspected we do document that the standard shard key for the chunks collection on GridFS is

{files_id: 1, n: 1}

so that is consistent with what you need as well. The shard key index is unique also so you won't get that error.

You can follow those steps to set up the shard key (as you probably went through already):
http://docs.mongodb.org/manual/tutorial/shard-gridfs-data/

Comment by Matt Kalan [ 22/Jan/14 ]

Hari, just talking with Andre to confirm what you talked about last week, you can generate files._id and then the chunks.files_id is necessarily that same ID to its matching files documents. You do not need to generate the chunks._id as largely that is not used other than for uniqueness. When Andre said id last week, he didn't specify which ID it was.

If you shard the files collection on files._id and the chunks collection on chunks.files_id, then the files and its chunks will be archived together assuming the same tag sharding ranges for both collections.

However, if the GridFS API created the unique index on

{files_id: 1, n: 1}

, then you would probably get the same error even with that. We'll review and figure out the right configuration.

Comment by Robert Stam [ 22/Jan/14 ]

I've emailed Andrew and Matt to ask them to take a look at this. Maybe there's something new that I haven't heard about...

The _id of the fs.chunks collection is already randomly generated, and you don't have any control over that (each chunk needs a unique _id anyway). What really matters is the files_id value (which is the same as the _id of the corresponding entry in the fs.files collection), and you are able to set that _id to a custom value if you want. You would use MongoCreateOptions, like this:

var gridFS = database.GridFS;
Stream stream = null; // some stream
BsonValue customId = 1234; // your custom _id
var remoteFileName = "somename";
var createOptions = new MongoGridFSCreateOptions { Id = customId };
var result = gridFS.Upload(stream, remoteFileName, createOptions);

Comment by Hari Dosapati [ 22/Jan/14 ]

As per our requirement, we need to shard by a key thats generated by us, can we create this key by programariclly? If so, please help us with a sample.
FYI: We worked with Andrew Spiegel and Matt Kalan from your team last week and designed the shard keys based on our requirements.

Comment by Robert Stam [ 22/Jan/14 ]

That would explain it. Try sharding on the files_id instead.

Comment by Hari Dosapati [ 22/Jan/14 ]

Yes, it is sharded. Shard key is _id

Comment by Robert Stam [ 22/Jan/14 ]

Is your researchdocs.fs.chunks collection sharded?

If so, what is the shard key?

GridFS requires that the shard key for the chunks collection be the files_id.

Generated at Wed Feb 07 21:38:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.