[CSHARP-546] not able to upload files to gridfs Created: 22/Aug/12  Updated: 20/Mar/14  Resolved: 24/Aug/12

Status: Closed
Project: C# Driver
Component/s: None
Affects Version/s: 1.5
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Ramesh S Assignee: Robert Stam
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Stage



 Description   

Receiving the below error while uploading files to GridFs using c# drivers (official), same code is working fine in our QA/Dev environments.

Command 'filemd5' failed: need an index on

{ files_id : 1 , n : 1 }

(response: { "errmsg" : "need an index on

{ files_id : 1 , n : 1 }

", "ok" : 0.0 })



 Comments   
Comment by Robert Stam [ 24/Aug/12 ]

Glad to hear that. Thanks for letting us know.

Comment by Ramesh S [ 24/Aug/12 ]

thanks for your prompt reposne and all your help Rob
dropping and recreating indexes fixed the issue - now I can upload files to GridFS

Comment by Robert Stam [ 24/Aug/12 ]

There might be entries in the log. Something like:

Thu Aug 23 20:18:42 [conn841] command csharpdriverunittests.$cmd command: { deleteIndexes: "fs.chunks", index: "*" } ntoreturn:1 keyUpdates:0 locks(micros) w:12588166 reslen:106 12587ms
Thu Aug 23 20:19:05 [conn841] build index csharpdriverunittests.fs.chunks { files_id: 1.0, n: 1.0 }
Thu Aug 23 20:19:05 [conn841] build index done.  scanned 1 total records. 0.002 secs

Comment by Ramesh S [ 23/Aug/12 ]

thanks Rob
I will try these and let you the results.
Several people use this mongo instance. I am not sure who has changed this, I will try to reach out to everyone who is using this instance and try to find how it got changed.

Comment by Robert Stam [ 23/Aug/12 ]

Actually, that index is all wrong (though we don't know why). Your key is:

"key" : {
    "{files_id:1, n:1}" : 1,
    "{unique: true}" : 1
}

the correct key is:

"key" : {
    "files_id" : 1,
    "n" : 1
}

You should drop that index and recreate it. You can do this in the shell:

> db.fs.chunks.dropIndexes()
> db.fs.chunks.ensureIndex({files_id:1,n:1},{unique:true})
> db.fs.chunks.getIndexes()

It would be nice to know how the index ended up being wrong. Do you have any insight into that?

Comment by Ramesh S [ 23/Aug/12 ]

Thanks for the prompt reposne Rob,
I am not sure how the index name got changed - , you want me to try dropping the index and recreating?
If so pls let me know what commands do I need to run -

Comment by Robert Stam [ 23/Aug/12 ]

Thanks for that last test. That's really helpful.

Do you happen to know how the second index in fs.chunks got that unusual name? I'm wondering if the name is throwing the server off. When I run getIndexes on my computer I have the same indexes but the name is different:

> db.fs.chunks.getIndexes()
[
        {
                "v" : 1,
                "key" : {
                        "_id" : 1
                },
                "ns" : "csharpdriverunittests.fs.chunks",
                "name" : "_id_"
        },
        {
                "v" : 1,
                "key" : {
                        "files_id" : 1,
                        "n" : 1
                },
                "unique" : true,
                "ns" : "csharpdriverunittests.fs.chunks",
                "name" : "files_id_1_n_1"
        }
]

Comment by Ramesh S [ 23/Aug/12 ]

Below are the reults:
{ "errmsg" : "need an index on

{ files_id : 1 , n : 1 }

", "ok" : 0 }

And also I ran getIndexes() after that, and it returns indexes

PRIMARY> use dpn
switched to db dpn
PRIMARY> var id=db.fs.files.findOne()._id
PRIMARY> db.runCommand(

{filemd5:id, root:"fs"}

)
{ "errmsg" : "need an index on

{ files_id : 1 , n : 1 }

", "ok" : 0 }
PRIMARY> var id = db.fs.files.findOne()._id
PRIMARY> db.runCommand(

{ filemd5 : id, root : "fs" }

)
{ "errmsg" : "need an index on

{ files_id : 1 , n : 1 }

", "ok" : 0 }
PRIMARY> db.fs.files.getIndexes()
[
{
"v" : 1,
"key" :

{ "_id" : 1 }

,
"ns" : "dpn.fs.files",
"name" : "id"
}
]
PRIMARY> db.fs.chunks.getIndexes()
[
{
"v" : 1,
"key" :

{ "_id" : 1 }

,
"ns" : "dpn.fs.chunks",
"name" : "id"
},
{
"v" : 1,
"key" : {
"

{files_id:1, n:1}

" : 1,
"

{unique: true}

" : 1
},
"ns" : "dpn.fs.chunks",
"name" : "

{files_id:1, n:1}

1

{unique: true}

_1"
}
]

Comment by Robert Stam [ 23/Aug/12 ]

I'm going to find someone else to look at this with me... so far it seems like it should work.

Here's another test you can try, which is to run an md5 command manually from the mongo shell:

> use dpn
> var id = db.fs.files.findOne()._id
> db.runCommand({ filemd5 : id, root : "fs" })

This runs an md5 command using an arbitrary files _id (the first one it finds). It does not alter any data on the server. You should get a reply that looks like this:

{ "numChunks" : 1, "md5" : "b10a8db164e0754105b7a99be72e3fe5", "ok" : 1 }

Comment by Ramesh S [ 23/Aug/12 ]

yes I did:
I ran it again, below are the results:

C:\mongodb-win32-i386-2.0.5\bin>mongo ***.us.deloitte.com:27018
MongoDB shell version: 2.0.5
connecting to: ****:27018/test
PRIMARY> use dpn
switched to db dpn
PRIMARY> db.fs.files.count()
42854
PRIMARY> db.fs.chunks.getIndexes()
[
{
"v" : 1,
"key" :

{ "_id" : 1 }

,
"ns" : "dpn.fs.chunks",
"name" : "id"
},
{
"v" : 1,
"key" : {
"

{files_id:1, n:1}

" : 1,
"

{unique: true}

" : 1
},
"ns" : "dpn.fs.chunks",
"name" : "

{files_id:1, n:1}

1

{unique: true}

_1"
}
]

Comment by Robert Stam [ 23/Aug/12 ]

Just to make sure... in your earlier reply where you checked the indexes, you did do a "use dpn" first, right?

> use dpn
> db.fs.chunks.getIndexes()

Comment by Ramesh S [ 23/Aug/12 ]

database.GridFS.Chunks.FullName : "dpn.fs.chunks"

I verified it agaian and I am sure that same server is being used:

connectionString = "mongodb://**:27018,**:27018,**:27018,***:27018/dpn?replicaSet=rs1;safe=true"

Comment by Robert Stam [ 23/Aug/12 ]

No, if the indexes already exist there is no point in dropping them and recreating them.

However, the server is saying that the index doesn't exist, so something is not right here...

Can you set a breakpoint on line 122 of the PersistPhotos method of your BaseRepository class where you call MongoGridFS.Upload and exxamine the value of the following (where gridFS is whatever variable you are using to access GridFS):

gridFS.Chunks.FullName

and verify that your program is actually using the same database and chunks collection that you examined in the mongo shell?

Comment by Ramesh S [ 23/Aug/12 ]

I verified connection strings and they are same.
So I need to drop all indexes from fs.chunks and ensureIndex on "_id" field?

Comment by Robert Stam [ 23/Aug/12 ]

The error message you are seeing is coming from the server, so there is nothing you can do at the client side (either in your application or the driver) other than to ensure that the required indexes exist.

According to the mongo shell the indexes do exist... which is hard to explain. Is it possible that your application is pointed at one mongo server instance but your mongo shell is pointed at a different one? You could compare the connection string used by your application to the command line arguments you passed the mongo shell.

Also, make sure you are using the right database before checking the indexes on fs.files and fs.chunks.

There is no need to drop the collection(s). At the most you would need to call ensureIndex to recreate the indexes if you can figure out whether they are actually missing.

Comment by Ramesh S [ 23/Aug/12 ]

we can drop the file collection if thats going to help resolve the issue

Comment by Ramesh S [ 23/Aug/12 ]

Thanks Rob,
Indexes are there in the collection - I am not sure if any indexes were dropped/recreated.
Is there anyway to address this issue ? We need to upload pictures to our production environment today.

Error details and stats are below:

PRIMARY> db.fs.files.count()
42854
PRIMARY> db.fs.chunks.getIndexes()
[
{
"v" : 1,
"key" :

{ "_id" : 1 }

,
"ns" : "dpn.fs.chunks",
"name" : "id"
},
{
"v" : 1,
"key" : {
"

{files_id:1, n:1}

" : 1,
"

{unique: true}

" : 1
},
"ns" : "dpn.fs.chunks",
"name" : "

{files_id:1, n:1}

1

{unique: true}

_1"
}
]

Error:
Command 'filemd5' failed: need an index on

{ files_id : 1 , n : 1 }

(response: { "errmsg" : "need an index on

{ files_id : 1 , n : 1 }

", "ok" : 0.0 })

Stack trace:

at MongoDB.Driver.MongoDatabase.RunCommandAs(Type commandResultType, IMongoCommand command)
at MongoDB.Driver.MongoDatabase.RunCommandAs[TCommandResult](IMongoCommand command)
at MongoDB.Driver.MongoDatabase.RunCommand(IMongoCommand command)
at MongoDB.Driver.GridFS.MongoGridFS.Upload(Stream stream, String remoteFileName, MongoGridFSCreateOptions createOptions)
at MongoDB.Driver.GridFS.MongoGridFS.Upload(Stream stream, String remoteFileName)
at Deloitte.Mobile.PeopleNetwork.DataManagement.Data.Repositories.BaseRepository.PersistPhotos() in C:\DeloitteVSProjects\SPNServices\trunk\Deloitte.Mobile.PeopleNetwork.DataManagement.Data\Repositories\BaseRepository.cs:line 122
at Deloitte.Mobile.PeopleNetwork.DataManagement.Importer.MongoImporter.ImportPhotos() in C:\DeloitteVSProjects\SPNServices\trunk\Deloitte.Mobile.PeopleNetwork.DataManagement\Importer\MongoImporter.cs:line 151
at Deloitte.Mobile.PeopleNetwork.DataManagement.Importer.MongoImporter.Import() in C:\DeloitteVSProjects\SPNServices\trunk\Deloitte.Mobile.PeopleNetwork.DataManagement\Importer\MongoImporter.cs:line 137
at Deloitte.Mobile.PeopleNetwork.DataManagement.ConsoleApp.Program.Main(String[] args) in C:\DeloitteVSProjects\SPNServices\trunk\Deloitte.Mobile.PeopleNetwork.DataManagement.ConsoleApp\Program.cs:line 14
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()

Comment by Robert Stam [ 23/Aug/12 ]

Can you please provide a stack trace?

The error message is indicating that an index is missing. Usually the index is created automatically, but all drivers only create the indexes automatically if the fs.files collection contains less than 1000 files. Any chance you deleted any indexes yourself?

You can check how many files you already have using the mongo shell:

> use <insert your database name here>
> db.fs.files.count()

You can see which indexes the fs.chunks collection (the one that is missing the index) has using:

> db.fs.chunks.getIndexes()

Generated at Wed Feb 07 21:37:09 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.