[SERVER-22678] Error while inserting file in gridFS with pymongo driver Created: 17/Feb/16  Updated: 25/Feb/16  Resolved: 25/Feb/16

Status: Closed
Project: Core Server
Component/s: GridFS, MMAPv1
Affects Version/s: 3.2.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Kedar Aitawdekar Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux Ubuntu14 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux


Attachments: Text File chunk-indexing-info.txt    
Operating System: Linux
Steps To Reproduce:

While inserting a file in gridFS.

Participants:

 Description   

We (project: gstudio) are using mongoDB from last 2yrs for saving files in gridFS. But recently I'm facing problem of inserting files (or chunks) in gridFS in existing DB. It's giving following exception:
assertion src/mongo/db/storage/mmap_v1/btree/key.cpp:612

After deeply analysing code, I comes to conclusion that code is breaking while inserting a chunk from: grid/grid_file.py file at __flush_data method's line: self._chunks.insert(chunk). And because of this following consequences are happening:

  • No chunk and hence no files got inserted into gridfs.
  • It returns fs.files object with length: 0 or probably it gives md5 for file with length 0.
  • So there are many such fs.files objects holding same values for keys: md5 and length.

MongoDB version:
3.2.1

collection:
Collection(Database(ConnectionWrapper: MongoClient('localhost', 27017), u'nroer-mongokit'), u'fs.chunks')

System details:
Linux Ubuntu14 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Following is error trace from diagnostic data capture with directory '/var/lib/mongodb/diagnostic.data':

2016-02-17T14:56:47.655+0530 I -        [conn1029] Assertion failure type == cbindata src/mongo/db/storage/mmap_v1/btree/key.cpp 612
2016-02-17T14:56:47.680+0530 I CONTROL  [conn1029] 
 0x12ea922 0x1288e84 0x1275cf4 0xfddb98 0xfc91aa 0xfd9bdb 0xfd9f92 0xfd964a 0xfd96f8 0xfd96f8 0xfdb85d 0xfc6a0e 0xc8ce6f 0xae2b69 0xae6f0f 0xae6fb0 0xac6225 0xac6591 0xac673b 0xb84b34 0xb878c0 0xb87ae3 0xb880fe 0xb8b1a8 0xba33a9 0xba4046 0xb01b70 0xcac5c5 0xcaee86 0x995bcc 0x129850d 0x7f19deb24182 0x7f19de85147d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"EEA922"},{"b":"400000","o":"E88E84"},{"b":"400000","o":"E75CF4"},{"b":"400000","o":"BDDB98"},{"b":"400000","o":"BC91AA"},{"b":"400000","o":"BD9BDB"},{"b":"400000","o":"BD9F92"},{"b":"400000","o":"BD964A"},{"b":"400000","o":"BD96F8"},{"b":"400000","o":"BD96F8"},{"b":"400000","o":"BDB85D"},{"b":"400000","o":"BC6A0E"},{"b":"400000","o":"88CE6F"},{"b":"400000","o":"6E2B69"},{"b":"400000","o":"6E6F0F"},{"b":"400000","o":"6E6FB0"},{"b":"400000","o":"6C6225"},{"b":"400000","o":"6C6591"},{"b":"400000","o":"6C673B"},{"b":"400000","o":"784B34"},{"b":"400000","o":"7878C0"},{"b":"400000","o":"787AE3"},{"b":"400000","o":"7880FE"},{"b":"400000","o":"78B1A8"},{"b":"400000","o":"7A33A9"},{"b":"400000","o":"7A4046"},{"b":"400000","o":"701B70"},{"b":"400000","o":"8AC5C5"},{"b":"400000","o":"8AEE86"},{"b":"400000","o":"595BCC"},{"b":"400000","o":"E9850D"},{"b":"7F19DEB1C000","o":"8182"},{"b":"7F19DE757000","o":"FA47D"}],"processInfo":{ "mongodbVersion" : "3.2.1", "gitVersion" : "a14d55980c2cdc565d4704a7e3ad37e4e535c1b2", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.13.0-77-generic", "version" : "#121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "D76764D44DE9B088362776AF243199FDCF5756E8" }, { "b" : "7FFF85981000", "elfType" : 3, "buildId" : "2BB17317A4E4F907D13D05EBB9665AC8184C1E6E" }, { "b" : "7F19DFD41000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "D08DD65F97859C71BB2CBBF1043BD968EFE18AAD" }, { "b" : "7F19DF966000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "F86FA9FB4ECEB4E06B40DBDF761A4172B70A4229" }, { "b" : "7F19DF75E000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "92FCF41EFE012D6186E31A59AD05BDBB487769AB" }, { "b" : "7F19DF55A000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C1AE4CB7195D337A77A3C689051DABAA3980CA0C" }, { "b" : "7F19DF256000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "D0E735DBECD63462DA114BD3F76E6EC7BB1FACCC" }, { "b" : "7F19DEF50000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "1D76B71E905CB867B27CEF230FCB20F01A3178F5" }, { "b" : "7F19DED3A000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7F19DEB1C000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9318E8AF0BFBE444731BB0461202EF57F7C39542" }, { "b" : "7F19DE757000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "30C94DC66A1FE95180C3D68D2B89E576D5AE213C" }, { "b" : "7F19DFFA0000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9F00581AB3C73E3AEA35995A0C50D24D59A01D47" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x12ea922]
 mongod(_ZN5mongo10logContextEPKc+0x134) [0x1288e84]
 mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xB4) [0x1275cf4]
 mongod(_ZNK5mongo5KeyV18dataSizeEv+0xE8) [0xfddb98]
 mongod(+0xBC91AA) [0xfc91aa]
 mongod(_ZN5mongo10BtreeLogicINS_13BtreeLayoutV1EE5splitEPNS_16OperationContextEPNS_13BtreeBucketV1ENS_7DiskLocEiS7_RKNS_5KeyV1ES7_S7_+0x32B) [0xfd9bdb]
 mongod(_ZN5mongo10BtreeLogicINS_13BtreeLayoutV1EE10insertHereEPNS_16OperationContextENS_7DiskLocEiRKNS_5KeyV1ES5_S5_S5_+0x152) [0xfd9f92]
 mongod(_ZN5mongo10BtreeLogicINS_13BtreeLayoutV1EE7_insertEPNS_16OperationContextEPNS_13BtreeBucketV1ENS_7DiskLocERKNS_5KeyV1ES7_bS7_S7_+0x1AA) [0xfd964a]
 mongod(_ZN5mongo10BtreeLogicINS_13BtreeLayoutV1EE7_insertEPNS_16OperationContextEPNS_13BtreeBucketV1ENS_7DiskLocERKNS_5KeyV1ES7_bS7_S7_+0x258) [0xfd96f8]
 mongod(_ZN5mongo10BtreeLogicINS_13BtreeLayoutV1EE7_insertEPNS_16OperationContextEPNS_13BtreeBucketV1ENS_7DiskLocERKNS_5KeyV1ES7_bS7_S7_+0x258) [0xfd96f8]
 mongod(_ZN5mongo10BtreeLogicINS_13BtreeLayoutV1EE6insertEPNS_16OperationContextERKNS_7BSONObjERKNS_7DiskLocEb+0x3FD) [0xfdb85d]
 mongod(+0xBC6A0E) [0xfc6a0e]
 mongod(_ZN5mongo17IndexAccessMethod6insertEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdERKNS_19InsertDeleteOptionsEPl+0x18F) [0xc8ce6f]
 mongod(_ZN5mongo12IndexCatalog21_indexFilteredRecordsEPNS_16OperationContextEPNS_17IndexCatalogEntryERKSt6vectorINS_10BsonRecordESaIS6_EE+0x109) [0xae2b69]
 mongod(_ZN5mongo12IndexCatalog13_indexRecordsEPNS_16OperationContextEPNS_17IndexCatalogEntryERKSt6vectorINS_10BsonRecordESaIS6_EE+0x11F) [0xae6f0f]
 mongod(_ZN5mongo12IndexCatalog12indexRecordsEPNS_16OperationContextERKSt6vectorINS_10BsonRecordESaIS4_EE+0x80) [0xae6fb0]
 mongod(_ZN5mongo10Collection16_insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_b+0x325) [0xac6225]
 mongod(_ZN5mongo10Collection15insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_bb+0x1B1) [0xac6591]
 mongod(_ZN5mongo10Collection14insertDocumentEPNS_16OperationContextERKNS_7BSONObjEbb+0x6B) [0xac673b]
 mongod(_ZN5mongo18WriteBatchExecutor10insertManyEPNS0_16ExecInsertsStateEmmPNS_5CurOpEPSt6vectorIPNS_16WriteErrorDetailESaIS7_EEb+0xC84) [0xb84b34]
 mongod(_ZN5mongo18WriteBatchExecutor11execInsertsERKNS_21BatchedCommandRequestEPSt6vectorIPNS_16WriteErrorDetailESaIS6_EE+0x350) [0xb878c0]
 mongod(_ZN5mongo18WriteBatchExecutor11bulkExecuteERKNS_21BatchedCommandRequestEPSt6vectorIPNS_19BatchedUpsertDetailESaIS6_EEPS4_IPNS_16WriteErrorDetailESaISB_EE+0x63) [0xb87ae3]
 mongod(_ZN5mongo18WriteBatchExecutor12executeBatchERKNS_21BatchedCommandRequestEPNS_22BatchedCommandResponseE+0x1DE) [0xb880fe]
 mongod(_ZN5mongo8WriteCmd3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderE+0x248) [0xb8b1a8]
 mongod(_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE+0x3F9) [0xba33a9]
 mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE+0x3E6) [0xba4046]
 mongod(_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE+0x1F0) [0xb01b70]
 mongod(+0x8AC5C5) [0xcac5c5]
 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x6C6) [0xcaee86]
 mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0xEC) [0x995bcc]
 mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x26D) [0x129850d]
 libpthread.so.0(+0x8182) [0x7f19deb24182]
 libc.so.6(clone+0x6D) [0x7f19de85147d]
-----  END BACKTRACE  -----



 Comments   
Comment by Kelsey Schubert [ 25/Feb/16 ]

That's great to hear, kedar2a.

Since the issue has been resolved, I'm going to close the ticket.

Regards,
Thomas

Comment by Kedar Aitawdekar [ 21/Feb/16 ]

Hello anonymous.user,

Thank you for your continuous consideration into the issue.
I have tried following on your suggestions:

  • reIndex()
  • repairDatabase()

And it solved my problem. Now files are getting into the gridFS.

Thank You anonymous.user!

Comment by Kelsey Schubert [ 19/Feb/16 ]

Hi kedar2a,

Thank you for the information. Unfortunately, removing recently added files or chunks will not resolve the issue.

I have a few additional questions as we continue to investigate this issue:

  1. Can you list the upgrade path of this node prior to MongoDB 3.2.1?
  2. Can you successfully insert a file using mongofiles?
  3. Would you be able to upload a tarball of your dbpath? I have created a secure portal here. These files would only be visible to MongoDB employees investigating this issue.

Assuming you are unable to insert files using mongofiles, you have two options since rebuilding the indexes did not correct the issue.

The first is repairDatabase(). If the data files are not corrupted, repairDatabase will retain every document. However, if the the data files are corrupted. This method may discard some documents that are physically present in the data files, but are unreachable from the "root" data structure in the .ns file.

mongodump --repair is your second option. It has a higher chance of finding all structural valid documents that can be reached, but also has a high probability of dumping the same documents multiple times as well as different versions of the same documents. This method relies on you to determine which duplicate represents the most current version of your data.

Kind regards,
Thomas

Comment by Kedar Aitawdekar [ 18/Feb/16 ]

Hi Thomas Schubert,

Thank you, for your attention into problem.

Following is replies to your set of questions:

  • If this node is part of a replicaset, are other members affected?
    • No. We are not using replication.
  • Around this operation, were there any other server errors logged?
    • No
  • Are you using journaling?
    • Yes
  • What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?
    • We are using ubuntu system having ext4 file system.
    • Storage devices are attached locally.
    • We are using HDDs.
  • Have you manipulated (coppied or moved) the underling database files?
  • Have you ever restored this instance from backups?
    • We had done copying of underlining DB files long back (say a 6 months before). And after that we usedsystem for all sorts of DB operations.*
  • What method do you use to create backups?
    • Using command: mongodump.

Some additional information:

  • Regarding operations with files/chunks, I'm able to read the files (or chunks) smoothly without any error. Only while saving file/s (or indirectly chunks) using fileobj_name.fs.files.put there is an issue.
  • I checked the indexes on fs.chunks, attaching chunk-indexing-info.txt file for additional info on chunks. And I think there is no any issue regarding indexing on chunks.

Some queries:

  • Will removing recently added files/chunks resolve the issue?
  • We can't afford to lose the data. What could be strategy so that we could presist good/non-corrupted data?
Comment by Kedar Aitawdekar [ 18/Feb/16 ]

Some of the operations performed on db.fs.chunks regarding indexing in mongo shell.

Comment by Kelsey Schubert [ 17/Feb/16 ]

Hi kedar2a,

The stack trace that you have provided indicates that some or all of the data data files have become corrupt in some way. In these situations, the corruption may be in the index or the data itself. Unfortunately, it is very difficult to determine whether the corruption is isolated beyond the file level.

I have compiled a list of routine questions about data storage and the configuration of your environment. We can use these questions to help get a better understanding of what is going on here.

  1. If this node is part of a replicaset, are other members affected?
  2. Around this operation, were there any other server errors logged?
  3. Are you using journaling?
  4. What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?
  5. Have you manipulated (coppied or moved) the underling database files?
  6. Have you ever restored this instance from backups?
  7. What method do you use to create backups?

I would recommend a clean resync from a node that is not affected. If that is not possible, I would recommend executing repairDatabase.

I would also suggest that you check the integrity of the affected nodes's disk drives. If the assertion failures continue to persist, you may need to replace them.

Regards,
Thomas

Generated at Thu Feb 08 04:01:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.