[SERVER-12375] Improper file access Created: 16/Jan/14  Updated: 10/Dec/14  Resolved: 03/Jul/14

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 2.4.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Bill Shipman Assignee: Unassigned
Resolution: Cannot Reproduce Votes: 0
Labels: database.cpp, files,, server
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux stsia0408 2.6.34-12-desktop #1 SMP PREEMPT x86_64 x86_64 x86_64 GNU/Linux (suse)


Operating System: Linux
Steps To Reproduce:

unknown

Participants:

 Description   

Attempting to use a db with approximately 50 collections using 604GB of disk space. Running db.stats() abruptly fails along with clients going into a "waiting" state.

/var/log/mongodb/mongo.log includes the line:
[conn1] getFile(304) _files.size:304 /var/lib/mongodb/preannotation.304

That file doesn't exist:

mongodb@stsia0408:/var/log/mongodb> ls /var/lib/mongodb/preannotation.30*
/var/lib/mongodb/preannotation.30 /var/lib/mongodb/preannotation.301 /var/lib/mongodb/preannotation.303
/var/lib/mongodb/preannotation.300 /var/lib/mongodb/preannotation.302

Wed Jan 15 17:06:30.062 [initandlisten] MongoDB starting : pid=15707 port=27017 dbpath=/var/lib/mongodb 64-bit host=stsia0408
Wed Jan 15 17:06:30.062 [initandlisten] db version v2.4.3
Wed Jan 15 17:06:30.062 [initandlisten] git version: nogitversion
Wed Jan 15 17:06:30.062 [initandlisten] build info: Linux stsia0508 2.6.34-12-desktop #1 SMP PREEMPT 2010-06-29 02:39:08 +0200 x86_64 BOOST_LIB_VERSION=1_49
Wed Jan 15 17:06:30.062 [initandlisten] allocator: tcmalloc
Wed Jan 15 17:06:30.062 [initandlisten] options:

{ config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", fork: "true", logappend: "true", logpath: "/var/log/mongodb/mongodb.log", nohttpinterface: "true", oplogSize: 8192, quiet: true }

Wed Jan 15 17:06:30.073 [initandlisten] journal dir=/var/lib/mongodb/journal
Wed Jan 15 17:06:30.073 [initandlisten] recover : no journal files present, no recovery needed
Wed Jan 15 17:06:30.113 [initandlisten] waiting for connections on port 27017
Wed Jan 15 17:06:42.119 [conn1] error: getFile() called in a read lock, yet file to return is not yet open
Wed Jan 15 17:06:42.120 [conn1] getFile(304) _files.size:304 /var/lib/mongodb/preannotation.304
Wed Jan 15 17:06:42.120 [conn1] context ns: preannotation.
Wed Jan 15 17:06:42.120 [conn1] preannotation. Assertion failure false src/mongo/db/database.cpp 300
0xda0006 0xd48e6f 0x8f6577 0x8e2d92 0xa899ae 0x9087d5 0x8fe52f 0x90032a 0x900d1b 0xaa2a8a 0xaa72e4 0xa2d1b1 0x73a131 0xd82520 0x7f65530d0a4f 0x7f6551eaf82d
/usr/sbin/mongod(_ZN5mongo15printStackTraceERSo+0x26) [0xda0006]
/usr/sbin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xbf) [0xd48e6f]
/usr/sbin/mongod(_ZN5mongo8Database7getFileEiib+0x317) [0x8f6577]
/usr/sbin/mongod(_ZN5mongo11DataFileMgr9getExtentERKNS_7DiskLocE+0x42) [0x8e2d92]
/usr/sbin/mongod(_ZNK5mongo16NamespaceDetails11storageSizeEPiPNS_16BSONArrayBuilderE+0x1e) [0xa899ae]
/usr/sbin/mongod(_ZN5mongo7DBStats3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x2d5) [0x9087d5]
/usr/sbin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x2f) [0x8fe52f]
/usr/sbin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xb4a) [0x90032a]
/usr/sbin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x5bb) [0x900d1b]
/usr/sbin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x3a) [0xaa2a8a]
/usr/sbin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0xce4) [0xaa72e4]
/usr/sbin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xc91) [0xa2d1b1]
/usr/sbin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xd1) [0x73a131]
/usr/sbin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x480) [0xd82520]
/lib64/libpthread.so.0(+0x6a4f) [0x7f65530d0a4f]
/lib64/libc.so.6(clone+0x6d) [0x7f6551eaf82d]



 Comments   
Comment by Thomas Rueckstiess [ 03/Jul/14 ]

Hi Bill,

as mentioned I'm resolving this as "cannot reproduce" since we don't have enough information to diagnose. Please feel free to re-open the issue if this should happen again.

Thanks,
Thomas

Comment by Thomas Rueckstiess [ 26/Jun/14 ]

Hi Bill,

Thanks for your additional input. Unfortunately, without any more details it's hard to track down the issue. Has this happened again since? If not, I think we have to mark the ticket as "cannot reproduce" for now, and you can re-open it if the problem appears again and you are able to collect fresh information.

Thanks,
Thomas

Comment by Bill Shipman [ 26/Jun/14 ]

It seems like we ran the repair tool maybe twice and it cleared it up. At this point I don't clearly recall what we did. Looking in my email from January it looks like we first ran a repair and then a ? mongodump repair ?

No nothing should have purged that file, it's in a limited access file tree on a machine with only 3 users in total. Myself, another programmer and the sysadmin. It is configured as a high availability replication cluster so that would have been the only thing migrating data other than a few (<10) processes doing inserts on the master but those are heavy loads.

Comment by Thomas Rueckstiess [ 26/Jun/14 ]

Hi Bill,

Sorry for the delay on this issue. It appears that the file /var/lib/mongodb/preannotation.304 was deleted somehow, as it doesn't show up on the ls anymore. I'm not aware of any reports of files being deleted due to a bug caused by MongoDB. Are you aware of anything that could have deleted the file? Any data migration, permission issues etc?

Have you recovered from this situation since reporting this?

Regards,
Thomas

Generated at Thu Feb 08 03:28:23 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.