[SERVER-5440] fd leak from opened and unclosed conns in webServerThread Created: 29/Mar/12  Updated: 29/Mar/12  Resolved: 29/Mar/12

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 2.0.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Greg Studer Assignee: Unassigned
Resolution: Duplicate Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
duplicates SERVER-4866 prevent Socket fd leaks Closed
Operating System: ALL
Participants:

 Description   

Apparently we're bad at closing connections somewhere in the web server thread. Valgrind --track-fds output from user with the problem:

==10784== Nulgrind, the minimal Valgrind tool
==10784== Copyright (C) 2002-2009, and GNU GPL'd, by Nicholas Nethercote.
==10784== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==10784== Command: /usr/bin/mongod --quiet -f /etc/mongodb.conf
==10784== Parent PID: 11981
==10784== 
==10784== 
==10784== FILE DESCRIPTORS: 5 open at exit.
==10784== Open file descriptor 4: /dev/urandom
==10784==    at 0x32D8CC5B40: __open_nocancel (in /lib64/libc-2.5.so)
==10784==    by 0x32D8C6C2F2: _IO_file_open (in /lib64/libc-2.5.so)
==10784==    by 0x32D8C6C43B: _IO_file_fopen@@GLIBC_2.2.5 (in /lib64/libc-2.5.so)
==10784==    by 0x32D8C61763: __fopen_internal (in /lib64/libc-2.5.so)
==10784==    by 0x32DA0BBC33: std::__basic_file<char>::open(char const*, std::_Ios_Openmode, int) (in /usr/lib64/libstdc++.so.6.0.8)
==10784==    by 0x32DA06A522: std::basic_filebuf<char, std::char_traits<char> >::open(char const*, std::_Ios_Openmode) (in /usr/lib64/libstdc++.so.6.0.8)
==10784==    by 0x544F8C: mongo::Security::init() (in /usr/bin/mongod)
==10784==    by 0x54526B: ??? (in /usr/bin/mongod)
==10784==    by 0xACDCC5: ??? (in /usr/bin/mongod)
==10784==    by 0x4FFBD2: ??? (in /usr/bin/mongod)
==10784== 
==10784== Open file descriptor 3: /root/valgrind.log
==10784==    <inherited from parent>
==10784== 
==10784== Open file descriptor 2: /dev/pts/0
==10784==    <inherited from parent>
==10784== 
==10784== Open file descriptor 1: /dev/pts/0
==10784==    <inherited from parent>
==10784== 
==10784== Open file descriptor 0: /dev/pts/0
==10784==    <inherited from parent>
==10784== 
==10784== 
==10784== 
==10784== FILE DESCRIPTORS: 5 open at exit.
==10784== Open file descriptor 4: /dev/urandom
==10784==    at 0x32D8CC5B40: __open_nocancel (in /lib64/libc-2.5.so)
==10784==    by 0x32D8C6C2F2: _IO_file_open (in /lib64/libc-2.5.so)
==10784==    by 0x32D8C6C43B: _IO_file_fopen@@GLIBC_2.2.5 (in /lib64/libc-2.5.so)
==10784==    by 0x32D8C61763: __fopen_internal (in /lib64/libc-2.5.so)
==10784==    by 0x32DA0BBC33: std::__basic_file<char>::open(char const*, std::_Ios_Openmode, int) (in /usr/lib64/libstdc++.so.6.0.8)
==10784==    by 0x32DA06A522: std::basic_filebuf<char, std::char_traits<char> >::open(char const*, std::_Ios_Openmode) (in /usr/lib64/libstdc++.so.6.0.8)
==10784==    by 0x544F8C: mongo::Security::init() (in /usr/bin/mongod)
==10784==    by 0x54526B: ??? (in /usr/bin/mongod)
==10784==    by 0xACDCC5: ??? (in /usr/bin/mongod)
==10784==    by 0x4FFBD2: ??? (in /usr/bin/mongod)
==10784== 
==10784== Open file descriptor 3: /root/valgrind.log
==10784==    <inherited from parent>
==10784== 
==10784== Open file descriptor 2: /dev/pts/0
==10784==    <inherited from parent>
==10784== 
==10784== Open file descriptor 1: /dev/pts/0
==10784==    <inherited from parent>
==10784== 
==10784== Open file descriptor 0: /dev/pts/0==9895== 
==9895== FILE DESCRIPTORS: 14 open at exit.
==9895== Open AF_INET socket 51: 10.1.101.211:39569 <-> unbound
==9895==    at 0x32D940DB3B: ??? (in /lib64/libpthread-2.5.so)
==9895==    by 0x5AB50E: mongo::Listener::initAndListen() (in /usr/bin/mongod)
==9895==    by 0x65459D: mongo::webServerThread(mongo::AdminAccess const*) (in /usr/bin/mongod)
==9895==    by 0xAAB7BF: thread_proxy (in /usr/bin/mongod)
==9895==    by 0x32D940673C: start_thread (in /lib64/libpthread-2.5.so)
==9895==    by 0x32D8CD3F6C: clone (in /lib64/libc-2.5.so)
==9895== 
==9895== Open AF_INET socket 190: 10.1.101.211:50059 <-> unbound
==9895==    at 0x32D940DB3B: ??? (in /lib64/libpthread-2.5.so)
==9895==    by 0x5AB50E: mongo::Listener::initAndListen() (in /usr/bin/mongod)
==9895==    by 0x65459D: mongo::webServerThread(mongo::AdminAccess const*) (in /usr/bin/mongod)
==9895==    by 0xAAB7BF: thread_proxy (in /usr/bin/mongod)
==9895==    by 0x32D940673C: start_thread (in /lib64/libpthread-2.5.so)
==9895==    by 0x32D8CD3F6C: clone (in /lib64/libc-2.5.so)
==9895== 
==9895== Open AF_INET socket 192: 10.1.101.211:47191 <-> unbound
==9895==    at 0x32D940DB3B: ??? (in /lib64/libpthread-2.5.so)
==9895==    by 0x5AB50E: mongo::Listener::initAndListen() (in /usr/bin/mongod)
==9895==    by 0x65459D: mongo::webServerThread(mongo::AdminAccess const*) (in /usr/bin/mongod)
==9895==    by 0xAAB7BF: thread_proxy (in /usr/bin/mongod)
==9895==    by 0x32D940673C: start_thread (in /lib64/libpthread-2.5.so)
==9895==    by 0x32D8CD3F6C: clone (in /lib64/libc-2.5.so)
==9895== 
==9895== Open AF_INET socket 39: 10.1.101.211:35835 <-> unbound
==9895==    at 0x32D940DB3B: ??? (in /lib64/libpthread-2.5.so)
==9895==    by 0x5AB50E: mongo::Listener::initAndListen() (in /usr/bin/mongod)
==9895==    by 0x65459D: mongo::webServerThread(mongo::AdminAccess const*) (in /usr/bin/mongod)
==9895==    by 0xAAB7BF: thread_proxy (in /usr/bin/mongod)
==9895==    by 0x32D940673C: start_thread (in /lib64/libpthread-2.5.so)
==9895==    by 0x32D8CD3F6C: clone (in /lib64/libc-2.5.so)
==9895== 
==9895== Open AF_INET socket 95: 10.1.101.211:46725 <-> unbound
==9895==    at 0x32D940DB3B: ??? (in /lib64/libpthread-2.5.so)
==9895==    by 0x5AB50E: mongo::Listener::initAndListen() (in /usr/bin/mongod)
==9895==    by 0x65459D: mongo::webServerThread(mongo::AdminAccess const*) (in /usr/bin/mongod)
==9895==    by 0xAAB7BF: thread_proxy (in /usr/bin/mongod)
==9895==    by 0x32D940673C: start_thread (in /lib64/libpthread-2.5.so)
==9895==    by 0x32D8CD3F6C: clone (in /lib64/libc-2.5.so)
==9895== 
==9895== Open AF_INET socket 194: 10.1.101.211:42425 <-> unbound
==9895==    at 0x32D940DB3B: ??? (in /lib64/libpthread-2.5.so)
==9895==    by 0x5AB50E: mongo::Listener::initAndListen() (in /usr/bin/mongod)
==9895==    by 0x65459D: mongo::webServerThread(mongo::AdminAccess const*) (in /usr/bin/mongod)
==9895==    by 0xAAB7BF: thread_proxy (in /usr/bin/mongod)
==9895==    by 0x32D940673C: start_thread (in /lib64/libpthread-2.5.so)
==9895==    by 0x32D8CD3F6C: clone (in /lib64/libc-2.5.so)
==9895== 
==9895== Open AF_INET socket 189: 10.1.101.211:57489 <-> unbound
==9895==    at 0x32D940DB3B: ??? (in /lib64/libpthread-2.5.so)
==9895==    by 0x5AB50E: mongo::Listener::initAndListen() (in /usr/bin/mongod)
==9895==    by 0x65459D: mongo::webServerThread(mongo::AdminAccess const*) (in /usr/bin/mongod)
==9895==    by 0xAAB7BF: thread_proxy (in /usr/bin/mongod)
==9895==    by 0x32D940673C: start_thread (in /lib64/libpthread-2.5.so)
==9895==    by 0x32D8CD3F6C: clone (in /lib64/libc-2.5.so)
==9895== 
==9895== Open AF_INET socket 183: 10.1.101.211:42136 <-> unbound
==9895==    at 0x32D940DB3B: ??? (in /lib64/libpthread-2.5.so)
==9895==    by 0x5AB50E: mongo::Listener::initAndListen() (in /usr/bin/mongod)
==9895==    by 0x65459D: mongo::webServerThread(mongo::AdminAccess const*) (in /usr/bin/mongod)
==9895==    by 0xAAB7BF: thread_proxy (in /usr/bin/mongod)
==9895==    by 0x32D940673C: start_thread (in /lib64/libpthread-2.5.so)
==9895==    by 0x32D8CD3F6C: clone (in /lib64/libc-2.5.so)
==9895== 
==9895== Open file descriptor 5: /mongodb/mongod.lock
==9895==    at 0x32D940E32F: open64 (in /lib64/libpthread-2.5.so)
==9895==    by 0x88FD2E: mongo::acquirePathLock(bool) (in /usr/bin/mongod)
==9895==    by 0xA94B91: mongo::_initAndListen(int) (in /usr/bin/mongod)
==9895==    by 0xA9558C: mongo::initAndListen(int) (in /usr/bin/mongod)
==9895==    by 0xA9F2B9: main (in /usr/bin/mongod)
==9895== 
==9895== Open file descriptor 1: /var/log/mongodb/mongodb.log
==9895==    at 0x32D8CC6AC7: dup2 (in /lib64/libc-2.5.so)
==9895==    by 0x586D3B: mongo::LoggingManager::rotate() (in /usr/bin/mongod)
==9895==    by 0x58718D: mongo::LoggingManager::start(std::string const&, bool) (in /usr/bin/mongod)
==9895==    by 0x8F8D91: mongo::CmdLine::store(int, char**, boost::program_options::options_description&, boost::program_options::options_description&, boost::program_options::positional_options_description&, boost::program_options::variables_map&) (in /usr/bin/mongod)
==9895==    by 0xA965C3: main (in /usr/bin/mongod)
==9895== 
==9895== Open file descriptor 2: /var/log/mongodb/mongodb.log
==9895==    at 0x32D8CC6AC7: dup2 (in /lib64/libc-2.5.so)
==9895==    by 0x586D4F: mongo::LoggingManager::rotate() (in /usr/bin/mongod)
==9895==    by 0x58718D: mongo::LoggingManager::start(std::string const&, bool) (in /usr/bin/mongod)
==9895==    by 0x8F8D91: mongo::CmdLine::store(int, char**, boost::program_options::options_description&, boost::program_options::options_description&, boost::program_options::positional_options_description&, boost::program_options::variables_map&) (in /usr/bin/mongod)
==9895==    by 0xA965C3: main (in /usr/bin/mongod)
==9895== 
==9895== Open file descriptor 0: /dev/null
==9895==    at 0x32D8CC5B7B: ??? (in /lib64/libc-2.5.so)
==9895==    by 0x32D8C6C2F2: _IO_file_open (in /lib64/libc-2.5.so)
==9895==    by 0x32D8C6C43B: _IO_file_fopen@@GLIBC_2.2.5 (in /lib64/libc-2.5.so)
==9895==    by 0x32D8C69FA3: freopen64 (in /lib64/libc-2.5.so)
==9895==    by 0x8F8B5A: mongo::CmdLine::store(int, char**, boost::program_options::options_description&, boost::program_options::options_description&, boost::program_options::positional_options_description&, boost::program_options::variables_map&) (in /usr/bin/mongod)
==9895==    by 0xA965C3: main (in /usr/bin/mongod)
==9895== 
==9895== Open file descriptor 4: /dev/urandom
==9895==    at 0x32D8CC5B40: __open_nocancel (in /lib64/libc-2.5.so)
==9895==    by 0x32D8C6C2F2: _IO_file_open (in /lib64/libc-2.5.so)
==9895==    by 0x32D8C6C43B: _IO_file_fopen@@GLIBC_2.2.5 (in /lib64/libc-2.5.so)
==9895==    by 0x32D8C61763: __fopen_internal (in /lib64/libc-2.5.so)
==9895==    by 0x32DA0BBC33: std::__basic_file<char>::open(char const*, std::_Ios_Openmode, int) (in /usr/lib64/libstdc++.so.6.0.8)
==9895==    by 0x32DA06A522: std::basic_filebuf<char, std::char_traits<char> >::open(char const*, std::_Ios_Openmode) (in /usr/lib64/libstdc++.so.6.0.8)
==9895==    by 0x544F8C: mongo::Security::init() (in /usr/bin/mongod)
==9895==    by 0x54526B: ??? (in /usr/bin/mongod)
==9895==    by 0xACDCC5: ??? (in /usr/bin/mongod)
==9895==    by 0x4FFBD2: ??? (in /usr/bin/mongod)
==9895== 
==9895== Open file descriptor 3: /root/valgrind.log
==9895==    <inherited from parent>

this results in lots of orphaned fds:

none-amd6 9898 root  194u     sock                0,5            61517133 can't identify protocol



 Comments   
Comment by Eric Milkie [ 29/Mar/12 ]

I think my changes to fix socket leaks in 2.1 will fix this. In 2.0 there is little protection against socket leaks if anything goes awry after an incoming connection on the webserver port is handed off. SERVER-4866

Generated at Thu Feb 08 03:08:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.