[SERVER-16991] errno:24 Too many open files on Windows Created: 22/Jan/15  Updated: 23/Jun/17  Resolved: 22/Jan/15

Status: Closed
Project: Core Server
Component/s: Index Maintenance
Affects Version/s: 2.6.7
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Laurent Dupuis Assignee: Mark Benvenuto
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-14572 Increase C runtime stdio file limit Closed
Related
is related to SERVER-17010 Reduce file handle usage in File base... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:
  • Create a very large collection of documents
  • Index them
Participants:

 Description   

While indexing a very large collection on Windows the indexer fails half way with "Too many open files".

Document format:

{ _id: 1234, nodes: [ 1,2,3,4,5, ... ] }

Index on nodes.

2015-01-22T22:53:29.002+0800 [conn73]           Index Build: 109022400/258070991        42%
2015-01-22T22:53:39.187+0800 [conn73]           Index Build: 109149600/258070991        42%
2015-01-22T22:53:42.000+0800 [conn73]           Index Build: 109261800/258070991        42%
2015-01-22T22:53:52.411+0800 [conn73] Assertion: 16818:error opening file "data/_tmp/extsort.509": errno:24 Too many open files
2015-01-22T22:53:56.758+0800 [conn73] mongod.exe  ...\src\mongo\util\stacktrace.cpp(169)                     mongo::printStackTrace+0x43
2015-01-22T22:53:56.758+0800 [conn73] mongod.exe  ...\src\mongo\util\log.cpp(127)                            mongo::logContext+0x9c
2015-01-22T22:53:56.758+0800 [conn73] mongod.exe  ...\src\mongo\util\assert_util.cpp(183)                    mongo::msgasserted+0xfb
2015-01-22T22:53:56.758+0800 [conn73] mongod.exe  ...\src\mongo\util\assert_util.cpp(174)                    mongo::msgasserted+0x13
2015-01-22T22:53:56.758+0800 [conn73] mongod.exe  ...\src\mongo\db\sorter\sorter.cpp(801)                    mongo::SortedFileWriter<mongo::BSONObj,mongo::DiskLoc>::SortedFileWriter<mongo::BSONObj,mongo::DiskLoc>+0x43e
2015-01-22T22:53:56.758+0800 [conn73] mongod.exe  ...\src\mongo\db\sorter\sorter.cpp(465)                    mongo::sorter::NoLimitSorter<mongo::BSONObj,mongo::DiskLoc,mongo::`anonymous namespace'::ComparatorWithInterruptCheck>::spill+0x106
2015-01-22T22:53:56.775+0800 [conn73] mongod.exe  ...\src\mongo\db\sorter\sorter.cpp(410)                    mongo::sorter::NoLimitSorter<mongo::BSONObj,mongo::DiskLoc,mongo::`anonymous namespace'::ComparatorWithInterruptCheck>::add+0x83
2015-01-22T22:53:56.775+0800 [conn73] mongod.exe  ...\src\mongo\db\extsort.h(61)                             mongo::BSONObjExternalSorter::add+0x91
2015-01-22T22:53:56.798+0800 [conn73] mongod.exe  ...\src\mongo\db\sort_phase_one.h(53)                      mongo::SortPhaseOne::addKeys+0x82
2015-01-22T22:53:56.798+0800 [conn73] mongod.exe  ...\src\mongo\db\index\btree_based_access_method.cpp(347)  mongo::BtreeBulk::insert+0x8f
2015-01-22T22:53:56.798+0800 [conn73] mongod.exe  ...\src\mongo\db\catalog\index_create.cpp(75)              mongo::addKeysToIndex+0x9a
2015-01-22T22:53:56.798+0800 [conn73] mongod.exe  ...\src\mongo\db\catalog\index_create.cpp(128)             mongo::addExistingToIndex+0x57c
2015-01-22T22:53:56.798+0800 [conn73] mongod.exe  ...\src\mongo\db\catalog\index_create.cpp(258)             mongo::buildAnIndex+0x62a
2015-01-22T22:53:56.798+0800 [conn73] mongod.exe  ...\src\mongo\db\catalog\index_catalog.cpp(336)            mongo::IndexCatalog::createIndex+0x773
2015-01-22T22:53:56.798+0800 [conn73] mongod.exe  ...\src\mongo\db\commands\create_indexes.cpp(187)          mongo::CmdCreateIndex::run+0x12e9
2015-01-22T22:53:56.799+0800 [conn73] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1385)                      mongo::_execCommand+0x5e
2015-01-22T22:53:56.799+0800 [conn73] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1592)                      mongo::Command::execCommand+0xf09
2015-01-22T22:53:56.799+0800 [conn73] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1724)                      mongo::_runCommands+0x4a7
2015-01-22T22:53:56.799+0800 [conn73] mongod.exe  ...\src\mongo\db\query\new_find.cpp(125)                   mongo::runCommands+0x41
2015-01-22T22:53:56.799+0800 [conn73] mongod.exe  ...\src\mongo\db\query\new_find.cpp(442)                   mongo::newRunQuery+0x49d
2015-01-22T22:53:56.799+0800 [conn73] osm.ways
2015-01-22T22:53:56.870+0800 [conn73] background addExistingToIndex exception error opening file "data/_tmp/extsort.509": errno:24 Too many open files
2015-01-22T22:53:57.931+0800 [conn73] index build failed. spec: { v: 1, key: { nodes: 1 }, name: "nodes_1", ns: "db.refs" } error: 16818 error opening file "data/_tmp/extsort.509": errno:24 Too many open files
2015-01-22T22:53:58.174+0800 [conn73] command db.$cmd command: createIndexes { createIndexes: "refs", indexes: [ { ns: "db.refs", name: "nodes_1", key: { nodes: 1 } } ] } keyUpdates:0 numYields:0 locks(micros) r:15839 w:6430126711 reslen:187 6430194ms
2015-01-22T22:53:58.234+0800 [conn65] killcursors  keyUpdates:0 numYields:0 locks(micros) r:352 3722031ms
2015-01-22T22:53:58.234+0800 [conn65] end connection 127.0.0.1:53138 (7 connections now open)
2015-01-22T22:53:58.356+0800 [conn66] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [127.0.0.1:53139]

Version infos:

/* 0 */
{
    "version" : "2.6.7",
    "gitVersion" : "a7d57ad27c382de82e9cb93bf983a80fd9ac9899",
    "targetMinOS" : "Windows 7/Windows Server 2008 R2",
    "OpenSSLVersion" : "",
    "sysInfo" : "windows sys.getwindowsversion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack 1') BOOST_LIB_VERSION=1_49",
    "loaderFlags" : "/nologo /DEBUG /INCREMENTAL:NO /LARGEADDRESSAWARE",
    "compilerFlags" : "/TP /nologo /EHsc /W3 /wd4355 /wd4800 /wd4267 /wd4244 /wd4290 /we4099 /Z7 /errorReport:none /MT /O2 /Oy-",
    "allocator" : "system",
    "versionArray" : [ 
        2, 
        6, 
        7, 
        0
    ],
    "javascriptEngine" : "V8",
    "bits" : 64,
    "debug" : false,
    "maxBsonObjectSize" : 16777216,
    "ok" : 1
}



 Comments   
Comment by Mark Benvenuto [ 22/Jan/15 ]

Thanks for reporting this issue. We have seen this issue on Windows before, and while there is no workaround at the moment, it has been fixed in the next release. I have asked for a backport of the original fix into the 2.6.x branch.

Comment by Laurent Dupuis [ 22/Jan/15 ]

Noted 263 opened files during shutdown:

2015-01-22T23:16:51.856+0800 [consoleTerminate]                 File Closing Progress: 6/263    2%  (files)
2015-01-22T23:16:53.001+0800 [consoleTerminate]                 File Closing Progress: 11/263   4%  (files)
2015-01-22T23:16:55.846+0800 [consoleTerminate]                 File Closing Progress: 14/263   5%  (files)
2015-01-22T23:16:57.528+0800 [consoleTerminate]                 File Closing Progress: 18/263   6%  (files)
2015-01-22T23:16:59.301+0800 [consoleTerminate]                 File Closing Progress: 54/263   20% (files)

Database is 500GB+

Generated at Thu Feb 08 03:42:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.