Loading...

XML

Word

Printable

JSON

Type: Bug
Resolution: Done
Priority: Major - P3
Fix Version/s: None
Affects Version/s: 2.6.1
Component/s: Storage
Labels:
None

Operating System:
ALL

We are using v2.6.1 and learned that powerOf2Sizes is now the default memory allocation strategy (see also https://jira.mongodb.org/browse/CS-13109 ). However, we prefer not to use it because it consumes always 20% more RAM than the conventional mem alloc strategy.
So, we added the folling line to our mongod.config, which is btw. badly documented, so perhaps we used the wrong syntax?

setParameter=newCollectionsUsePowerOf2Sizes=false

However I assume that's the right syntax since the mongod logs showed that the parameter was picked up successfully:

2014-07-11T17:48:22.712+0200 ***** SERVER RESTARTED *****
2014-07-11T17:48:22.727+0200 [initandlisten] MongoDB starting : pid=12808 port=27017 dbpath=/data/mongod/db 64-bit host=s488
2014-07-11T17:48:22.736+0200 [initandlisten] db version v2.6.1
2014-07-11T17:48:22.736+0200 [initandlisten] git version: 4b95b086d2374bdcfcdf2249272fb552c9c726e8
2014-07-11T17:48:22.736+0200 [initandlisten] build info: Linux build14.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2014-07-11T17:48:22.736+0200 [initandlisten] allocator: tcmalloc
2014-07-11T17:48:22.736+0200 [initandlisten] options: { config: "/etc/mongod_mongod.conf", net: { http: { RESTInterfaceEnabled: true, enabled: true }, port: 27017 }, processManagement: { pidFilePath: "/data/mongod/mongod.pid" }, replication: { replSet: "offerStoreDE5" }, security: { keyFile: "/data/keyfile.txt" }, setParameter: { newCollectionsUsePowerOf2Sizes: "false" }, storage: { dbPath: "/data/mongod/db", directoryPerDB: true, journal: { enabled: false } }, systemLog: { destination: "file", logAppend: true, path: "/data/mongod/log/mongod.log" } }
2014-07-11T17:48:22.763+0200 [FileAllocator] allocating new datafile /data/mongod/db/local/local.ns, filling with zeroes...
2014-07-11T17:48:22.763+0200 [FileAllocator] creating directory /data/mongod/db/local/_tmp
2014-07-11T17:48:22.766+0200 [FileAllocator] done allocating datafile /data/mongod/db/local/local.ns, size: 16MB,  took 0 secs
2014-07-11T17:48:22.768+0200 [FileAllocator] allocating new datafile /data/mongod/db/local/local.0, filling with zeroes...
2014-07-11T17:48:22.768+0200 [FileAllocator] done allocating datafile /data/mongod/db/local/local.0, size: 64MB,  took 0 secs
2014-07-11T17:48:22.770+0200 [initandlisten] build index on: local.startup_log properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "local.startup_log" }
2014-07-11T17:48:22.770+0200 [initandlisten]     added index to empty collection
2014-07-11T17:48:22.772+0200 [initandlisten] waiting for connections on port 27017
2014-07-11T17:48:22.772+0200 [websvr] admin web console waiting for connections on port 28017
2014-07-11T17:48:22.773+0200 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)

Before restarting mongod, we completely emptied the db directory, to be sure that all data will be allocated using the old memory allocation strategy.

Two days later I wondered why RAM consumption was still very high and I discovered, that db.offer.stats().userFlags:1, which means that powerOf2Sizes is still used.
So I executed the following command which I found in the doc to switch back to the old memory allocation strategy:

offerStoreDE5:SECONDARY> db.getSiblingDB('admin').runCommand( { setParameter: 1, newCollectionsUsePowerOf2Sizes: false } )
{ "was" : false, "ok" : 1 }
offerStoreDE5:SECONDARY>
offerStoreDE5:SECONDARY> use offerStore
switched to db offerStore
offerStoreDE5:SECONDARY> db.offer.stats()
{
        "ns" : "offerStore.offer",
        "count" : 55905283,
        "size" : 146846463152,
        "avgObjSize" : 2626,
        "storageSize" : 150970108880,
        "numExtents" : 91,
        "nindexes" : 10,
        "lastExtentSize" : 2146426864,
        "paddingFactor" : 1.0090000000000001,
        "systemFlags" : 1,
        "userFlags" : 1,
        "totalIndexSize" : 17563699552,
        "indexSizes" : {
                "_id_" : 1449596624,
                "bokey_1" : 3597717984,
                "categoryBokey_1" : 2737545552,
                "mappedCatalogCategory_1" : 1780765504,
                "clickCount_1" : 1037501696,
                "missingSince_1" : 1339891056,
                "shopId_1_merchantId_1" : 2271668896,
                "asin_1" : 533647520,
                "importGroup_1_importId_1_missingSince_1" : 834442560,
                "shopId_1__id_1" : 1980922160
        },
        "ok" : 1
}

However, db.offer.stats().userFlags is still 1, meaning that powerOf2Sizes is still used:

offerStoreDE5:SECONDARY> db.offer.stats()
{
        "ns" : "offerStore.offer",
        "count" : 55902199,
        "size" : 146841155952,
        "avgObjSize" : 2626,
        "storageSize" : 150970108880,
        "numExtents" : 91,
        "nindexes" : 10,
        "lastExtentSize" : 2146426864,
        "paddingFactor" : 1,
        "systemFlags" : 1,
        "userFlags" : 1,
        "totalIndexSize" : 17563740432,
        "indexSizes" : {
                "_id_" : 1449596624,
                "bokey_1" : 3597742512,
                "categoryBokey_1" : 2737578256,
                "mappedCatalogCategory_1" : 1780806384,
                "clickCount_1" : 1037518048,
                "missingSince_1" : 1339882880,
                "shopId_1_merchantId_1" : 2271652544,
                "asin_1" : 533663872,
                "importGroup_1_importId_1_missingSince_1" : 834377152,
                "shopId_1__id_1" : 1980922160
        },
        "ok" : 1
}

So I executed the other command I've found in the doc to switch back to the old memory allocation strategy:

root@s488:/home/admin# mongod --setParameter newCollectionsUsePowerOf2Sizes=false
2014-07-13T19:05:17.500+0200 [initandlisten] MongoDB starting : pid=58356 port=27017 dbpath=/data/db 64-bit host=s488
2014-07-13T19:05:17.501+0200 [initandlisten]
2014-07-13T19:05:17.501+0200 [initandlisten] ** WARNING: You are running on a NUMA machine.
2014-07-13T19:05:17.501+0200 [initandlisten] **          We suggest launching mongod like this to avoid performance problems:
2014-07-13T19:05:17.501+0200 [initandlisten] **              numactl --interleave=all mongod [other options]
2014-07-13T19:05:17.517+0200 [initandlisten]
2014-07-13T19:05:17.517+0200 [initandlisten] db version v2.6.1
2014-07-13T19:05:17.517+0200 [initandlisten] git version: 4b95b086d2374bdcfcdf2249272fb552c9c726e8
2014-07-13T19:05:17.517+0200 [initandlisten] build info: Linux build14.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2014-07-13T19:05:17.517+0200 [initandlisten] allocator: tcmalloc
2014-07-13T19:05:17.517+0200 [initandlisten] options: { setParameter: { newCollectionsUsePowerOf2Sizes: "false" } }
2014-07-13T19:05:17.529+0200 [initandlisten] exception in initAndListen: 10296
*********************************************************************
 ERROR: dbpath (/data/db) does not exist.
 Create this directory or give existing directory in --dbpath.
 See http://dochub.mongodb.org/core/startingandstoppingmongo
*********************************************************************
, terminating
2014-07-13T19:05:17.529+0200 [initandlisten] dbexit:
2014-07-13T19:05:17.529+0200 [initandlisten] shutdown: going to close listening sockets...
2014-07-13T19:05:17.529+0200 [initandlisten] shutdown: going to flush diaglog...
2014-07-13T19:05:17.529+0200 [initandlisten] shutdown: going to close sockets...
2014-07-13T19:05:17.529+0200 [initandlisten] shutdown: waiting for fs preallocator...
2014-07-13T19:05:17.529+0200 [initandlisten] shutdown: lock for final commit...
2014-07-13T19:05:17.529+0200 [initandlisten] shutdown: final commit...
2014-07-13T19:05:17.529+0200 [initandlisten] shutdown: closing all files...
2014-07-13T19:05:17.529+0200 [initandlisten] closeAllFiles() finished
2014-07-13T19:05:17.529+0200 [initandlisten] dbexit: really exiting now
root@s488:/home/admin# ps aux|grep mongo
mongodb   12808 17.5 79.0 205348560 104490476 ? Sl   Jul11 518:38 /usr/bin/mongod --config /etc/mongod_mongod.conf
root      58384  0.0  0.0  14760   920 pts/0    S+   19:05   0:00 grep mongo
root@s488:/home/admin#

I'm also wondering why mongo is looking for /data/db because our dbPath is configured /data/mongod/db (see above the log when the server restarted).
Furthermore, the mongod process was and is still running even though mongod pretended "dbexit: really exiting now".

My main question is, how can I revert to the old memory allocation strategy? Neither config entry nor mongo shell command nor mongod command seems to work.

Assignee:: Ramon Fernandez Marina

Reporter:: Kay Agahd

Participants:: Asya Kamsky, Eric Milkie, Kay Agahd, Ramon Fernandez Marina

Votes:: 0 Vote for this issue

Watchers:: 5 Start watching this issue

Created:: Jul 13 2014 05:41:47 PM UTC

Updated:: Dec 10 2014 11:11:47 PM UTC

Resolved:: Jul 14 2014 06:25:43 PM UTC

Details

Description

Attachments

Activity

People

Dates