[WT-2531] in-memory tables are allocating unnecessary memory - MongoDB

Details

Type: Bug
Status: Resolved
Priority: Major - P3
Resolution: Fixed
Affects Version/s: None
Fix Version/s: WT2.9.0
Labels:
None

# Replies:
12
Last comment by Customer:
true

Description

In-memory tables are no longer "writing" their descriptor blocks, but they are truncating the "file" to the initial size, which causes an empty buffer to be allocated.

Issue Links

is depended on by

SERVER-23517 WiredTiger changes for MongoDB 3.3.5

Closed

SERVER-23682 WiredTiger changes for MongoDB 3.2.6

Closed

links to

https://github.com/wiredtiger/wiredtiger/pull/2629

Activity

Comments

Ascending order - Click to sort in descending order

Hide

Permalink

Sue LoVerso added a comment - Apr 04 2016 03:22:29 PM +00:00

At a higher-level, maybe the secondaries shouldn't be looking at on-disk sizes to decide if a copy needs to be made? Or maybe they should copy any object that exists, regardless of the size?

(I think it is the primary looking at the on-disk size to determine what to send to a secondary on a resync).
I believe the code in mongo/db/repl/master_slave.cpp:423:forceResync() is the code in question. The listDatabases() command in mongo/db/commands/list_databases.cpp adds an empty boolean of true if the size on disk returned is 0. Then the forceResync code calls resyncDrop if it is empty. There are no comments nearby for that decision.

Show

Sue LoVerso added a comment - Apr 04 2016 03:22:29 PM +00:00 At a higher-level, maybe the secondaries shouldn't be looking at on-disk sizes to decide if a copy needs to be made? Or maybe they should copy any object that exists, regardless of the size? (I think it is the primary looking at the on-disk size to determine what to send to a secondary on a resync). I believe the code in mongo/db/repl/master_slave.cpp:423:forceResync() is the code in question. The listDatabases() command in mongo/db/commands/list_databases.cpp adds an empty boolean of true if the size on disk returned is 0. Then the forceResync code calls resyncDrop if it is empty. There are no comments nearby for that decision.

6 older comments

Hide

Permalink

Githook User added a comment - Apr 05 2016 04:47:49 AM +00:00

Author:

{u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexg@wiredtiger.com'}

Message: Import wiredtiger-wiredtiger-2.8.0-134-g5047aab.tar.gz from wiredtiger branch mongodb-3.4

ref: 9cf8eb2..5047aab

~~SERVER-23504~~ Coverity analysis defect 98177: Resource leak
~~WT-2330~~ in-memory configurations should not create on-disk collection files
~~WT-2513~~ conversion from 'int64_t' to 'uint32_t'
~~WT-2522~~ Incorrect format code in message
~~WT-2525~~ in-memory configurations: miscellaneous cleanups
~~WT-2527~~ OS X compile error, missing POSIX_FADV_WILLNEED #define
~~WT-2528~~ style error in WiredTiger build
~~WT-2529~~ The readonly test case is crashing with a stack overflow
~~WT-2531~~ in-memory tables are allocating unnecessary memory
~~WT-2532~~ WT_STREAM_APPEND and WT_STREAM_LINE_BUFFER flag overlap
~~WT-2533~~ Ensure that in-memory tables don't report a zero size
~~SERVER-23517~~ WiredTiger changes for MongoDB 3.3.5
Branch: master
https://github.com/mongodb/mongo/commit/de6f136d83b20f8a58ba6fe4ba02be229b6c9159

Show

Githook User added a comment - Apr 05 2016 04:47:49 AM +00:00 Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexg@wiredtiger.com'} Message: Import wiredtiger-wiredtiger-2.8.0-134-g5047aab.tar.gz from wiredtiger branch mongodb-3.4 ref: 9cf8eb2..5047aab SERVER-23504 Coverity analysis defect 98177: Resource leak WT-2330 in-memory configurations should not create on-disk collection files WT-2513 conversion from 'int64_t' to 'uint32_t' WT-2522 Incorrect format code in message WT-2525 in-memory configurations: miscellaneous cleanups WT-2527 OS X compile error, missing POSIX_FADV_WILLNEED #define WT-2528 style error in WiredTiger build WT-2529 The readonly test case is crashing with a stack overflow WT-2531 in-memory tables are allocating unnecessary memory WT-2532 WT_STREAM_APPEND and WT_STREAM_LINE_BUFFER flag overlap WT-2533 Ensure that in-memory tables don't report a zero size SERVER-23517 WiredTiger changes for MongoDB 3.3.5 Branch: master https://github.com/mongodb/mongo/commit/de6f136d83b20f8a58ba6fe4ba02be229b6c9159

Hide

Permalink

Githook User added a comment - Apr 07 2016 07:31:48 AM +00:00

Author:

{u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'}

Message: ~~WT-2531~~: in-memory tables are wasting space in truncation

The in-memory truncate call wasn't extending the file (by setting the
size of the WT_ITEM after truncation extended it).
Branch: mongodb-3.2
https://github.com/wiredtiger/wiredtiger/commit/659c977eb08d9c08ef903f5d876ac242daf424cc

Show

Githook User added a comment - Apr 07 2016 07:31:48 AM +00:00 Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'} Message: WT-2531 : in-memory tables are wasting space in truncation The in-memory truncate call wasn't extending the file (by setting the size of the WT_ITEM after truncation extended it). Branch: mongodb-3.2 https://github.com/wiredtiger/wiredtiger/commit/659c977eb08d9c08ef903f5d876ac242daf424cc

Hide

Permalink

Githook User added a comment - Apr 07 2016 07:31:49 AM +00:00

Author:

{u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'}

Message: ~~WT-2531~~: in-memory tables are wasting space in truncation

Don't truncate the file when running in-memory, it will allocate relatively
large chunks of buffer memory in the underlying OS layer that we'll never use.
Branch: mongodb-3.2
https://github.com/wiredtiger/wiredtiger/commit/af4e3f14bd51f3c4149f64bcf4e2e249c61c5136

Show

Githook User added a comment - Apr 07 2016 07:31:49 AM +00:00 Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'} Message: WT-2531 : in-memory tables are wasting space in truncation Don't truncate the file when running in-memory, it will allocate relatively large chunks of buffer memory in the underlying OS layer that we'll never use. Branch: mongodb-3.2 https://github.com/wiredtiger/wiredtiger/commit/af4e3f14bd51f3c4149f64bcf4e2e249c61c5136

Hide

Permalink

Githook User added a comment - Apr 07 2016 07:31:57 AM +00:00

Author:

{u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

Message: Merge pull request #2629 from wiredtiger/wt-2531

~~WT-2531~~: in-memory tables are wasting space in truncation
Branch: mongodb-3.2
https://github.com/wiredtiger/wiredtiger/commit/7a5fce2b618d51e0bb42a12b0a2e44a3632d3d10

Show

Githook User added a comment - Apr 07 2016 07:31:57 AM +00:00 Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Merge pull request #2629 from wiredtiger/wt-2531 WT-2531 : in-memory tables are wasting space in truncation Branch: mongodb-3.2 https://github.com/wiredtiger/wiredtiger/commit/7a5fce2b618d51e0bb42a12b0a2e44a3632d3d10

Hide

Permalink

Githook User added a comment - Apr 14 2016 02:29:51 PM +00:00

Author:

{u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'}

Message: Import wiredtiger-wiredtiger-2.8.0-201-g7ea2631.tar.gz from wiredtiger branch mongodb-3.2

ref: 43e885a..7ea2631

~~SERVER-23504~~ Coverity analysis defect 98177: Resource leak
~~SERVER-23526~~ Replication relies on storage engines reporting a non-zero size for correctness
~~SERVER-23588~~ mongod with WiredTiger won't start on Windows when built with --dbg=on --opt=off
~~SERVER-23682~~ WiredTiger changes for MongoDB 3.2.6
~~WT-2330~~ in-memory configurations should not create on-disk collection files
~~WT-2507~~ Add upgrading documentation in preparation for 2.8 release.
~~WT-2512~~ wtperf: MSVC complains about float conversion in throttle code
~~WT-2513~~ conversion from 'int64_t' to 'uint32_t'
~~WT-2517~~ wtperf uses setvbuf in a way that isn't supported on Windows
~~WT-2522~~ Incorrect format code in message
~~WT-2525~~ in-memory configurations: miscellaneous cleanups
~~WT-2527~~ OS X compile error, missing POSIX_FADV_WILLNEED #define
~~WT-2528~~ style error in WiredTiger build
~~WT-2529~~ The readonly test case is crashing with a stack overflow
~~WT-2531~~ in-memory tables are allocating unnecessary memory
~~WT-2532~~ WT_STREAM_APPEND and WT_STREAM_LINE_BUFFER flag overlap
~~WT-2533~~ Ensure that in-memory tables don't report a zero size
~~WT-2534~~ Invalid transaction snapshots on PowerPC
Branch: v3.2
https://github.com/mongodb/mongo/commit/7ee4e4e493c3785fea489ee3508ca18526709c16

Show

Githook User added a comment - Apr 14 2016 02:29:51 PM +00:00 Author: {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'} Message: Import wiredtiger-wiredtiger-2.8.0-201-g7ea2631.tar.gz from wiredtiger branch mongodb-3.2 ref: 43e885a..7ea2631 SERVER-23504 Coverity analysis defect 98177: Resource leak SERVER-23526 Replication relies on storage engines reporting a non-zero size for correctness SERVER-23588 mongod with WiredTiger won't start on Windows when built with --dbg=on --opt=off SERVER-23682 WiredTiger changes for MongoDB 3.2.6 WT-2330 in-memory configurations should not create on-disk collection files WT-2507 Add upgrading documentation in preparation for 2.8 release. WT-2512 wtperf: MSVC complains about float conversion in throttle code WT-2513 conversion from 'int64_t' to 'uint32_t' WT-2517 wtperf uses setvbuf in a way that isn't supported on Windows WT-2522 Incorrect format code in message WT-2525 in-memory configurations: miscellaneous cleanups WT-2527 OS X compile error, missing POSIX_FADV_WILLNEED #define WT-2528 style error in WiredTiger build WT-2529 The readonly test case is crashing with a stack overflow WT-2531 in-memory tables are allocating unnecessary memory WT-2532 WT_STREAM_APPEND and WT_STREAM_LINE_BUFFER flag overlap WT-2533 Ensure that in-memory tables don't report a zero size WT-2534 Invalid transaction snapshots on PowerPC Branch: v3.2 https://github.com/mongodb/mongo/commit/7ee4e4e493c3785fea489ee3508ca18526709c16

People

Assignee:

Keith Bostic

Reporter:

Keith Bostic

Participants:

Githook User, Keith Bostic, Sue LoVerso

Last commenter:

Michael Cahill

Votes:

0 Vote for this issue

Watchers:

2 Start watching this issue

Dates

Created:

Apr 04 2016 02:02:13 PM +00:00

Updated:

Apr 15 2016 04:07:38 AM +00:00

Resolved:

Apr 05 2016 01:02:03 AM +00:00

Days since reply:

1 year, 10 weeks, 5 days ago

Date of 1st Reply:

04/Apr/16 3:22 PM

Agile

View on Board