[WT-2999] Memory leak in join WT_CURSOR with custom extractor Created: 31/Oct/16  Updated: 12/Oct/17  Resolved: 03/Nov/16

Status: Closed
Project: WiredTiger
Component/s: None
Affects Version/s: None
Fix Version/s: WT2.9.0, 3.2.11, 3.4.0-rc3

Type: Bug Priority: Major - P3
Reporter: Anton Dunaev Assignee: Donald Anderson
Resolution: Fixed Votes: 0
Labels: None

Attachments: File wt.c    
Sprint: Storage 2016-11-21

 Description   

Hi!

I use a custom extractor for the join index.

And it seems that there is a memory leak when using custom extractor, which calls result_cursor->set_key()

After test completion there are some memory blocks remain that have been allocated for the next stack:

        __realloc_func           at os_alloc.c:129:11
        __wt_realloc_noclear     at os_alloc.c:170:10
        __wt_buf_grow_worker     at scratch.c:48:4
        __wt_buf_grow            at buf.i:18:6
        __wt_buf_initsize        at buf.i:61:2
        __wt_cursor_set_keyv     at cur_std.c:365:4
        __wt_cursor_set_key      at cur_std.c:183:2
        custom_extract2
        __curjoin_entry_member   at cur_join.c:727:3
        __curjoin_next           at cur_join.c:780:10

The example file is attached.



 Comments   
Comment by Githook User [ 02/Nov/16 ]

Author:

{u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@mongodb.com'}

Message: WT-2999 free memory related to cursor used with custom extractors during joins. (#3117)
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/938020666a9a6b5d2ae6f224af05435948fde5b0

Comment by Githook User [ 03/Nov/16 ]

Author:

{u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@mongodb.com'}

Message: WT-2999 Added contributed test case that demonstrated the leak. (#3122)
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/64e9e4d4191791f864b9fdbc6d876a1ee9ab7d0c

Comment by Michael Cahill [ 03/Nov/16 ]

Thanks Don Anderson, the csuite tests are run in address sanitizer builds, so that is sufficient.

Comment by Githook User [ 03/Nov/16 ]

Author:

{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

Message: WT-2999 Remove C99ism.
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/7663089b8fad9a17eca12bc4d8b8cec59251dbf3

Comment by Githook User [ 04/Nov/16 ]

Author:

{u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@mongodb.com'}

Message: WT-2999 free memory related to cursor used with custom extractors during joins. (#3117)
Branch: mongodb-3.4
https://github.com/wiredtiger/wiredtiger/commit/938020666a9a6b5d2ae6f224af05435948fde5b0

Comment by Githook User [ 04/Nov/16 ]

Author:

{u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@mongodb.com'}

Message: WT-2999 Added contributed test case that demonstrated the leak. (#3122)
Branch: mongodb-3.4
https://github.com/wiredtiger/wiredtiger/commit/64e9e4d4191791f864b9fdbc6d876a1ee9ab7d0c

Comment by Githook User [ 04/Nov/16 ]

Author:

{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

Message: WT-2999 Remove C99ism.
Branch: mongodb-3.4
https://github.com/wiredtiger/wiredtiger/commit/7663089b8fad9a17eca12bc4d8b8cec59251dbf3

Comment by Githook User [ 04/Nov/16 ]

Author:

{u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@mongodb.com'}

Message: WT-2999 free memory related to cursor used with custom extractors during joins. (#3117)
Branch: mongodb-3.2
https://github.com/wiredtiger/wiredtiger/commit/938020666a9a6b5d2ae6f224af05435948fde5b0

Comment by Githook User [ 04/Nov/16 ]

Author:

{u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@mongodb.com'}

Message: WT-2999 Added contributed test case that demonstrated the leak. (#3122)
Branch: mongodb-3.2
https://github.com/wiredtiger/wiredtiger/commit/64e9e4d4191791f864b9fdbc6d876a1ee9ab7d0c

Comment by Githook User [ 04/Nov/16 ]

Author:

{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

Message: WT-2999 Remove C99ism.
Branch: mongodb-3.2
https://github.com/wiredtiger/wiredtiger/commit/7663089b8fad9a17eca12bc4d8b8cec59251dbf3

Comment by Githook User [ 07/Nov/16 ]

Author:

{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

Message: Import wiredtiger: 74430da40c96247c0a0d92f011ccfb2b175f1f32 from branch mongodb-3.4

ref: ef9a7983ea..74430da40c
for: 3.4.0

SERVER-26753 Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2
WT-2415 Add option for join cursors to return false positives
WT-2945 Occasional hang running reconfigure fuzz test
WT-2947 test suite populate functions need refactoring
WT-2953 Test format failure - stuck cache with 1/42MB dirty
WT-2961 Add a version drop-down to the web version of the docs
WT-2964 WTPERF parallel-pop-btree hung in populate phase
WT-2968 backup cursor opens every file
WT-2971 Add documentation for raw compression to the WT docs
WT-2975 Reduce the memory allocated for statistics
WT-2991 Coverity issue 1365198
WT-2998 Add error messages to error returns that might be confusing.
WT-2999 Memory leak in join WT_CURSOR with custom extractor
WT-3001 WT_EXTENSION_API references are named inconsistently.
WT-3003 log generation should not auto generate the record and op types
WT-3005 Add top-level .gitignore file.
WT-3007 Remove duplicated word in comment
WT-3008 Run wtperf stress testing against all maintained branches
WT-3009 Test format hung/aborted with small cache (sub 10mB)
Branch: master
https://github.com/mongodb/mongo/commit/0f0db45770f45eab89bd7c2530a259f5ed5b6334

Comment by Githook User [ 07/Nov/16 ]

Author:

{u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'}

Message: Import wiredtiger: 2e2d5fe23badac038944bdc6639eb62d77136f1a from branch mongodb-3.2

ref: b11ed312ce..2e2d5fe23b
for: 3.2.11

SERVER-26753 Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2
WT-2415 Add option for join cursors to return false positives from a Bloom filter
WT-2945 Occasional hang running reconfigure fuzz test
WT-2947 test suite populate functions need refactoring
WT-2953 Test format failure - stuck cache with 1/42MB dirty
WT-2961 Add a version drop-down to the web version of the docs
WT-2964 WTPERF parallel-pop-btree hung in populate phase
WT-2968 backup cursor opens every file
WT-2971 Add documentation for raw-compression to the tune_compression page in WT docs
WT-2975 Reduce the memory allocated for statistics
WT-2991 Coverity issue 1365198
WT-2998 Add error messages to error returns that might be confusing.
WT-2999 Memory leak in join WT_CURSOR with custom extractor
WT-3001 WT_EXTENSION_API references are named inconsistently.
WT-3003 log generation should not auto generate the record and op types
WT-3005 Add top-level .gitignore file.
WT-3007 Remove duplicated word in comment
WT-3008 Run wtperf stress testing against all maintained branches
WT-3009 Test format hung/aborted with small cache (sub 10mB)
Branch: v3.2
https://github.com/mongodb/mongo/commit/7c626977cf052314d09648d1061e07c44e964264

Generated at Sat Oct 20 10:40:58 UTC 2018 using Jira 7.12.1#712002-sha1:609a50578ba6bc73dbf8b05dddd7c04a04b6807c.