Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2999

Memory leak in join WT_CURSOR with custom extractor

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.9.0, 3.2.11, 3.4.0-rc3
    • Labels:
      None
    • # Replies:
      12
    • Last comment by Customer:
      true
    • Sprint:
      Storage 2016-11-21

      Description

      Hi!

      I use a custom extractor for the join index.

      And it seems that there is a memory leak when using custom extractor, which calls result_cursor->set_key()

      After test completion there are some memory blocks remain that have been allocated for the next stack:

              __realloc_func           at os_alloc.c:129:11
              __wt_realloc_noclear     at os_alloc.c:170:10
              __wt_buf_grow_worker     at scratch.c:48:4
              __wt_buf_grow            at buf.i:18:6
              __wt_buf_initsize        at buf.i:61:2
              __wt_cursor_set_keyv     at cur_std.c:365:4
              __wt_cursor_set_key      at cur_std.c:183:2
              custom_extract2
              __curjoin_entry_member   at cur_join.c:727:3
              __curjoin_next           at cur_join.c:780:10
      
      

      The example file is attached.

      1. wt.c
        4 kB
        Anton Dunaev

        Activity

        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@mongodb.com'}

        Message: WT-2999 free memory related to cursor used with custom extractors during joins. (#3117)
        Branch: mongodb-3.2
        https://github.com/wiredtiger/wiredtiger/commit/938020666a9a6b5d2ae6f224af05435948fde5b0

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@mongodb.com'} Message: WT-2999 free memory related to cursor used with custom extractors during joins. (#3117) Branch: mongodb-3.2 https://github.com/wiredtiger/wiredtiger/commit/938020666a9a6b5d2ae6f224af05435948fde5b0
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@mongodb.com'}

        Message: WT-2999 Added contributed test case that demonstrated the leak. (#3122)
        Branch: mongodb-3.2
        https://github.com/wiredtiger/wiredtiger/commit/64e9e4d4191791f864b9fdbc6d876a1ee9ab7d0c

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'ddanderson', u'name': u'Don Anderson', u'email': u'dda@mongodb.com'} Message: WT-2999 Added contributed test case that demonstrated the leak. (#3122) Branch: mongodb-3.2 https://github.com/wiredtiger/wiredtiger/commit/64e9e4d4191791f864b9fdbc6d876a1ee9ab7d0c
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

        Message: WT-2999 Remove C99ism.
        Branch: mongodb-3.2
        https://github.com/wiredtiger/wiredtiger/commit/7663089b8fad9a17eca12bc4d8b8cec59251dbf3

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: WT-2999 Remove C99ism. Branch: mongodb-3.2 https://github.com/wiredtiger/wiredtiger/commit/7663089b8fad9a17eca12bc4d8b8cec59251dbf3
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

        Message: Import wiredtiger: 74430da40c96247c0a0d92f011ccfb2b175f1f32 from branch mongodb-3.4

        ref: ef9a7983ea..74430da40c
        for: 3.4.0

        SERVER-26753 Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2
        WT-2415 Add option for join cursors to return false positives
        WT-2945 Occasional hang running reconfigure fuzz test
        WT-2947 test suite populate functions need refactoring
        WT-2953 Test format failure - stuck cache with 1/42MB dirty
        WT-2961 Add a version drop-down to the web version of the docs
        WT-2964 WTPERF parallel-pop-btree hung in populate phase
        WT-2968 backup cursor opens every file
        WT-2971 Add documentation for raw compression to the WT docs
        WT-2975 Reduce the memory allocated for statistics
        WT-2991 Coverity issue 1365198
        WT-2998 Add error messages to error returns that might be confusing.
        WT-2999 Memory leak in join WT_CURSOR with custom extractor
        WT-3001 WT_EXTENSION_API references are named inconsistently.
        WT-3003 log generation should not auto generate the record and op types
        WT-3005 Add top-level .gitignore file.
        WT-3007 Remove duplicated word in comment
        WT-3008 Run wtperf stress testing against all maintained branches
        WT-3009 Test format hung/aborted with small cache (sub 10mB)
        Branch: master
        https://github.com/mongodb/mongo/commit/0f0db45770f45eab89bd7c2530a259f5ed5b6334

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Import wiredtiger: 74430da40c96247c0a0d92f011ccfb2b175f1f32 from branch mongodb-3.4 ref: ef9a7983ea..74430da40c for: 3.4.0 SERVER-26753 Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2 WT-2415 Add option for join cursors to return false positives WT-2945 Occasional hang running reconfigure fuzz test WT-2947 test suite populate functions need refactoring WT-2953 Test format failure - stuck cache with 1/42MB dirty WT-2961 Add a version drop-down to the web version of the docs WT-2964 WTPERF parallel-pop-btree hung in populate phase WT-2968 backup cursor opens every file WT-2971 Add documentation for raw compression to the WT docs WT-2975 Reduce the memory allocated for statistics WT-2991 Coverity issue 1365198 WT-2998 Add error messages to error returns that might be confusing. WT-2999 Memory leak in join WT_CURSOR with custom extractor WT-3001 WT_EXTENSION_API references are named inconsistently. WT-3003 log generation should not auto generate the record and op types WT-3005 Add top-level .gitignore file. WT-3007 Remove duplicated word in comment WT-3008 Run wtperf stress testing against all maintained branches WT-3009 Test format hung/aborted with small cache (sub 10mB) Branch: master https://github.com/mongodb/mongo/commit/0f0db45770f45eab89bd7c2530a259f5ed5b6334
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'}

        Message: Import wiredtiger: 2e2d5fe23badac038944bdc6639eb62d77136f1a from branch mongodb-3.2

        ref: b11ed312ce..2e2d5fe23b
        for: 3.2.11

        SERVER-26753 Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2
        WT-2415 Add option for join cursors to return false positives from a Bloom filter
        WT-2945 Occasional hang running reconfigure fuzz test
        WT-2947 test suite populate functions need refactoring
        WT-2953 Test format failure - stuck cache with 1/42MB dirty
        WT-2961 Add a version drop-down to the web version of the docs
        WT-2964 WTPERF parallel-pop-btree hung in populate phase
        WT-2968 backup cursor opens every file
        WT-2971 Add documentation for raw-compression to the tune_compression page in WT docs
        WT-2975 Reduce the memory allocated for statistics
        WT-2991 Coverity issue 1365198
        WT-2998 Add error messages to error returns that might be confusing.
        WT-2999 Memory leak in join WT_CURSOR with custom extractor
        WT-3001 WT_EXTENSION_API references are named inconsistently.
        WT-3003 log generation should not auto generate the record and op types
        WT-3005 Add top-level .gitignore file.
        WT-3007 Remove duplicated word in comment
        WT-3008 Run wtperf stress testing against all maintained branches
        WT-3009 Test format hung/aborted with small cache (sub 10mB)
        Branch: v3.2
        https://github.com/mongodb/mongo/commit/7c626977cf052314d09648d1061e07c44e964264

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'} Message: Import wiredtiger: 2e2d5fe23badac038944bdc6639eb62d77136f1a from branch mongodb-3.2 ref: b11ed312ce..2e2d5fe23b for: 3.2.11 SERVER-26753 Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2 WT-2415 Add option for join cursors to return false positives from a Bloom filter WT-2945 Occasional hang running reconfigure fuzz test WT-2947 test suite populate functions need refactoring WT-2953 Test format failure - stuck cache with 1/42MB dirty WT-2961 Add a version drop-down to the web version of the docs WT-2964 WTPERF parallel-pop-btree hung in populate phase WT-2968 backup cursor opens every file WT-2971 Add documentation for raw-compression to the tune_compression page in WT docs WT-2975 Reduce the memory allocated for statistics WT-2991 Coverity issue 1365198 WT-2998 Add error messages to error returns that might be confusing. WT-2999 Memory leak in join WT_CURSOR with custom extractor WT-3001 WT_EXTENSION_API references are named inconsistently. WT-3003 log generation should not auto generate the record and op types WT-3005 Add top-level .gitignore file. WT-3007 Remove duplicated word in comment WT-3008 Run wtperf stress testing against all maintained branches WT-3009 Test format hung/aborted with small cache (sub 10mB) Branch: v3.2 https://github.com/mongodb/mongo/commit/7c626977cf052314d09648d1061e07c44e964264

          People

          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:
              Days since reply:
              20 weeks, 1 day ago
              Date of 1st Reply:

                Agile