Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-3034

Add support for single-writer named snapshots.

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2.12, WT2.9.1, 3.5.1, 3.4.2
    • Labels:
      None
    • # Replies:
      6
    • Last comment by Customer:
      true
    • Sprint:
      Storage 2016-12-12

      Description

      Message #1:

      Our application is designed to use many threads to coordinate the work done for a transaction, and we want to use snapshot isolation for that transaction. We are able to do this for read-only operations by having multiple Sessions which share a named snapshot.

      Further, we designate one of those Session objects as the "writer". Any mutations contend for that writer. This works fine. However, once we have applied a mutation to the "writer" Session, that mutation is not visible in other Sessions objects for the same named snapshot. It would be nice to have such mutations become visible among the Session objects that we use to coordinate the work for a transaction (e.g., same named snapshot), but to not become visible outside of that scope until a commit.

      I am trying to use multiple threads doing work in parallel as part of the same transaction. This works fine if there are no mutations. All I need to do is have each thread use a Session object which is sharing the same named snapshot. They all see the same data.

      Ideally I would like to find a way to have those threads see the updates as well. That is, I would like them to be part of the same transaction.

      I was hoping that WT named snapshots would have these semantics. So I could designate one Session as the "writer" and the other Sessions sharing the same named snapshot would see the same data.

      But this does not appear to be the case. The named snapshot let's the different Session objects see the same starting state, but updates do not become visible. Thus if there is a mutation applied, I need to single thread all subsequent operations through the same Session in order to see those updates.

      Basically, WT Session = Transaction, which implies that a multiple threads can not coordinate to service the transaction since a Session can only be used by a single thread at a time.


      Message #2:

      WiredTiger's named snapshots were designed to allow multiple transactions to read from the same snapshot. The requirement in MongoDB is that many of those transactions reading from the same snapshot may go on to do (independent) updates. For that reason, they don't see each other's writes.

      We could support what you're asking for relatively easily, if the "writer" can create the named snapshot and "readers" are forced to be read-only. I'd suggest adding a new configuration to WT_SESSION::snapshot to support this mode (called something like "include_updates"). Transactions started from such a snapshot would see the updates of the writer but would be forced to be read-only. There are data structures inside the session that cache the transaction's write set and log records that are not thread safe, so it would be a very different change to support multiple writer sessions for a single transaction.

        Activity

        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

        Message: WT-3034 Add support for named snapshots including updates. (#3161)

        This supports a model where one session performs updates in a transaction after creating a named snapshot and other sessions can use that snapshot and read the updates. In other words, they see exactly what the updating session sees.
        Branch: develop
        https://github.com/wiredtiger/wiredtiger/commit/2f18a859cea47f0352cea9fbd64b396e52095ed8

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: WT-3034 Add support for named snapshots including updates. (#3161) This supports a model where one session performs updates in a transaction after creating a named snapshot and other sessions can use that snapshot and read the updates. In other words, they see exactly what the updating session sees. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/2f18a859cea47f0352cea9fbd64b396e52095ed8
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

        Message: Import wiredtiger: 1b6c815a3fd34f14c20d5cd627155799d1de535c from branch mongodb-3.6

        ref: ca6eee06ff..1b6c815a3f
        for: 3.5.1

        WT-2336 Add a test validating schema operations via file system call monitoring
        WT-2670 Add option to configure read-ahead per table and change default behavior
        WT-2960 Inserting multi-megabyte values can cause pathological lookaside usage
        WT-2969 Fix a bug that could cause snapshot corruption during compaction
        WT-3014 Add GCC/clang support for ELF symbol visibility.
        WT-3021 Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode
        WT-3025 fix error path in log_force_sync
        WT-3028 Workloads with all dirty pages could trigger diagnostic stuck check
        WT-3030 Test failure indicating invalid key order during traversal
        WT-3034 Add support for single-writer named snapshots.
        WT-3037 Fix some outdated comments in logging
        WT-3048 WiredTiger maximum size warning uses the wrong format.
        WT-3051 Remove external __wt_hex symbol.
        WT-3052 Improve search if an index hint is wrong
        WT-3053 Review Python and Java calls to internal WiredTiger functions
        WT-3054 Java PackTest, PackTest03 do not compile
        WT-3055 Java AsyncTest faults
        WT-3057 WiredTiger hazard pointers should use the WT_REF, not the WT_PAGE.
        WT-3064 minor tree cleanups: .gitignore, NEWS misspelling
        Branch: master
        https://github.com/mongodb/mongo/commit/21a6f07d859c132154166bd3d83bbed238d5d719

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Import wiredtiger: 1b6c815a3fd34f14c20d5cd627155799d1de535c from branch mongodb-3.6 ref: ca6eee06ff..1b6c815a3f for: 3.5.1 WT-2336 Add a test validating schema operations via file system call monitoring WT-2670 Add option to configure read-ahead per table and change default behavior WT-2960 Inserting multi-megabyte values can cause pathological lookaside usage WT-2969 Fix a bug that could cause snapshot corruption during compaction WT-3014 Add GCC/clang support for ELF symbol visibility. WT-3021 Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode WT-3025 fix error path in log_force_sync WT-3028 Workloads with all dirty pages could trigger diagnostic stuck check WT-3030 Test failure indicating invalid key order during traversal WT-3034 Add support for single-writer named snapshots. WT-3037 Fix some outdated comments in logging WT-3048 WiredTiger maximum size warning uses the wrong format. WT-3051 Remove external __wt_hex symbol. WT-3052 Improve search if an index hint is wrong WT-3053 Review Python and Java calls to internal WiredTiger functions WT-3054 Java PackTest, PackTest03 do not compile WT-3055 Java AsyncTest faults WT-3057 WiredTiger hazard pointers should use the WT_REF, not the WT_PAGE. WT-3064 minor tree cleanups: .gitignore, NEWS misspelling Branch: master https://github.com/mongodb/mongo/commit/21a6f07d859c132154166bd3d83bbed238d5d719
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

        Message: WT-3034 Add support for named snapshots including updates. (#3161)

        This supports a model where one session performs updates in a transaction after creating a named snapshot and other sessions can use that snapshot and read the updates. In other words, they see exactly what the updating session sees.
        Branch: mongodb-3.4
        https://github.com/wiredtiger/wiredtiger/commit/2f18a859cea47f0352cea9fbd64b396e52095ed8

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: WT-3034 Add support for named snapshots including updates. (#3161) This supports a model where one session performs updates in a transaction after creating a named snapshot and other sessions can use that snapshot and read the updates. In other words, they see exactly what the updating session sees. Branch: mongodb-3.4 https://github.com/wiredtiger/wiredtiger/commit/2f18a859cea47f0352cea9fbd64b396e52095ed8
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

        Message: WT-3034 Add support for named snapshots including updates. (#3161)

        This supports a model where one session performs updates in a transaction after creating a named snapshot and other sessions can use that snapshot and read the updates. In other words, they see exactly what the updating session sees.
        Branch: mongodb-3.2
        https://github.com/wiredtiger/wiredtiger/commit/2f18a859cea47f0352cea9fbd64b396e52095ed8

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: WT-3034 Add support for named snapshots including updates. (#3161) This supports a model where one session performs updates in a transaction after creating a named snapshot and other sessions can use that snapshot and read the updates. In other words, they see exactly what the updating session sees. Branch: mongodb-3.2 https://github.com/wiredtiger/wiredtiger/commit/2f18a859cea47f0352cea9fbd64b396e52095ed8
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'}

        Message: Import wiredtiger: d48181f6f4db08761ed7b80b0332908b272ad0d0 from branch mongodb-3.2

        ref: 040e3d6f76..d48181f6f4
        for: 3.2.12

        SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers
        WT-2336 Add a test validating schema operations via file system call monitoring
        WT-2402 Misaligned structure accesses lead to undefined behavior
        WT-2670 Inefficient I/O when read full DB (poor readahead)
        WT-283 Add a way to change persistent object settings
        WT-2960 Inserting multi-megabyte values can cause pathological lookaside usage
        WT-2969 Possible snapshot corruption during compaction
        WT-3014 Add GCC/clang support for ELF symbol visibility.
        WT-3021 Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode
        WT-3025 fix error path in log_force_sync
        WT-3028 Workloads with all dirty pages could trigger diagnostic stuck check
        WT-3030 Test failure indicating invalid key order during traversal
        WT-3034 Add support for single-writer named snapshots.
        WT-3037 Fix some outdated comments in logging
        WT-3048 WiredTiger maximum size warning uses the wrong format.
        WT-3051 Remove external __wt_hex symbol.
        WT-3052 Improve search if an index hint is wrong
        WT-3053 Review Python and Java calls to internal WiredTiger functions
        WT-3054 Java PackTest, PackTest03 do not compile
        WT-3055 Java AsyncTest faults
        WT-3056 For cursors with projections, keys should be allowed
        WT-3057 WiredTiger hazard pointers should use the WT_REF, not the WT_PAGE.
        WT-3061 syscall test runs with checkpoint_sync=false and doesn't acknowledge pwrite64
        WT-3064 minor tree cleanups: .gitignore, NEWS misspelling
        WT-3066 lint
        WT-3068 Copy wtperf artifacts when running Jenkins tests
        WT-3069 Fix build failures in LevelDB APIs
        WT-3070 Fix search_near() for index cursor
        WT-3071 Java: fix build with -Werror=sign-conversion
        WT-3075 Document and enforce that WiredTiger now depends on Python 2.7
        WT-3078 Fix a hang in the reconfiguration test.
        WT-3084 Fix Coverity resource leak complaint.
        Branch: v3.2
        https://github.com/mongodb/mongo/commit/52b68fa86ea43e909ad42c901d0579bced6b205f

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'} Message: Import wiredtiger: d48181f6f4db08761ed7b80b0332908b272ad0d0 from branch mongodb-3.2 ref: 040e3d6f76..d48181f6f4 for: 3.2.12 SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers WT-2336 Add a test validating schema operations via file system call monitoring WT-2402 Misaligned structure accesses lead to undefined behavior WT-2670 Inefficient I/O when read full DB (poor readahead) WT-283 Add a way to change persistent object settings WT-2960 Inserting multi-megabyte values can cause pathological lookaside usage WT-2969 Possible snapshot corruption during compaction WT-3014 Add GCC/clang support for ELF symbol visibility. WT-3021 Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode WT-3025 fix error path in log_force_sync WT-3028 Workloads with all dirty pages could trigger diagnostic stuck check WT-3030 Test failure indicating invalid key order during traversal WT-3034 Add support for single-writer named snapshots. WT-3037 Fix some outdated comments in logging WT-3048 WiredTiger maximum size warning uses the wrong format. WT-3051 Remove external __wt_hex symbol. WT-3052 Improve search if an index hint is wrong WT-3053 Review Python and Java calls to internal WiredTiger functions WT-3054 Java PackTest, PackTest03 do not compile WT-3055 Java AsyncTest faults WT-3056 For cursors with projections, keys should be allowed WT-3057 WiredTiger hazard pointers should use the WT_REF, not the WT_PAGE. WT-3061 syscall test runs with checkpoint_sync=false and doesn't acknowledge pwrite64 WT-3064 minor tree cleanups: .gitignore, NEWS misspelling WT-3066 lint WT-3068 Copy wtperf artifacts when running Jenkins tests WT-3069 Fix build failures in LevelDB APIs WT-3070 Fix search_near() for index cursor WT-3071 Java: fix build with -Werror=sign-conversion WT-3075 Document and enforce that WiredTiger now depends on Python 2.7 WT-3078 Fix a hang in the reconfiguration test. WT-3084 Fix Coverity resource leak complaint. Branch: v3.2 https://github.com/mongodb/mongo/commit/52b68fa86ea43e909ad42c901d0579bced6b205f
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'}

        Message: Import wiredtiger: 8d2324943364286056ae399043f70b8a937de312 from branch mongodb-3.4

        ref: ca6eee06ff..8d23249433
        for: 3.4.2

        SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers
        WT-2336 Add a test validating schema operations via file system call monitoring
        WT-2402 Misaligned structure accesses lead to undefined behavior
        WT-2670 Inefficient I/O when read full DB (poor readahead)
        WT-283 Add a way to change persistent object settings
        WT-2960 Inserting multi-megabyte values can cause pathological lookaside usage
        WT-2969 Possible snapshot corruption during compaction
        WT-3014 Add GCC/clang support for ELF symbol visibility.
        WT-3021 Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode
        WT-3025 fix error path in log_force_sync
        WT-3028 Workloads with all dirty pages could trigger diagnostic stuck check
        WT-3030 Test failure indicating invalid key order during traversal
        WT-3034 Add support for single-writer named snapshots.
        WT-3037 Fix some outdated comments in logging
        WT-3048 WiredTiger maximum size warning uses the wrong format.
        WT-3051 Remove external __wt_hex symbol.
        WT-3052 Improve search if an index hint is wrong
        WT-3053 Review Python and Java calls to internal WiredTiger functions
        WT-3054 Java PackTest, PackTest03 do not compile
        WT-3055 Java AsyncTest faults
        WT-3056 For cursors with projections, keys should be allowed
        WT-3057 WiredTiger hazard pointers should use the WT_REF, not the WT_PAGE.
        WT-3061 syscall test runs with checkpoint_sync=false and doesn't acknowledge pwrite64
        WT-3064 minor tree cleanups: .gitignore, NEWS misspelling
        WT-3066 lint
        WT-3068 Copy wtperf artifacts when running Jenkins tests
        WT-3069 Fix build failures in LevelDB APIs
        WT-3070 Fix search_near() for index cursor
        WT-3071 Java: fix build with -Werror=sign-conversion
        WT-3075 Document and enforce that WiredTiger now depends on Python 2.7
        WT-3078 Fix a hang in the reconfiguration test.
        WT-3084 Fix Coverity resource leak complaint.
        Branch: v3.4
        https://github.com/mongodb/mongo/commit/d2c64ac8c526b70eadeb859ec41370a5f03a64aa

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'} Message: Import wiredtiger: 8d2324943364286056ae399043f70b8a937de312 from branch mongodb-3.4 ref: ca6eee06ff..8d23249433 for: 3.4.2 SERVER-26545 Remove fixed-size limitation on WiredTiger hazard pointers WT-2336 Add a test validating schema operations via file system call monitoring WT-2402 Misaligned structure accesses lead to undefined behavior WT-2670 Inefficient I/O when read full DB (poor readahead) WT-283 Add a way to change persistent object settings WT-2960 Inserting multi-megabyte values can cause pathological lookaside usage WT-2969 Possible snapshot corruption during compaction WT-3014 Add GCC/clang support for ELF symbol visibility. WT-3021 Fixes needed for Java log cursor example, Java raw mode cursors, log cursors in raw mode WT-3025 fix error path in log_force_sync WT-3028 Workloads with all dirty pages could trigger diagnostic stuck check WT-3030 Test failure indicating invalid key order during traversal WT-3034 Add support for single-writer named snapshots. WT-3037 Fix some outdated comments in logging WT-3048 WiredTiger maximum size warning uses the wrong format. WT-3051 Remove external __wt_hex symbol. WT-3052 Improve search if an index hint is wrong WT-3053 Review Python and Java calls to internal WiredTiger functions WT-3054 Java PackTest, PackTest03 do not compile WT-3055 Java AsyncTest faults WT-3056 For cursors with projections, keys should be allowed WT-3057 WiredTiger hazard pointers should use the WT_REF, not the WT_PAGE. WT-3061 syscall test runs with checkpoint_sync=false and doesn't acknowledge pwrite64 WT-3064 minor tree cleanups: .gitignore, NEWS misspelling WT-3066 lint WT-3068 Copy wtperf artifacts when running Jenkins tests WT-3069 Fix build failures in LevelDB APIs WT-3070 Fix search_near() for index cursor WT-3071 Java: fix build with -Werror=sign-conversion WT-3075 Document and enforce that WiredTiger now depends on Python 2.7 WT-3078 Fix a hang in the reconfiguration test. WT-3084 Fix Coverity resource leak complaint. Branch: v3.4 https://github.com/mongodb/mongo/commit/d2c64ac8c526b70eadeb859ec41370a5f03a64aa

          People

          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:
              Days since reply:
              24 weeks, 5 days ago
              Date of 1st Reply:

                Agile