Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2534

Invalid transaction snapshots on PowerPC

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: WT2.8.0
    • Fix Version/s: WT2.9.0
    • Labels:
      None
    • # Replies:
      9
    • Last comment by Customer:
      true

      Description

      While testing MongoDB on PowerPC, we found a situation where reads were not repeatable in a transaction running at snapshot isolation. In particular, with multiple threads inserting and deleting records, we saw:

      begin transaction with isolation=snapshot
      search for ID=X, found
      ...
      search for ID=X, not found
      

        Issue Links

          Activity

          Hide
          michael.cahill Michael Cahill added a comment -

          The underlying issue here was assumptions in WiredTiger about atomic loads and stores of 64-bit values.

          In particular, we maintain a "current transaction ID" in the WT_TXN_GLOBAL struct, usually referred to as txn_global->current. When establishing a snapshot, we read that value into a local variable and rely on getting an atomic read. The local copy is then used to establish the snapshot. It is critical for correctness that allocated snapshot IDs are visible in the global transaction state table before and thread sees txn_global->current incremented. Otherwise, the result is an invalid transaction snapshot where values are visible before their transaction commits.

          We update txn_global->currrent when a new transaction ID is allocated, and that is done inside a spinlock to guarantee uniqueness. The ID is published with a write barrier before executing ++txn_global->current, then the spinlock is released.

          On x86, this is sufficient, and readers only see the increment after the ID is published. On PowerPC, we have seen a variety of results where reading txn_global->current results in unexpected values and thus invalid snapshots. Replacing the ordinary increment with an atomic_fetch_and_add eliminates these errors in testing.

          Show
          michael.cahill Michael Cahill added a comment - The underlying issue here was assumptions in WiredTiger about atomic loads and stores of 64-bit values. In particular, we maintain a "current transaction ID" in the WT_TXN_GLOBAL struct, usually referred to as txn_global->current . When establishing a snapshot, we read that value into a local variable and rely on getting an atomic read. The local copy is then used to establish the snapshot. It is critical for correctness that allocated snapshot IDs are visible in the global transaction state table before and thread sees txn_global->current incremented. Otherwise, the result is an invalid transaction snapshot where values are visible before their transaction commits. We update txn_global->currrent when a new transaction ID is allocated, and that is done inside a spinlock to guarantee uniqueness. The ID is published with a write barrier before executing ++txn_global->current , then the spinlock is released. On x86, this is sufficient, and readers only see the increment after the ID is published. On PowerPC, we have seen a variety of results where reading txn_global->current results in unexpected values and thus invalid snapshots. Replacing the ordinary increment with an atomic_fetch_and_add eliminates these errors in testing.
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: WT-2534 Use atomic add when allocating transaction IDs.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/c0446cf3b7a6fa0e596f103458e53ef7302b17d5

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: WT-2534 Use atomic add when allocating transaction IDs. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/c0446cf3b7a6fa0e596f103458e53ef7302b17d5
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: Merge pull request #2637 from wiredtiger/wt-2534

          WT-2534 Use atomic add when allocating transaction IDs.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/5535876ce00b27135a728dee4b5171ab0732481b

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Merge pull request #2637 from wiredtiger/wt-2534 WT-2534 Use atomic add when allocating transaction IDs. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/5535876ce00b27135a728dee4b5171ab0732481b
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: WT-2534 Use atomic add when allocating transaction IDs.
          Branch: mongodb-3.4
          https://github.com/wiredtiger/wiredtiger/commit/c0446cf3b7a6fa0e596f103458e53ef7302b17d5

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: WT-2534 Use atomic add when allocating transaction IDs. Branch: mongodb-3.4 https://github.com/wiredtiger/wiredtiger/commit/c0446cf3b7a6fa0e596f103458e53ef7302b17d5
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: Merge pull request #2637 from wiredtiger/wt-2534

          WT-2534 Use atomic add when allocating transaction IDs.
          Branch: mongodb-3.4
          https://github.com/wiredtiger/wiredtiger/commit/5535876ce00b27135a728dee4b5171ab0732481b

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Merge pull request #2637 from wiredtiger/wt-2534 WT-2534 Use atomic add when allocating transaction IDs. Branch: mongodb-3.4 https://github.com/wiredtiger/wiredtiger/commit/5535876ce00b27135a728dee4b5171ab0732481b
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: Import wiredtiger-wiredtiger-2.8.0-137-g94c171d.tar.gz from wiredtiger branch mongodb-3.4

          ref: 5047aab..94c171d

          WT-2534 Invalid transaction snapshots on PowerPC
          SERVER-23517 WiredTiger changes for MongoDB 3.3.5
          Branch: master
          https://github.com/mongodb/mongo/commit/281700981dca1a43e4fc32364951b09c562f1720

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Import wiredtiger-wiredtiger-2.8.0-137-g94c171d.tar.gz from wiredtiger branch mongodb-3.4 ref: 5047aab..94c171d WT-2534 Invalid transaction snapshots on PowerPC SERVER-23517 WiredTiger changes for MongoDB 3.3.5 Branch: master https://github.com/mongodb/mongo/commit/281700981dca1a43e4fc32364951b09c562f1720
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: WT-2534 Use atomic add when allocating transaction IDs.
          Branch: mongodb-3.2
          https://github.com/wiredtiger/wiredtiger/commit/c0446cf3b7a6fa0e596f103458e53ef7302b17d5

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: WT-2534 Use atomic add when allocating transaction IDs. Branch: mongodb-3.2 https://github.com/wiredtiger/wiredtiger/commit/c0446cf3b7a6fa0e596f103458e53ef7302b17d5
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: Merge pull request #2637 from wiredtiger/wt-2534

          WT-2534 Use atomic add when allocating transaction IDs.
          Branch: mongodb-3.2
          https://github.com/wiredtiger/wiredtiger/commit/5535876ce00b27135a728dee4b5171ab0732481b

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Merge pull request #2637 from wiredtiger/wt-2534 WT-2534 Use atomic add when allocating transaction IDs. Branch: mongodb-3.2 https://github.com/wiredtiger/wiredtiger/commit/5535876ce00b27135a728dee4b5171ab0732481b
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'}

          Message: Import wiredtiger-wiredtiger-2.8.0-201-g7ea2631.tar.gz from wiredtiger branch mongodb-3.2

          ref: 43e885a..7ea2631

          SERVER-23504 Coverity analysis defect 98177: Resource leak
          SERVER-23526 Replication relies on storage engines reporting a non-zero size for correctness
          SERVER-23588 mongod with WiredTiger won't start on Windows when built with --dbg=on --opt=off
          SERVER-23682 WiredTiger changes for MongoDB 3.2.6
          WT-2330 in-memory configurations should not create on-disk collection files
          WT-2507 Add upgrading documentation in preparation for 2.8 release.
          WT-2512 wtperf: MSVC complains about float conversion in throttle code
          WT-2513 conversion from 'int64_t' to 'uint32_t'
          WT-2517 wtperf uses setvbuf in a way that isn't supported on Windows
          WT-2522 Incorrect format code in message
          WT-2525 in-memory configurations: miscellaneous cleanups
          WT-2527 OS X compile error, missing POSIX_FADV_WILLNEED #define
          WT-2528 style error in WiredTiger build
          WT-2529 The readonly test case is crashing with a stack overflow
          WT-2531 in-memory tables are allocating unnecessary memory
          WT-2532 WT_STREAM_APPEND and WT_STREAM_LINE_BUFFER flag overlap
          WT-2533 Ensure that in-memory tables don't report a zero size
          WT-2534 Invalid transaction snapshots on PowerPC
          Branch: v3.2
          https://github.com/mongodb/mongo/commit/7ee4e4e493c3785fea489ee3508ca18526709c16

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'} Message: Import wiredtiger-wiredtiger-2.8.0-201-g7ea2631.tar.gz from wiredtiger branch mongodb-3.2 ref: 43e885a..7ea2631 SERVER-23504 Coverity analysis defect 98177: Resource leak SERVER-23526 Replication relies on storage engines reporting a non-zero size for correctness SERVER-23588 mongod with WiredTiger won't start on Windows when built with --dbg=on --opt=off SERVER-23682 WiredTiger changes for MongoDB 3.2.6 WT-2330 in-memory configurations should not create on-disk collection files WT-2507 Add upgrading documentation in preparation for 2.8 release. WT-2512 wtperf: MSVC complains about float conversion in throttle code WT-2513 conversion from 'int64_t' to 'uint32_t' WT-2517 wtperf uses setvbuf in a way that isn't supported on Windows WT-2522 Incorrect format code in message WT-2525 in-memory configurations: miscellaneous cleanups WT-2527 OS X compile error, missing POSIX_FADV_WILLNEED #define WT-2528 style error in WiredTiger build WT-2529 The readonly test case is crashing with a stack overflow WT-2531 in-memory tables are allocating unnecessary memory WT-2532 WT_STREAM_APPEND and WT_STREAM_LINE_BUFFER flag overlap WT-2533 Ensure that in-memory tables don't report a zero size WT-2534 Invalid transaction snapshots on PowerPC Branch: v3.2 https://github.com/mongodb/mongo/commit/7ee4e4e493c3785fea489ee3508ca18526709c16

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                1 year, 10 weeks, 3 days ago
                Date of 1st Reply: