[WT-2237] Make committed changes visible immediately Created: 24/Nov/15  Updated: 08/Nov/17  Resolved: 25/Nov/15

Status: Closed
Project: WiredTiger
Component/s: None
Affects Version/s: None
Fix Version/s: WT2.7.0

Type: Bug Priority: Major - P3
Reporter: Michael Cahill Assignee: Michael Cahill
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Depends
is depended on by SERVER-21442 WiredTiger changes for MongoDB 3.0.8 Closed
is depended on by SERVER-21549 WiredTiger changes for 3.2.0-rc4 Closed
is depended on by SERVER-21686 WiredTiger changes for 3.2.0-rc5 Closed
Duplicate
is duplicated by SERVER-21275 Document not found due to WT commit v... Closed
Related
Sub-Tasks:
Key
Summary
Type
Status
Assignee
WT-2241 Improve performance of transaction ID... Sub-task Closed Michael Cahill  

 Description   

A bug in WiredTiger caused updates to not be visible to concurrent readers immediately upon commit. Applications that rely on updates being visible immediately (including MongoDB) temporarily see unexpected results. After waiting or restarting, all committed data would be present and visible, but reads during this window could see "holes" with recent changes missing.

Background: WiredTiger uses transaction snapshots to determine which versions of updates should be visible to readers. When a transaction T starts, it builds a list of transaction IDs of running transactions (the transactions concurrent with T). Updates created with those transaction IDs are not visible to reads in T.

WiredTiger has avoided locking in the transaction subsystem by having threads race to allocate transaction IDs. Multiple threads can publish the same ID but only one thread wins the race and goes on to use that ID for updates. Other threads could temporarily leave a transaction ID published in the global state table. If T completes quickly enough that another thread still has T's ID published after T commits, then T's ID will appear in snapshots after T commits, delaying when T's updates become visible.



 Comments   
Comment by Michael Cahill [ 24/Nov/15 ]

CR: https://github.com/wiredtiger/wiredtiger/pull/2331

Comment by Githook User [ 24/Nov/15 ]

Author:

{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

Message: WT-2237 Have threads publish unique transaction IDs so that updates always become visible immediately on commit.

Previously, WiredTiger avoided locking in the transaction subsystem by having
threads race to allocate transaction IDs. Only one thread wins the race, but
losing threads could temporarily leave a transaction ID published in the global
state table. If the winning thread completed its transaction quickly enough,
its changes may not become visible immediately to readers (until the losing
threads have woken up and allocated a new ID).

This change separates ID allocation from the notion of a shared "current"
value. Every thread gets a unique transaction ID, and spins to update the
current value.
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/0ba1737427565b679ea6a8dd43c4ca175dd3f613

Comment by Githook User [ 24/Nov/15 ]

Author:

{u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

Message: Merge pull request #2331 from wiredtiger/WT-2237

WT-2237 Have threads publish unique transaction IDs so that updates always become visible immediately on commit
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/0a52a80a39fc47145fb755d792792ae820b266ed

Comment by Githook User [ 24/Nov/15 ]

Author:

{u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

Message: Merge pull request #2331 from wiredtiger/WT-2237

WT-2237 Have threads publish unique transaction IDs so that updates always become visible immediately on commit
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/0a52a80a39fc47145fb755d792792ae820b266ed

Comment by Michael Cahill [ 24/Nov/15 ]

There is a significant perf impact of the original change on parallel workloads including YCSB.

Comment by Michael Cahill [ 24/Nov/15 ]

New CR: https://github.com/wiredtiger/wiredtiger/pull/2333

Comment by Githook User [ 24/Nov/15 ]

Author:

{u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

Message: Merge pull request #2333 from wiredtiger/WT-2337-perf

WT-2237 Avoid yields if we race allocating transaction IDs.
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/e90b5906b969e9e34c8aa7e304a5983ef8077bca

Comment by Githook User [ 24/Nov/15 ]

Author:

{u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'}

Message: Import wiredtiger-wiredtiger-mongodb-3.2.0-rc3-206-gb65381f.tar.gz from wiredtiger branch mongodb-3.2

ref: 4d72349..b65381f

e90b590 WT-2237 Avoid yields if we race allocating transaction IDs.
0a52a80 WT-2237 Have threads publish unique transaction IDs so that updates always become visible immediately on commit
Branch: master
https://github.com/mongodb/mongo/commit/327660ff3324dfcb3ea38fc63a4fc7f1cdcb9078

Comment by Githook User [ 24/Nov/15 ]

Author:

{u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'}

Message: Import wiredtiger-wiredtiger-mongodb-3.2.0-rc3-206-gb65381f.tar.gz from wiredtiger branch mongodb-3.2

ref: 4d72349..b65381f

e90b590 WT-2237 Avoid yields if we race allocating transaction IDs.
0a52a80 WT-2237 Have threads publish unique transaction IDs so that updates always become visible immediately on commit
Branch: master
https://github.com/mongodb/mongo/commit/327660ff3324dfcb3ea38fc63a4fc7f1cdcb9078

Comment by Sue LoVerso [ 24/Nov/15 ]

Michael Cahill the Jenkins plots still show significant performance drops with the parallel-perf-stress and medium-multi-lsm tests. I compared a changeset from yesterday, 2cf57ab with today's changeset with the WT_PAUSE fix, e90b5906.

On my AWS HDD box, I see a very large drop in parallel-perf-stress.wtperf. The pre-2237 changeset completes 10M inserts in 17 seconds. With e90b5906, it takes 307 seconds. I ran zoom on it and confirmed that all of the time is spent in the wt_yield call in wt_txn_id_alloc. I bumped the WT_PAUSE/yield threshold to 1000 (from 100) and that changed the time to 276 seconds.

Comment by Githook User [ 02/Dec/15 ]

Author:

{u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

Message: Merge pull request #2331 from wiredtiger/WT-2237

WT-2237 Have threads publish unique transaction IDs so that updates always become visible immediately on commit.

(cherry picked from commit 0a52a80a39fc47145fb755d792792ae820b266ed)
Branch: mongodb-3.0
https://github.com/wiredtiger/wiredtiger/commit/16a418b471953ad04973f5b013b1647e940ab769

Comment by Githook User [ 02/Dec/15 ]

Author:

{u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

Message: Merge pull request #2331 from wiredtiger/WT-2237

WT-2237 Have threads publish unique transaction IDs so that updates always become visible immediately on commit.

(cherry picked from commit 0a52a80a39fc47145fb755d792792ae820b266ed)
Branch: mongodb-3.0
https://github.com/wiredtiger/wiredtiger/commit/16a418b471953ad04973f5b013b1647e940ab769

Comment by Githook User [ 02/Dec/15 ]

Author:

{u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

Message: Merge pull request #2333 from wiredtiger/WT-2337-perf

WT-2237 Avoid yields if we race allocating transaction IDs.

(cherry picked from commit e90b5906b969e9e34c8aa7e304a5983ef8077bca)
Branch: mongodb-3.0
https://github.com/wiredtiger/wiredtiger/commit/06a5c7b7a99053110395209c23ff26d39ba22db4

Comment by Githook User [ 02/Dec/15 ]

Author:

{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

Message: Import wiredtiger-wiredtiger-mongodb-3.0.7-20-g27d0cbd.tar.gz from wiredtiger branch mongodb-3.0

ref: deb2d81..27d0cbd

6feaa28 WT-2251 Fix leaf of deleted page addresses.
1da2d3a SERVER-21568 Fix a use-after-free.
4228295 SERVER-21553 Free blocks during reverse splits.
0398515 WT-2241 Use a lock to protect transaction ID allocation.
06a5c7b WT-2237 Avoid yields if we race allocating transaction IDs.
16a418b WT-2237 Have threads publish unique transaction IDs so that updates always become visible immediately on commit.
Branch: v3.0
https://github.com/mongodb/mongo/commit/eb8bc24a0d36b673022faa83976a4b33bc8f676b

Comment by Githook User [ 02/Dec/15 ]

Author:

{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

Message: Import wiredtiger-wiredtiger-mongodb-3.0.7-20-g27d0cbd.tar.gz from wiredtiger branch mongodb-3.0

ref: deb2d81..27d0cbd

6feaa28 WT-2251 Fix leaf of deleted page addresses.
1da2d3a SERVER-21568 Fix a use-after-free.
4228295 SERVER-21553 Free blocks during reverse splits.
0398515 WT-2241 Use a lock to protect transaction ID allocation.
06a5c7b WT-2237 Avoid yields if we race allocating transaction IDs.
16a418b WT-2237 Have threads publish unique transaction IDs so that updates always become visible immediately on commit.
Branch: v3.0
https://github.com/mongodb/mongo/commit/eb8bc24a0d36b673022faa83976a4b33bc8f676b

Generated at Thu Apr 26 09:14:46 UTC 2018 using JIRA 7.2.10#72012-sha1:2651463a07e52d81c0fcf01da710ca333fcb42bc.