[SERVER-17115] insert workload stalls on oplog truncates Created: 29/Jan/15  Updated: 18/Sep/15  Resolved: 31/Jan/15

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.0.0-rc7
Fix Version/s: 3.0.0-rc8

Type: Bug Priority: Critical - P2
Reporter: Asya Kamsky Assignee: Michael Cahill (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File capped-throttle-deadlock.diff     PNG File dropout-after-fix.png     PNG File gdbmon.png     Text File gdbmon.txt     Text File serverStatus.txt     HTML File ss.html     Text File ss.log     PNG File ss.png     Text File ssRC7.log     File stuck1G.gdbmon     File stuck1G_2.gdbmon     File stuck2G.gdbmon     HTML File stuck2G.html     PNG File stuck2G.png     File stuck94cb08.gdbmon    
Issue Links:
Related
related to SERVER-17121 Throughput Dropout on mongo-sim inser... Closed
related to SERVER-17094 repairing local db with WiredTiger wi... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

YCSB load phase, running with no journal, oplog size 2GB, cache_size 2GB.
Inserting 30M documents fieldcount=1 (so 151 bytes each).

After period of time the inserts stall, seemingly forever waiting for eviction threads.

Have also reproduced this with journal - needed a smaller cache (used 1G).



 Comments   
Comment by Githook User [ 02/Feb/15 ]

Author:

{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@wiredtiger.com'}

Message: SERVER-17115 Avoid deadlocks in oplog back-pressure when the WiredTiger cache is full.

(cherry picked from commit 51ae76922e18a097e4ec89aa0a65a1d6a9a51cef)
Branch: v3.0
https://github.com/mongodb/mongo/commit/6f50ace8c8a57d34490965291cd5638ee4554c03

Comment by Githook User [ 31/Jan/15 ]

Author:

{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@wiredtiger.com'}

Message: SERVER-17115 Avoid deadlocks in oplog back-pressure when the WiredTiger cache is full.
Branch: master
https://github.com/mongodb/mongo/commit/51ae76922e18a097e4ec89aa0a65a1d6a9a51cef

Comment by Michael Cahill (Inactive) [ 29/Jan/15 ]

The gdbmon output suggests that most threads are waiting on the oplog truncation. Without a journal throttling inserts, my suspicion is that the inserts can overwhelm the oplog truncation thread.

Comment by Bruce Lucas (Inactive) [ 29/Jan/15 ]

With line numbers.

Generated at Thu Feb 08 03:43:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.