[SERVER-6816] Improve journal data handling after multithreaded batch writing Created: 21/Aug/12 Updated: 11/Jul/16 Resolved: 07/Sep/12 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication, Storage |
| Affects Version/s: | 2.2.0-rc1 |
| Fix Version/s: | 2.2.1, 2.3.0 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Eric Milkie | Assignee: | Eric Milkie |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||
| Description |
|
On a secondary in a replica set, no journal commit is allowed during an oplog application after a replicated batch write. The batch write currently can write up to 256 megabytes of ops. If some amount of data close to 256 megs is written and then applied to the oplog, the journal tends to warn about this with warning messages such as
especially if a high volume of many small writes. The hardcoded trigger levels for these messages should be adjusted so that end users do not see them for routine use. Alternatively, the maximum batch size should be reevaluated (is it too large?). |
| Comments |
| Comment by auto [ 13/Sep/12 ] |
|
Author: {u'date': u'2012-09-13T09:09:11-07:00', u'email': u'milkie@10gen.com', u'name': u'Eric Milkie'}Message: |
| Comment by auto [ 13/Sep/12 ] |
|
Author: {u'date': u'2012-09-13T09:09:11-07:00', u'email': u'milkie@10gen.com', u'name': u'Eric Milkie'}Message: |
| Comment by auto [ 12/Sep/12 ] |
|
Author: {u'date': u'2012-09-07T09:35:35-07:00', u'email': u'milkie@10gen.com', u'name': u'Eric Milkie'}Message: Even with a byte cap on replication batches, it is still possible to Conflicts: src/mongo/db/repl/rs_sync.cpp |
| Comment by auto [ 12/Sep/12 ] |
|
Author: {u'date': u'2012-09-06T13:06:18-07:00', u'email': u'milkie@10gen.com', u'name': u'Eric Milkie'}Message: Calling commitIfNeeded() in a local db lock is prohibited so I removed it. Instead, we will make the batch sizes smaller than the journal commit limit size to avoid DR102 messages. |
| Comment by auto [ 12/Sep/12 ] |
|
Author: {u'date': u'2012-09-05T07:39:47-07:00', u'email': u'kristina@10gen.com', u'name': u'Kristina'}Message: Flush journal after every oplog write (if necessary) |
| Comment by auto [ 07/Sep/12 ] |
|
Author: {u'date': u'2012-09-07T09:35:35-07:00', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}Message: Even with a byte cap on replication batches, it is still possible to |
| Comment by auto [ 07/Sep/12 ] |
|
Author: {u'date': u'2012-09-06T13:06:18-07:00', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}Message: Calling commitIfNeeded() in a local db lock is prohibited so I removed it. Instead, we will make the batch sizes smaller than the journal commit limit size to avoid DR102 messages. |
| Comment by Eric Milkie [ 06/Sep/12 ] |
|
Not quite done yet – we can't flush the journal in that state, so I'm going to do it a different way. Composing commit now. |
| Comment by Eric Milkie [ 05/Sep/12 ] |
|
I'll do it in a different ticket. |
| Comment by Kristina Chodorow (Inactive) [ 05/Sep/12 ] |
|
Not sure if you want to lower byte limit before resolving, too. |
| Comment by auto [ 05/Sep/12 ] |
|
Author: {u'date': u'2012-09-05T07:39:47-07:00', u'email': u'kristina@10gen.com', u'name': u'Kristina'}Message: Flush journal after every oplog write (if necessary) |
| Comment by Eric Milkie [ 05/Sep/12 ] |
|
We should probably just call commitIfNeeded() while in fast_oplog_insert(). |