[SERVER-6816] Improve journal data handling after multithreaded batch writing Created: 21/Aug/12  Updated: 11/Jul/16  Resolved: 07/Sep/12

Status: Closed
Project: Core Server
Component/s: Replication, Storage
Affects Version/s: 2.2.0-rc1
Fix Version/s: 2.2.1, 2.3.0

Type: Improvement Priority: Major - P3
Reporter: Eric Milkie Assignee: Eric Milkie
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-6925 DR102 too much data written uncommitted Closed
Duplicate
is duplicated by SERVER-9048 Out of memory leads to crash and node... Closed
Related
related to SERVER-7414 Test improved journal data handling a... Closed
is related to SERVER-6806 [journal] warning assertion failure a... Closed
Participants:

 Description   

On a secondary in a replica set, no journal commit is allowed during an oplog application after a replicated batch write. The batch write currently can write up to 256 megabytes of ops. If some amount of data close to 256 megs is written and then applied to the oplog, the journal tends to warn about this with warning messages such as

[rsSync] warning: DR102 too much data written uncommitted 314.864MB
[rsSync] warning: DR102 too much data written uncommitted 314.868MB
[rsSync] warning: DR102 too much data written uncommitted 314.872MB
[rsSync] warning: DR102 too much data written uncommitted 314.876MB
 
 warning assertion failure d.size() < 1024 src/mongo/util/concurrency/task.cpp 
 [journal] warning assertion failure a <= 256*1024*1024 src/mongo/util/alignedbuilder.cpp 90
 [rsSync] local.oplog.rs warning assertion failure _intents.size() < 2000000 src/mongo/db/dur_commitjob.h 101

especially if a high volume of many small writes. The hardcoded trigger levels for these messages should be adjusted so that end users do not see them for routine use. Alternatively, the maximum batch size should be reevaluated (is it too large?).



 Comments   
Comment by auto [ 13/Sep/12 ]

Author:

{u'date': u'2012-09-13T09:09:11-07:00', u'email': u'milkie@10gen.com', u'name': u'Eric Milkie'}

Message: SERVER-6816 cap repl batches at 1 second or 5000 ops
Branch: v2.2
https://github.com/mongodb/mongo/commit/f6dede5f105f3824881f0f3e258f0fc261247077

Comment by auto [ 13/Sep/12 ]

Author:

{u'date': u'2012-09-13T09:09:11-07:00', u'email': u'milkie@10gen.com', u'name': u'Eric Milkie'}

Message: SERVER-6816 cap repl batches at 1 second or 5000 ops
Branch: master
https://github.com/mongodb/mongo/commit/3b7bb12a556868be2082dbd9eb81431082a75d7b

Comment by auto [ 12/Sep/12 ]

Author:

{u'date': u'2012-09-07T09:35:35-07:00', u'email': u'milkie@10gen.com', u'name': u'Eric Milkie'}

Message: SERVER-6816 add a 5 sec time limit to replication batches

Even with a byte cap on replication batches, it is still possible to
batch up a large number of writes which can take a long time to apply.
This commit ensures that batches do not get too large. 5 seconds' worth
of replicated ops over the network will take much less than 5 seconds to
write to the database.

Conflicts:

src/mongo/db/repl/rs_sync.cpp
Branch: v2.2
https://github.com/mongodb/mongo/commit/d89e787dec901d79f595a31de7e2e66d0898fc4b

Comment by auto [ 12/Sep/12 ]

Author:

{u'date': u'2012-09-06T13:06:18-07:00', u'email': u'milkie@10gen.com', u'name': u'Eric Milkie'}

Message: SERVER-6816 cap the batch on the journal commit limit

Calling commitIfNeeded() in a local db lock is prohibited so I removed it. Instead, we will make the batch sizes smaller than the journal commit limit size to avoid DR102 messages.
Branch: v2.2
https://github.com/mongodb/mongo/commit/5ab736e73e59d17b0867cb05f15c89866bc68657

Comment by auto [ 12/Sep/12 ]

Author:

{u'date': u'2012-09-05T07:39:47-07:00', u'email': u'kristina@10gen.com', u'name': u'Kristina'}

Message: Flush journal after every oplog write (if necessary) SERVER-6816
Branch: v2.2
https://github.com/mongodb/mongo/commit/e9e9cc60a49f9bc4ec833d54ed72a9e83918a84a

Comment by auto [ 07/Sep/12 ]

Author:

{u'date': u'2012-09-07T09:35:35-07:00', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: SERVER-6816 add a 5 sec time limit to replication batches

Even with a byte cap on replication batches, it is still possible to
batch up a large number of writes which can take a long time to apply.
This commit ensures that batches do not get too large. 5 seconds' worth
of replicated ops over the network will take much less than 5 seconds to
write to the database.
Branch: master
https://github.com/mongodb/mongo/commit/2eeaad837463a929e1bd3d0488cf2884687e9359

Comment by auto [ 07/Sep/12 ]

Author:

{u'date': u'2012-09-06T13:06:18-07:00', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: SERVER-6816 cap the batch on the journal commit limit

Calling commitIfNeeded() in a local db lock is prohibited so I removed it. Instead, we will make the batch sizes smaller than the journal commit limit size to avoid DR102 messages.
Branch: master
https://github.com/mongodb/mongo/commit/00f7df9fb484313af0e6091c83c4112cc1842aff

Comment by Eric Milkie [ 06/Sep/12 ]

Not quite done yet – we can't flush the journal in that state, so I'm going to do it a different way. Composing commit now.

Comment by Eric Milkie [ 05/Sep/12 ]

I'll do it in a different ticket.

Comment by Kristina Chodorow (Inactive) [ 05/Sep/12 ]

Not sure if you want to lower byte limit before resolving, too.

Comment by auto [ 05/Sep/12 ]

Author:

{u'date': u'2012-09-05T07:39:47-07:00', u'email': u'kristina@10gen.com', u'name': u'Kristina'}

Message: Flush journal after every oplog write (if necessary) SERVER-6816
Branch: master
https://github.com/mongodb/mongo/commit/cebad1a2aee9b5142417f343109ac50f3d7fe70d

Comment by Eric Milkie [ 05/Sep/12 ]

We should probably just call commitIfNeeded() while in fast_oplog_insert().

Generated at Thu Feb 08 03:12:46 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.