[SERVER-17206] WT Secondaries fall behind on heavy insert workload but MMAPv1 secondaries don't Created: 06/Feb/15  Updated: 19/Sep/15  Resolved: 20/Feb/15

Status: Closed
Project: Core Server
Component/s: Replication, Storage
Affects Version/s: 3.0.0-rc8
Fix Version/s: 3.0.0-rc9, 3.1.0

Type: Bug Priority: Critical - P2
Reporter: Michael Grundy Assignee: Darren Wood
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File 1-op-cs.log     HTML File 1-oplog-cs.html     Text File 1-ss.log     HTML File 1lag.html     Text File 2-op-cs.log     HTML File 2-oplog-cs.html     Text File 2-ss.log     HTML File 2lag.html     Text File 3-op-cs.log     HTML File 3-oplog-cs.html     Text File 3-ss.log     HTML File 3lag.html     PNG File Dashboard___MMS__MongoDB_Management_Service.png     File mongod-1-.log.gz     File mongod-2-.log.gz     File mongod-3-.log.gz     PNG File node 1 WT primary timeline.png     PNG File node 2 MMAPv1 secondary timeline.png     PNG File node 3 WT secondary timeline.png     PNG File socialite-3wtnode.png     PNG File socialite_oplog_1G.png    
Issue Links:
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Steps To Reproduce:

Socialite load workload (available at https://github.com/10gen-labs/socialite).

java -jar target/socialite-0.0.1-SNAPSHOT.jar load  --users 10000000 --maxfollows 1000 --messages 2000 --threads 32 sample-config.yml

3 node replica set c3.2xlarge instances, 8 cpu, 15g.
Replica set configuration

{
	"_id" : "shard1",
	"version" : 1,
	"members" : [
		{
			"_id" : 1,
			"host" : "shard1-01.knuckleboys.com:27017"
		},
		{
			"_id" : 2,
			"host" : "shard1-02.knuckleboys.com:27017"
		},
		{
			"_id" : 3,
			"host" : "shard1-03.knuckleboys.com:27017"
		}
	]
}

Socialite load workload: java -jar target/socialite-0.0.1-SNAPSHOT.jar load --users 10000000 --maxfollows 1000 --messages 2000 --threads 32 sample-config.yml

Indexes:

[
	{
		"v" : 1,
		"key" : {
			"_id" : 1
		},
		"name" : "_id_",
		"ns" : "socialite.content"
	},
	{
		"v" : 1,
		"key" : {
			"_a" : 1,
			"_id" : 1
		},
		"name" : "_a_1__id_1",
		"ns" : "socialite.content"
	}
]
[
	{
		"v" : 1,
		"key" : {
			"_id" : 1
		},
		"name" : "_id_",
		"ns" : "socialite.followers"
	},
	{
		"v" : 1,
		"unique" : true,
		"key" : {
			"_f" : 1,
			"_t" : 1
		},
		"name" : "_f_1__t_1",
		"ns" : "socialite.followers"
	}
]
[
	{
		"v" : 1,
		"key" : {
			"_id" : 1
		},
		"name" : "_id_",
		"ns" : "socialite.following"
	},
	{
		"v" : 1,
		"unique" : true,
		"key" : {
			"_f" : 1,
			"_t" : 1
		},
		"name" : "_f_1__t_1",
		"ns" : "socialite.following"
	}
]
[
	{
		"v" : 1,
		"key" : {
			"_id" : 1
		},
		"name" : "_id_",
		"ns" : "socialite.users"
	}
]

Participants:

 Description   

Running the socialite load workload (primarily writes) against a three member replica set, 4g oplog with one secondary using MMAPv1 and the other using wiredTiger, the WT secondary starts falling behind. The MMAPv1 secondary keeps up.
Typically, I'll see the WT secondary get to about 1500s behind the primary, then it will clear up. This take about 2 hours to reproduce. After about 8-10 hours total run time the WT secondary starts to fall behind again, then falls off the tail of the oplog.
Screen shots of MMS during a recovered lag, and of timeline output (from https://github.com/10gen/support-tools) with correlated activity. The full timeline files are attached.



 Comments   
Comment by Daniel Pasette (Inactive) [ 12/Feb/15 ]

This commit in WT was merged in this morning. This allows checkpoints and statistics cursors to not block application of oplog operations to secondaries.

commit 33c146b51fdac86999e2eaa67f5636490eb441fb
Author: Michael Cahill <michael.cahill@wiredtiger.com>
Date:   Thu Feb 12 13:44:35 2015 +1100
 
    Disable aggregation across all open checkpoints if statistics cursors don't specify a checkpoint.

Comment by Githook User [ 11/Feb/15 ]

Author:

{u'username': u'monkey101', u'name': u'Dan Pasette', u'email': u'dan@10gen.com'}

Message: SERVER-17206 capped collection performance tweaks

(cherry picked from commit 47eafb2a05f902ffd9df5324a4c3490eaa581842)
Branch: v3.0
https://github.com/mongodb/mongo/commit/81c466950cf39ac9b17e8cf41139123351a67302

Comment by Githook User [ 11/Feb/15 ]

Author:

{u'username': u'monkey101', u'name': u'Dan Pasette', u'email': u'dan@10gen.com'}

Message: SERVER-17206 capped collection performance tweaks
Branch: master
https://github.com/mongodb/mongo/commit/47eafb2a05f902ffd9df5324a4c3490eaa581842

Comment by Michael Grundy [ 07/Feb/15 ]

Attaching the oplog collection statstimeline from each member, plus the raw data

Comment by Daniel Pasette (Inactive) [ 07/Feb/15 ]

Do you have collection stats for the oplog?

Generated at Thu Feb 08 03:43:38 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.