[SERVER-18987] Write Slow in WiredTiger Database Created: 16/Jun/15  Updated: 18/Jun/15  Resolved: 18/Jun/15

Status: Closed
Project: Core Server
Component/s: WiredTiger, Write Ops
Affects Version/s: 3.0.3
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: carl dong Assignee: Sam Kleinman (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: JPEG File OpsManager.jpg     Text File db.log    
Participants:

 Description   

Hi Team ,

There is a performance issue with write operations , data insert to the database very slow in a sudden and will get back to normal in some seconds .

Strange Logs like this :

2015-06-16T10:23:58.212+0800 I COMMAND  [conn287864] command ISH_Fact.$cmd command: insert { insert: "FactAppLoginLog", ordered: true, documents: [ { _id: ObjectId('557f88bda310a27a4f3893eb'), userid: "557f5befa310c0e3f3324e2d", username: "15026835043", passauthtime: 578, locationinfo: { height: 0.0, precision: 43.0, gps: [ 31.21235656738281, 121.4836959838867 ] }, ispass: true, uuid: "58a41e3c-2392-3e97-be73-0a4542c3ff0f", os: "Android", type: "PAD", version: "9", eventtime: new Date(1434421366267), eventt: 6, ct: new Date(1434421437738) } ] } keyUpdates:0 writeConflicts:0 numYields:0 reslen:80 locks:{ Global: { acquireCount: { w: 3 } }, Database: { acquireCount: { w: 3 } }, Collection: { acquireCount: { w: 2 } }, oplog: { acquireCount: { w: 1 } } } 474ms

2015-06-16T10:23:58.213+0800 I COMMAND  [conn287789] command ISH_Fact.$cmd command: insert { insert: "FactPortalADPresent", ordered: true, documents: [ { _id: ObjectId('557f88bda310a27a4f3893ef'), spotid: "new_ishmobile_rotate_B03", batchid: "9aa09df5-4929-8bfc-13af-556fbe1dd41c", os: "iOS 7", type: "Smartphone", op: "telecom", buv: "14dfa2e18f0-067523f0-e621-4f96-8df9-39e78e4ba0c7", un: "13501880398", site: "i-shanghai", et: new Date(1434421437899) } ] } keyUpdates:0 writeConflicts:0 numYields:0 reslen:80 locks:{ Global: { acquireCount: { w: 3 } }, Database: { acquireCount: { w: 3 } }, Collection: { acquireCount: { w: 2 } }, oplog: { acquireCount: { w: 1 } } } 290ms

As you can see , the op only locks the db for 2 or 3 ms but it cost 300ms to finish , there is only _id index on those tables , about 4,000 write ops every minute on those tables.

Could you advise how to trouble shooting the issue ?

Thanks,

Carl Dong



 Comments   
Comment by Sam Kleinman (Inactive) [ 18/Jun/15 ]

Just as a reminder, the SERVER project is for bug reports and feature requests of the core MongoDB software. It might make more sense to tracking troubleshooting and investigative projects via one of our support channels. I'm going to go ahead and close this issue, but we can reopen or open another if we find a specific related issue here.

  • Are the slow insert operations happening in a capped collection? There are a number of issues with the performance of capped collections with WiredTiger that we're currently tracking. For example, SERVER-18875, SERVER-18081 and SERVER-17033.
  • Does modifying the size of the WiredTiger cache impact the performance characteristics that you observe? The rate of cache evictions may account for some of your observed performance.
  • Is performance consistently slow, or is the slowness intermittent only at certain points of day or on some other interval?
  • Are only inserts impacted?
  • What write concern did you specify on these write operations?

I hope that this provides you with a good starting point to begin your troubleshooting!

Regards,
sam

Comment by carl dong [ 16/Jun/15 ]

Mongo Instance Config file :

processManagement :
fork : true
pidFilePath : /home/ISH/ISH.pid
systemLog:
destination : file
path : /home/ISH/Log/iSH_DB_Log
logAppend : true

storage:
dbPath : /home/ISH/Data_WiredTiger
engine : wiredTiger
wiredTiger :
engineConfig :
cacheSizeGB : 20
directoryPerDB : true
net:
port : 30001
maxIncomingConnections : 2000

security:
authorization : enabled
keyFile : /home/ISH/Keys/MongoKey

replication:
replSetName : ish

auditLog:
destination : file
format : JSON
path : /home/ISH/Log/ISHDBAduit.json
filter : '{ atype : {$nin : ["authenticate"]} }'

Generated at Thu Feb 08 03:49:27 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.