[SERVER-39654] Storage statistics not logged for a slow transaction Created: 19/Feb/19  Updated: 29/Oct/23  Resolved: 21/Mar/19

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: 4.0.9, 4.1.10

Type: Bug Priority: Major - P3
Reporter: Sulabh Mahajan Assignee: Sulabh Mahajan
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File repro.js    
Issue Links:
Backports
Related
related to SERVER-39361 Synchronise collecting storage engine... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.0
Sprint: Storage Engines 2019-03-11, Storage Engines 2019-03-25
Participants:
Story Points: 8

 Description   

Slow operations extract the operation statistics from the storage engine and report them as part of logging and profiling. For a slow transaction, these storage statistics are meant to be cumulative over the operations performed in that transaction.

At the moment, we do not get any storage information for the slow transaction. It looks like we are not collecting the storage stats at the correct place, we might be collecting them too late, past the point where other metrics for a transaction get accumulated.



 Comments   
Comment by Githook User [ 03/Apr/19 ]

Author:

{'email': 'sulabh.mahajan@mongodb.com', 'name': 'Sulabh Mahajan', 'username': 'sulabhM'}

Message: SERVER-38239 Added getOperationStatistics() API to fetch storage stats from WiredTiger

(cherry picked from commit ba3894493a94ed3c18458f391ff181d57475f010)

SERVER-38240 Added storage statistics information into the slowop log messages.

(cherry picked from commit 86b6aca9fa1940e85bba87261d1494ef2c208a4a)

SERVER-38240 work around uint64_t conversion on s390x in WiredTigerOperationStats::fetchStats()

(cherry picked from commit 82161eec79fc74652dc07b1c83fe500dc4f95e79)

SERVER-38243 Test presence of the storage stats in slowop logs and system.profile.

(cherry picked from commit 3b4c6a689a3fdaa923d427ae112ea599513ef8ce)

SERVER-39026 Use correct type for retrieving WiredTiger stats

(cherry picked from commit 6a9a5855048df1f4796a4032276d01318c398691)

SERVER-39488 Look for storage stats in the find command's profiled entry.

The find command will always have the storage stats because the
test has created a collection with btree spanning multiple pages. Scanning
this collection after a server restart will trigger read from the disk
and have the disk read stats.

(cherry picked from commit 6125b5fb078a316854f0299b96b7d16eacb944de)

SERVER-39361 Synchronise collecting storage engine stats with shutdown

(cherry picked from commit bacb6b67706a2c057fcd0f76a38f416b225aa69a)

SERVER-39061 Fix the wt_operation_stats test to wait for the operation log to appear.

(cherry picked from commit 371197e4bab715a83272a4472e118ee5c5cbbf7c)

SERVER-39934 Fix locking for slow ops storage stats
SERVER-39654 Make slow ops storage stats work with transactions

(cherry picked from commit 23ca771f76f85638f23bca2a4a6ac196a81fdc21)
Branch: v4.0
https://github.com/mongodb/mongo/commit/57edce7396271531dde4499458b22c9cde1f03d4

Comment by Githook User [ 21/Mar/19 ]

Author:

{'email': 'sulabh.mahajan@mongodb.com', 'name': 'Sulabh Mahajan', 'username': 'sulabhM'}

Message: SERVER-39934 Fix locking for slow ops storage stats
SERVER-39654 Make slow ops storage stats work with transactions
Branch: master
https://github.com/mongodb/mongo/commit/23ca771f76f85638f23bca2a4a6ac196a81fdc21

Comment by Sulabh Mahajan [ 27/Feb/19 ]

geert.bosch,
At the moment we do not know if the cost of calling into the storage engine per operation would have a significant impact. Without that data, we should not decide against having this ability. We can definitely choose to not collect per operation statistics if it turns out to be a bottleneck in transaction performance.

Comment by Geert Bosch [ 27/Feb/19 ]

It seems to me that having storage engine stats on the level of individual operations in a transaction is going to be necessarily slow, as we'll have to make an extra expensive call into the storage engine for each operation, even if it is not slow. So, I suggest that we just give up on that idea, and just remember if a transaction had any slow operations. If so, then at transaction commit or abort log the storage engine stats for the transaction.

Generated at Thu Feb 08 04:52:42 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.