Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-8962

Modify dir_store to collect/report estimated cost

    • Storage Engines
    • 5
    • StorEng - Defined Pipeline

      Estimate costs for running various tests


      • Ability to run various tiered storage tests backed by dir_store and have an idea of how much they cost per run.
      • Is this issue urgent?
        Not urgent.  Appropriate for new hire, intern or skunk works.

      Acceptance Criteria (Definition of Done)
      The "dir_store" extension should be modified so that it has an option to collect statistics on bytes transferred (read/write), total bytes stored, etc.  We should have a mechanism to show these when the storage source is closed (that is, when the connection is closed).  We can apply a simple dollar calculation using numbers from https://aws.amazon.com/s3/pricing/?nc=sn&loc=4 .  Perhaps use "standard pricing" for simplicity.

      Suggested Implementation
      S3_store uses the msg_printf callback to transmit these to the caller.  Unfortunately, this puts the burden on the individual python tests (or test/format or test_tiered_abort, etc.) to ignore these kinds of messages, but report or collate them at the right time.  Part of the issue is that many of these tests will open/close connections many times, and so will want to total the statistics at the end of the entire run. Perhaps better is to have a log file location passed as argument to the extension, the log file is appended to and the entire "cost log" can be collated/added at the end of a run, by a separate program. (rather than having the adding up done separately by each test program).  For purposes of this ticket, just printing them to a shared log file would be enough.

            backlog-server-storage-engines [DO NOT USE] Backlog - Storage Engines Team
            donald.anderson@mongodb.com Donald Anderson
            0 Vote for this issue
            5 Start watching this issue