[SERVER-21942] Make WiredTiger better suited for testing scenarios Created: 18/Dec/15  Updated: 06/Dec/22  Resolved: 28/Mar/16

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.0
Fix Version/s: None

Type: Improvement Priority: Minor - P4
Reporter: ITWEBTF SAXOBANK Assignee: Backlog - Storage Execution Team
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows


Issue Links:
Related
Assigned Teams:
Storage Execution
Participants:

 Description   

WiredTiger is not very fast for testing scenarios in which a potentially large number of tests read and write relatively small amounts of data.

Using ephemeralForTest is hardly any faster.

I suggest a --forTest mode for mongod with WiredTiger. This mode must be optimized for speed for the testing scenario. Data can be "persisted" in RAM.



 Comments   
Comment by Ramon Fernandez Marina [ 28/Mar/16 ]

Since this is no longer an issue, and no further information can be provided at the moment to enhance WiredTiger in the requested manner I'm going to close this ticket for the time being. We can always reopen it in the future if there's renewed interest in the matter.

Regards,
Ramón.

Comment by ITWEBTF SAXOBANK [ 17/Mar/16 ]

Thanks, sending you the statistics is a good suggestion.

However, due to problems with backwards compatibility with Mongo 3.x we have been forced to refocus, so we will not upgrade to WT right now and I have not tried a RAM drive.

So, this is not a problem for us right now. Out of personal interest I would like to do it anyway, but I cannot promise an exact ETA.

If you feel that this will most likely help other users as well, I will try to do it soon?

BTW, right now I cannot send you logs, not even from a test run, as you have not signed an NDA with us. Perhaps you could push whoever is stalling that in your end so we can get rid of that obstacle?

Regards,
Brian

Comment by Michael Cahill (Inactive) [ 17/Mar/16 ]

itwebtf@saxobank.com, have you had a chance to test with a RAM drive?

If test speed is still an issue for you, another thing we could do is take a look at statistics during the test run to see whether anything jumps out. Can you upload a MongoDB log file and zipped diagnostic.data directory from inside the database path after a test run?

Comment by Michael Cahill (Inactive) [ 18/Feb/16 ]

Hi Brian,

There is a list of choices here:

https://en.wikipedia.org/wiki/List_of_RAM_drive_software

In particular, this one is free:

http://www.ltr-data.se/opencode.html/#ImDisk

Comment by ITWEBTF SAXOBANK [ 11/Feb/16 ]

Hi,

I certainly expect that WiredTiger has been designed for big data, not for few documents in few collections.

However, I specifically do not drop collections until after the last test, so that is hardly the issue.

It might be that the fsync calls are still relevant, but I am not sure how that relates to Windows. A --forTest option could surely eat my data .

I could try a RAM-disk of some sort. Can you recommend one for Windows?

Regards,
Brian

Comment by Michael Cahill (Inactive) [ 11/Feb/16 ]

Hi Brian,

Thanks for the extra information.

In CS-26613 you mention the following as appearing in profiles:

MongoDB.Driver.Internal.MongoConnection.ReceiveMessage
MongoDB.Bson.IO.ByteBufferFactory.LoadFrom
MongoDB.Driver.MongoCursorEnumerator`1.MoveNext
MongoDB.Driver.MongoCursorEnumerator`1.GetFirst
MongoDB.Driver.MongoCursorEnumerator`1.GetReply

None of those relate directly to storage engine I/O. It may be that the strong persistence guarantees from WiredTiger, which involve fsync calls during collection create and drop operations, are causing tests to run slower. That would not necessarily appear in CPU profiles.

One way to confirm would be to set up mongod to run on an in-memory filesystem (e.g., tmpfs on Linux) during testing. Another option would be to use something like https://www.flamingspork.com/projects/libeatmydata/ to avoid durability guarantees during testing.

Please let us know whether either of those options helps.

Comment by ITWEBTF SAXOBANK [ 05/Feb/16 ]

Hi Michael,

The workload consists of a few hundred tests. Before running the entire set, indexes are ensured on all collections. At this time the database is empty, most collections are not there and the few collections that may be there are populated with very few documents, typically 1 to 3 documents.

For each test, all documents are removed from all collections. Each test run will populate a few collections with documents, typically 1 to 5 collections with each 1 to 3 documents from JSon text. When the test runs, these collections will be read and possibly written to. Also a few more collections will be written to in any test.

After the last test has run, collections with no documents are dropped.

As you can see in CS-26613 I profiled this once, and I found that the read operations in the database engine might be the bottle neck?

Let me know if you need more details.

Regarding a --forTest flag, I suggest that it will instruct the engine to work 100% in memory (with 0% I/O overhead), if the assumption about I/O being the bottle neck is true. Naturally, this will only work for very small amounts of data, which is quite OK.

Best regards,
Brian Elgaard Bennett

Comment by Michael Cahill (Inactive) [ 05/Feb/16 ]

itwebtf@saxobank.com, can you please explain the workload you mention above that completes in 1.21 on MMAPv1 but takes longer with WiredTiger? What operations are included in this measurement: is most of the time involved in starting / stopping mongod? Have you measured on any platforms other than Windows?

What would you expect a "--forTest" flag to do differently to regular mode in order to go faster? Is your sense that a lot of time in your tests is spent writing data to disk for a clean shutdown when you are about to delete it?

Comment by ITWEBTF SAXOBANK [ 04/Jan/16 ]

Hi Dan,

Surely,these engines have different characteristics that will affect integration test differently from the production web site.

The tests are mostly single-threaded, so they will not benefit from enhanced concurrency support in WT. In fact, as you suggest, the overhead may have an impact in the testing usage scenario. However, I did not expect such a significant impact since I see no reason for significant optimistic concurrency retries in these tests.

We do not have large, in-place updates in these tests.

As I mentioned before, it is quite OK for developers to do the first wave of initial testing using whatever engine is the fastest for that purpose.

But I am slightly worried about the performance of WT which we will probably switch to in the near future.

I suggest that the MongoDb Engine Team,

  • Verifies the performance of the usage scenario that I have outlined.
    • If as designed, i.e. no performance problems for the important usage scenarios for which WT was designed, then not a problem.
    • If not as designed, then go fix
  • Considers a --ForTest mode of WT
Comment by Daniel Pasette (Inactive) [ 18/Dec/15 ]

WT and MMAPv1 have some properties which your test case may be tickling in unexpected ways. For example, MMAPv1 is often more performant in single-threaded applications (because of the overhead of enabling high concurrency applications). This is also true when you have many workers trying to update a small set of docs, because optimistic concurrency control causes retries in WT. In addition, large, in-place updates are more efficiently executed in the MMAPv1 engine than in WT because WT uses copy-on-write to allow for concurrent write operations. These conditions are often the case for small integration test suites. Could any of these issues be in play?

Comment by ITWEBTF SAXOBANK [ 18/Dec/15 ]

Hi Ramon,

It made a difference, as execution time with Wired Tiger went from 4:58 with 3.2.0 to 3:58 with 3.2.1-rc0.

However, with mmapv1 on 3.2.0, execution time is typically 1.21.

I can live with using another engine for initial testing, but using the engine we are going to go Live with is preferable.

And potentially, a non-persisting and otherwise optimized-for-test-only mode could make Wired Tiger even faster.

Best regards,
Brian

Comment by Ramon Fernandez Marina [ 18/Dec/15 ]

itwebtf@saxobank.com, since you're running 3.2.0 on Windows you may have been affected by SERVER-21792. Could you please download 3.2.1-rc0 and re-run your experiments? It would be very useful to see how much direct I/O was affecting your use case, and whether 3.2.1 will meet your specific needs.

Thanks,
Ramón.

Generated at Thu Feb 08 03:58:52 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.