[SERVER-21942] Make WiredTiger better suited for testing scenarios Created: 18/Dec/15 Updated: 06/Dec/22 Resolved: 28/Mar/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | 3.2.0 |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Minor - P4 |
| Reporter: | ITWEBTF SAXOBANK | Assignee: | Backlog - Storage Execution Team |
| Resolution: | Incomplete | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Windows |
||
| Issue Links: |
|
||||
| Assigned Teams: |
Storage Execution
|
||||
| Participants: | |||||
| Description |
|
WiredTiger is not very fast for testing scenarios in which a potentially large number of tests read and write relatively small amounts of data. Using ephemeralForTest is hardly any faster. I suggest a --forTest mode for mongod with WiredTiger. This mode must be optimized for speed for the testing scenario. Data can be "persisted" in RAM. |
| Comments |
| Comment by Ramon Fernandez Marina [ 28/Mar/16 ] | |||||
|
Since this is no longer an issue, and no further information can be provided at the moment to enhance WiredTiger in the requested manner I'm going to close this ticket for the time being. We can always reopen it in the future if there's renewed interest in the matter. Regards, | |||||
| Comment by ITWEBTF SAXOBANK [ 17/Mar/16 ] | |||||
|
Thanks, sending you the statistics is a good suggestion. However, due to problems with backwards compatibility with Mongo 3.x we have been forced to refocus, so we will not upgrade to WT right now and I have not tried a RAM drive. So, this is not a problem for us right now. Out of personal interest I would like to do it anyway, but I cannot promise an exact ETA. If you feel that this will most likely help other users as well, I will try to do it soon? BTW, right now I cannot send you logs, not even from a test run, as you have not signed an NDA with us. Perhaps you could push whoever is stalling that in your end so we can get rid of that obstacle? Regards, | |||||
| Comment by Michael Cahill (Inactive) [ 17/Mar/16 ] | |||||
|
itwebtf@saxobank.com, have you had a chance to test with a RAM drive? If test speed is still an issue for you, another thing we could do is take a look at statistics during the test run to see whether anything jumps out. Can you upload a MongoDB log file and zipped diagnostic.data directory from inside the database path after a test run? | |||||
| Comment by Michael Cahill (Inactive) [ 18/Feb/16 ] | |||||
|
Hi Brian, There is a list of choices here: https://en.wikipedia.org/wiki/List_of_RAM_drive_software In particular, this one is free: | |||||
| Comment by ITWEBTF SAXOBANK [ 11/Feb/16 ] | |||||
|
Hi, I certainly expect that WiredTiger has been designed for big data, not for few documents in few collections. However, I specifically do not drop collections until after the last test, so that is hardly the issue. It might be that the fsync calls are still relevant, but I am not sure how that relates to Windows. A --forTest option could surely eat my data I could try a RAM-disk of some sort. Can you recommend one for Windows? Regards, | |||||
| Comment by Michael Cahill (Inactive) [ 11/Feb/16 ] | |||||
|
Hi Brian, Thanks for the extra information. In CS-26613 you mention the following as appearing in profiles:
None of those relate directly to storage engine I/O. It may be that the strong persistence guarantees from WiredTiger, which involve fsync calls during collection create and drop operations, are causing tests to run slower. That would not necessarily appear in CPU profiles. One way to confirm would be to set up mongod to run on an in-memory filesystem (e.g., tmpfs on Linux) during testing. Another option would be to use something like https://www.flamingspork.com/projects/libeatmydata/ to avoid durability guarantees during testing. Please let us know whether either of those options helps. | |||||
| Comment by ITWEBTF SAXOBANK [ 05/Feb/16 ] | |||||
|
Hi Michael, The workload consists of a few hundred tests. Before running the entire set, indexes are ensured on all collections. At this time the database is empty, most collections are not there and the few collections that may be there are populated with very few documents, typically 1 to 3 documents. For each test, all documents are removed from all collections. Each test run will populate a few collections with documents, typically 1 to 5 collections with each 1 to 3 documents from JSon text. When the test runs, these collections will be read and possibly written to. Also a few more collections will be written to in any test. After the last test has run, collections with no documents are dropped. As you can see in CS-26613 I profiled this once, and I found that the read operations in the database engine might be the bottle neck? Let me know if you need more details. Regarding a --forTest flag, I suggest that it will instruct the engine to work 100% in memory (with 0% I/O overhead), if the assumption about I/O being the bottle neck is true. Naturally, this will only work for very small amounts of data, which is quite OK. Best regards, | |||||
| Comment by Michael Cahill (Inactive) [ 05/Feb/16 ] | |||||
|
itwebtf@saxobank.com, can you please explain the workload you mention above that completes in 1.21 on MMAPv1 but takes longer with WiredTiger? What operations are included in this measurement: is most of the time involved in starting / stopping mongod? Have you measured on any platforms other than Windows? What would you expect a "--forTest" flag to do differently to regular mode in order to go faster? Is your sense that a lot of time in your tests is spent writing data to disk for a clean shutdown when you are about to delete it? | |||||
| Comment by ITWEBTF SAXOBANK [ 04/Jan/16 ] | |||||
|
Hi Dan, Surely,these engines have different characteristics that will affect integration test differently from the production web site. The tests are mostly single-threaded, so they will not benefit from enhanced concurrency support in WT. In fact, as you suggest, the overhead may have an impact in the testing usage scenario. However, I did not expect such a significant impact since I see no reason for significant optimistic concurrency retries in these tests. We do not have large, in-place updates in these tests. As I mentioned before, it is quite OK for developers to do the first wave of initial testing using whatever engine is the fastest for that purpose. But I am slightly worried about the performance of WT which we will probably switch to in the near future. I suggest that the MongoDb Engine Team,
| |||||
| Comment by Daniel Pasette (Inactive) [ 18/Dec/15 ] | |||||
|
WT and MMAPv1 have some properties which your test case may be tickling in unexpected ways. For example, MMAPv1 is often more performant in single-threaded applications (because of the overhead of enabling high concurrency applications). This is also true when you have many workers trying to update a small set of docs, because optimistic concurrency control causes retries in WT. In addition, large, in-place updates are more efficiently executed in the MMAPv1 engine than in WT because WT uses copy-on-write to allow for concurrent write operations. These conditions are often the case for small integration test suites. Could any of these issues be in play? | |||||
| Comment by ITWEBTF SAXOBANK [ 18/Dec/15 ] | |||||
|
Hi Ramon, It made a difference, as execution time with Wired Tiger went from 4:58 with 3.2.0 to 3:58 with 3.2.1-rc0. However, with mmapv1 on 3.2.0, execution time is typically 1.21. I can live with using another engine for initial testing, but using the engine we are going to go Live with is preferable. And potentially, a non-persisting and otherwise optimized-for-test-only mode could make Wired Tiger even faster. Best regards, | |||||
| Comment by Ramon Fernandez Marina [ 18/Dec/15 ] | |||||
|
itwebtf@saxobank.com, since you're running 3.2.0 on Windows you may have been affected by Thanks, |