[SERVER-45694] Larger RamLog maximum log line length when testing Created: 22/Jan/20 Updated: 06/Dec/22 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | Logging, Testing Infrastructure |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Kevin Pulo | Assignee: | Backlog - Security Team |
| Resolution: | Unresolved | Votes: | 1 |
| Labels: | move-sec | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||
| Assigned Teams: |
Server Security
|
||||||||||||||||||||
| Backwards Compatibility: | Minor Change | ||||||||||||||||||||
| Participants: | |||||||||||||||||||||
| Description |
|
Examining server logs from jstests is sometimes necessary (eg. to check behaviour of commands that aren't profiled, such as auth admin commands or update/delete bulk commands (SERVER-23266)). This is typically done using the getLog command (or the checkLog shell helper). However, the current RamLog line length limit may be overrun, especially for commands in sharded environments. This makes tests that need to examine the logs this way inherently fragile. Simply increasing the maximum length adds only modest memory usage, but may have implications for any downstream users of the getLog command (especially if the getLog output is stored/processed). So a better solution would be to increase the maximum line length only during testing, ie. when --enableTestCommands has been specified. This would allow the fragility of log-based tests to be greatly reduced, without affecting non-test consumers of getLog output. |
| Comments |
| Comment by Kevin Pulo [ 02/Mar/20 ] |
|
The changes in |
| Comment by Kevin Pulo [ 28/Jan/20 ] |
|
Sorry, I should have been clearer. While 2048 is currently long enough, one of the log messages being checked is around 1700 bytes (with another in-progress ticket likely to increase that by up to 130 bytes). So although the immediate need is much reduced, I'm concerned that in the future we might still want something more flexible (on the assumption that downstream consumers will eventually see some impact if we just keep naively bumping the limit when tests break). It's also hard to diagnose when jstests break because of this, since when reviewing the logs the necessary matching line appears to be present and fine. Another possibility could be to adjust the checkLog js helper, and/or the getLog command, to output some warning if a line is encountered which is exactly C-1 bytes long (ie. has almost certainly been truncated). |
| Comment by Henrik Edin [ 27/Jan/20 ] |
|
Great! kevin.pulo Does this mean we can close this ticket? Or do you feel we need something more flexible? |
| Comment by Kevin Pulo [ 24/Jan/20 ] |
|
Thanks maria.vankeulen and henrik.edin! It turns out that 2048 is also sufficient for my current needs, so I've repurposed this ticket accordingly. |
| Comment by Maria van Keulen [ 23/Jan/20 ] |
|
As an update, I've included the 2048 byte increase as part of |
| Comment by Maria van Keulen [ 23/Jan/20 ] |
|
That suggestion sounds great kevin.pulo. In the interest of proceeding with |
| Comment by Kevin Pulo [ 23/Jan/20 ] |
|
Actually, it occurs to me that although the line length is currently a compile-time constant, it might not be too hard to make it adjustable (allocate lines on the heap, and provide a setLineLength() which makes a new one and memcpys the old lines into it, etc). Then it could stay at 1024 by default, but with a hook in --enableTestCommands that makes it as large as we like for testing purposes (for the global log). |