[SERVER-21627] invariant failure in mmapv1 code found in fsm_all.js concurrency test on ASAN builder Created: 05/Nov/15  Updated: 25/Nov/15  Resolved: 23/Nov/15

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: 3.2.0-rc4

Type: Bug Priority: Major - P3
Reporter: Spencer Brody (Inactive) Assignee: Geert Bosch
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: QuInt C (11/23/15)
Participants:

 Description   

task
logs

	
[MongoDFixture:job0] 2015-11-05T21:06:05.058+0000 I -        [durability] Invariant failure _len <= 256 * MB src/mongo/db/storage/mmap_v1/aligned_builder.cpp 114
[MongoDFixture:job0] 2015-11-05T21:06:05.058+0000 I -        [durability]
[MongoDFixture:job0] 
[MongoDFixture:job0] ***aborting after invariant() failure
[MongoDFixture:job0] 
[MongoDFixture:job0] 
[MongoDFixture:job0] 2015-11-05T21:06:05.060+0000 F -        [durability] Got signal: 6 (Aborted).
[MongoDFixture:job0] 
[MongoDFixture:job0]  0xb8e975 0x3127156 0x31255ef 0x7fc6211f9c90 0x7fc620936e37 0x7fc620938528 0x2f66552 0x255b7b3 0x2637447 0x2635f9e 0x25f7310 0x7fc620f95dd0 0x7fc6211f20a5 0x7fc6209f9cfd
[MongoDFixture:job0] ----- BEGIN BACKTRACE -----
[MongoDFixture:job0] {"backtrace":[{"b":"400000","o":"78E975"},{"b":"400000","o":"2D27156"},{"b":"400000","o":"2D255EF"},{"b":"7FC6211EA000","o":"FC90"},{"b":"7FC620900000","o":"36E37"},{"b":"7FC620900000","o":"38528"},{"b":"400000","o":"2B66552"},{"b":"400000","o":"215B7B3"},{"b":"400000","o":"2237447"},{"b":"400000","o":"2235F9E"},{"b":"400000","o":"21F7310"},{"b":"7FC620EDB000","o":"BADD0"},{"b":"7FC6211EA000","o":"80A5"},{"b":"7FC620900000","o":"F9CFD"}],"processInfo":{ "mongodbVersion" : "3.2.0-rc2-47-gfef65b6", "gitVersion" : "fef65b69202c850fd05cd22b94351688e4785188", "compiledModules" : [ "enterprise" ], "uname" : { "sysname" : "Linux", "release" : "3.16.0-25-generic", "version" : "#33-Ubuntu SMP Tue Nov 4 12:06:54 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFAEDFC000", "elfType" : 3, "buildId" : "B43B3054EC49983B9ECFC6BD437E70C564D0AE41" }, { "b" : "7FC622F21000", "path" : "/usr/lib/x86_64-linux-gnu/libnetsnmpmibs.so.30", "elfType" : 3, "buildId" : "B91F83FB5396891415AC793A269FADA85E9AE8E6" }, { "b" : "7FC622D12000", "path" : "/usr/lib/x86_64-linux-gnu/libsensors.so.4", "elfType" : 3, "buildId" : "859FDBFDD82F0EFDEB44A433D9D8020A232A35E2" }, { "b" : "7FC622B0E000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "794CD87201C9778112E22BF5E2C0FBFB3390D29F" }, { "b" : "7FC6228A4000", "path" : "/usr/lib/x86_64-linux-gnu/libnetsnmpagent.so.30", "elfType" : 3, "buildId" : "2592E4D30F4E48F824D9B28C70C417C913D23370" }, { "b" : "7FC62269A000", "path" : "/lib/x86_64-linux-gnu/libwrap.so.0", "elfType" : 3, "buildId" : "54FCBC5B0F994A13A9B0EAD46F23E7DA7F7FE75B" }, { "b" : "7FC6223BD000", "path" : "/usr/lib/x86_64-linux-gnu/libnetsnmp.so.30", "elfType" : 3, "buildId" : "15EDC916EE74D63897687E2821397153AA553F33" }, { "b" : "7FC621FDA000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "013EF089B82F17AD7F2BDB94FEC56724D84D7A17" }, { "b" : "7FC621CD4000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "4E96203F4FE17D3446F48226AAEA8DA6DEA8FFD0" }, { "b" : "7FC621AB9000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "087FF353C3AF449E68D982CC726D25F190CFA16F" }, { "b" : "7FC621871000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "9D161DFBB0934C7B67617845CDDEC792596479DF" }, { "b" : "7FC621610000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "44ADE6AD24BE7656EDC5B05A53974EA73F35D423" }, { "b" : "7FC621408000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "54EF3A97A3E71418DD088B40AF51A00457834A17" }, { "b" : "7FC6211EA000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "921196598AF41AFF8DE42EEFB8561243610F34C3" }, { "b" : "7FC620EDB000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "ADEF33B83967BBB41525AE439354F030694250C4" }, { "b" : "7FC620CC4000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "7C6E98219378EBD1AA0D4CD671E8FF1589C04C4A" }, { "b" : "7FC620900000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "95287BE8ACCCC7B5723F4306E6A5ECA6DFE7BFFD" }, { "b" : "7FC623391000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9240DBBD1DB14E756141EEE1FDDB67D3B77864E7" }, { "b" : "7FC62054C000", "path" : "/usr/lib/x86_64-linux-gnu/libperl.so.5.20", "elfType" : 3, "buildId" : "235B9327BA4CFF99A5BA06C8ABDF5540C428FEC2" }, { "b" : "7FC620332000", "path" : "/lib/x86_64-linux-gnu/libnsl.so.1", "elfType" : 3, "buildId" : "849CEB1735076A2B2DA8D37E02975C089B90BE1F" }, { "b" : "7FC620063000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "739DB0E7DDC123EBFC32746EDF4547F995FA2709" }, { "b" : "7FC61FE33000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "A83A134804E9ED6CA3DEB4E048F380B0A2C836CC" }, { "b" : "7FC61FC2F000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "2918F49225E4DD544A987E649DF9E83C916ED133" }, { "b" : "7FC61FA24000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "4DD597E9AB56E3F0348682F3AA2B71C9FEA69DA2" }, { "b" : "7FC61F7EB000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "932BFF4AB8275FE410B21411AA2F42DF42901904" }, { "b" : "7FC61F5E7000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "BBF6C707F97D9DF43C35062706137A46DA0D0AC6" }, { "b" : "7FC61F3CC000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "A98328780E1B2842B6073FD3021EB6902BBFFE7D" } ] }}
[MongoDFixture:job0]  mongod(__interceptor_backtrace+0x95) [0xb8e975]
[MongoDFixture:job0]  mongod(_ZN5mongo15printStackTraceERSo+0xF6) [0x3127156]
[MongoDFixture:job0]  mongod(+0x2D255EF) [0x31255ef]
[MongoDFixture:job0]  libpthread.so.0(+0xFC90) [0x7fc6211f9c90]
[MongoDFixture:job0]  libc.so.6(gsignal+0x37) [0x7fc620936e37]
[MongoDFixture:job0]  libc.so.6(abort+0x148) [0x7fc620938528]
[MongoDFixture:job0]  mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xDC2) [0x2f66552]
[MongoDFixture:job0]  mongod(_ZN5mongo14AlignedBuilder14growReallocateEj+0x753) [0x255b7b3]
[MongoDFixture:job0]  mongod(+0x2237447) [0x2637447]
[MongoDFixture:job0]  mongod(_ZN5mongo3dur13PREPLOGBUFFERERNS0_11JSectHeaderERNS_14AlignedBuilderE+0x50E) [0x2635f9e]
[MongoDFixture:job0]  mongod(+0x21F7310) [0x25f7310]
[MongoDFixture:job0]  libstdc++.so.6(+0xBADD0) [0x7fc620f95dd0]
[MongoDFixture:job0]  libpthread.so.0(+0x80A5) [0x7fc6211f20a5]
[MongoDFixture:job0]  libc.so.6(clone+0x6D) [0x7fc6209f9cfd]
[MongoDFixture:job0] -----  END BACKTRACE  -----



 Comments   
Comment by Githook User [ 23/Nov/15 ]

Author:

{u'username': u'GeertBosch', u'name': u'Geert Bosch', u'email': u'geert@mongodb.com'}

Message: SERVER-21627: Increase journal buffer limit for debug builds
Branch: master
https://github.com/mongodb/mongo/commit/d91c4c047197a408628a09f89c2489380b0e1f77

Comment by Geert Bosch [ 23/Nov/15 ]

As it looks like the journal overflows are legitimate given the workload and the mmapv1 flush design, I think it is reasonable to increase the limit. I don't think we want to spend much time doing any fundamental design improvements to MMAPv1 flushing at this point.

Comment by Daniel Pasette (Inactive) [ 08/Nov/15 ]

There's not much we can do about this as the maximum amount we allow to journal in debug builds is 256mb. This worked when all our tests were single threaded, but these tests now do concurrent large $out aggregations which overflow the journal buffer from time to time. We could raise the limit a bit – it's 2gb on non-debug builds.

Comment by Spencer Brody (Inactive) [ 05/Nov/15 ]

geert.bosch, I wasn't sure who owns mmapv1 these days, feel free to reassign if you think there is someone better to look into this.

Generated at Thu Feb 08 03:57:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.