[SERVER-38764] External sorter should use 64-bit integers for file offsets Created: 22/Dec/18 Updated: 29/Oct/23 Resolved: 18/Jan/19 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Index Maintenance |
| Affects Version/s: | 4.1.6 |
| Fix Version/s: | 3.4.21, 3.6.12, 4.0.7, 4.1.8 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | O | Assignee: | Louis Williams |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||
| Operating System: | ALL | ||||||||||||
| Backport Requested: |
v4.0, v3.6
|
||||||||||||
| Steps To Reproduce: |
|
||||||||||||
| Sprint: | Storage NYC 2019-01-28 | ||||||||||||
| Participants: | |||||||||||||
| Description |
|
I've a big collection, about 1.2bi documents, and I tried to create an index on this collection. I reproduced this bug with several fields.
Bellow the logs of the crash:
2018-12-22T21:25:04.993+0000 I INDEX [conn8] build index on: dashboard.delivery properties: { v: 2, unique: true, key: { address: 1.0 }, name: "address_1", ns: "dashboard.delivery" } ***aborting after invariant() failure 2018-12-22T22:05:49.597+0000 F - [conn8] Got signal: 6 (Aborted). {"backtrace":[ ,{"b":"56187AD3F000","o":"2472109"},{"b":"56187AD3F000","o":"24725ED"},{"b":"7F3AA87D4000","o":"11390"},{"b":"7F3AA840A000","o":"35428","s":"gsignal"},{"b":"7F3AA840A000","o":"3702A","s":"abort"}, {"b":"56187AD3F000","o":"A453A3","s":"_ZN5mongo22invariantFailedWithMsgEPKcRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES1_j"},{"b":"56187AD3F000","o":"1ABFA78","s":"_ZN5mongo6sorter12FileIteratorINS_7BSONObjENS_8RecordIdEE4readEPvm"},{"b":"56187AD3F000","o":"1ABFAE1","s":"_ZN5mongo6sorter12FileIteratorINS_7BSONObjENS_8RecordIdEE18fillBufferFromDiskEv"},{"b":"56187AD3F000","o":"1AC00CB","s":"_ZN5mongo6sorter12FileIteratorINS_7BSONObjENS_8RecordIdEE4moreEv"},{"b":"56187AD3F000","o":"1ABD91F","s":"_ZN5mongo6sorter13MergeIteratorINS_7BSONObjENS_8RecordIdENS_27BtreeExternalSortComparisonEE4moreEv"},{"b":"56187AD3F000","o":"1ABB718","s":"_ZN5mongo25AbstractIndexAccessMethod10commitBulkEPNS_16OperationContextEPNS_17IndexAccessMethod11BulkBuilderEbbPSt3setINS_8RecordIdESt4lessIS7_ESaIS7_EEPSt6vectorINS_7BSONObjESaISE_EE"},{"b":"56187AD3F000","o":"12C3128","s":"_ZN5mongo19MultiIndexBlockImpl20_dumpInsertsFromBulkEPSt3setINS_8RecordIdESt4lessIS2_ESaIS2_EEPSt6vectorINS_7BSONObjESaIS9_EE"},{"b":"56187AD3F000","o":"12C3424","s":"_ZN5mongo19MultiIndexBlockImpl19dumpInsertsFromBulkEv"},{"b":"56187AD3F000","o":"12C2308","s":"_ZN5mongo19MultiIndexBlockImpl30insertAllDocumentsInCollectionEv"},{"b":"56187AD3F000","o":"118D034"}, {"b":"56187AD3F000","o":"1DF9356","s":"_ZN5mongo23ErrmsgCommandDeprecated3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjERNS_14BSONObjBuilderE"},{"b":"56187AD3F000","o":"1DFC534","s":"_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE"},{"b":"56187AD3F000","o":"EB9E34"},{"b":"56187AD3F000","o":"EBC1AE"},{"b":"56187AD3F000","o":"EBD94C"},{"b":"56187AD3F000","o":"EBE926","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE"},{"b":"56187AD3F000","o":"EAAB6A","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"56187AD3F000","o":"EB6F73","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"56187AD3F000","o":"EB2767","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"56187AD3F000","o":"EB6151"},{"b":"56187AD3F000","o":"1C1D0F2","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"56187AD3F000","o":"EB0840","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"56187AD3F000","o":"EB3B85","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"56187AD3F000","o":"EB1D9F","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"56187AD3F000","o":"EB27ED","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"56187AD3F000","o":"EB6151"},{"b":"56187AD3F000","o":"1C1D665"},{"b":"56187AD3F000","o":"23C87D4"},{"b":"7F3AA87D4000","o":"76BA"},{"b":"7F3AA840A000","o":"10741D","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.1.6", "gitVersion" : "55e72b015e2aa7297c00db29e4d93451ea61a7ca", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.9.0-8-amd64", "version" : "#1 SMP Debian 4.9.130-2 (2018-10-27)", "machine" : "x86_64" }, "somap" : [ { "b" : "56187AD3F000", "elfType" : 3, "buildId" : "107FA13E3E2D27E599436B56147F0382ADA8826C" }, { "b" : "7FFFFAFA0000", "elfType" : 3, "buildId" : "0B1047D7814362A9CE0444D05F6284EDA613920B" }, { "b" : "7F3AA9BE4000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "E2D1DB1857AE8F07908959381751796E89FF56BB" }, { "b" : "7F3AA99C9000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "6EF73266978476EF9F2FD2CF31E57F4597CB74F8" }, { "b" : "7F3AA9585000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "250E875F74377DFC74DE48BF80CCB237BB4EFF1D" }, { "b" : "7F3AA931C000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "513282AC7EB386E2C0133FD9E1B6B8A0F38B047D" }, { "b" : "7F3AA9118000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "8CC8D0D119B142D839800BFF71FB71E73AEA7BD4" }, { "b" : "7F3AA8F10000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "89C34D7A182387D76D5CDA1F7718F5D58824DFB3" }, { "b" : "7F3AA8C07000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "DFB85DE42DAFFD09640C8FE377D572DE3E168920" }, { "b" : "7F3AA89F1000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F3AA87D4000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "CE17E023542265FC11D9BC8F534BB4F070493D30" }, { "b" : "7F3AA840A000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B5381A457906D279073822A5CEB24C4BFEF94DDB" }, { "b" : "7F3AA9E53000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "5D7B6259552275A3C17BD4C3FD05F5A6BF40CAA5" }, { "b" : "7F3AA81D7000", "path" : "/usr/lib/x86_64-linux-gnu/libidn.so.11", "elfType" : 3, "buildId" : "E09D3783AD1D0BBCD3204FA01E4EF6D756E18F57" }, { "b" : "7F3AA7FBB000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "8D1CC1204D6B6D33BD1D2C5A2A0516A2234322CF" }, { "b" : "7F3AA7D71000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "DB5180B568097E2A4690A5B40D36BD134C893FEE" }, { "b" : "7F3AA7B62000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "5B146086EC173C299BA45F0C92E068D68CC7AD37" }, { "b" : "7F3AA7911000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "D02A94DB90567C2C45F40C8565809A041FD22A55" }, { "b" : "7F3AA76F7000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "8D9BD4CE26E45EF16075C67D5F5EEAFD8B562832" }, { "b" : "7F3AA73C7000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "3CE893F6D1382C2C7648DCCB06E71B1C7E0861CC" }, { "b" : "7F3AA7194000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "B11678F560199547DCF726384EA39153EE0DFABF" }, { "b" : "7F3AA6F5E000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "D6B36C5A463EE0FA84FDD6D5FD3F7726EDB90D54" }, { "b" : "7F3AA6CDE000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "7B3533D5998D20EE1A1BE3F87789B69041E7F620" }, { "b" : "7F3AA6A0C000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "16E3DBC6D048145939BB43BBFD7954D27421B00F" }, { "b" : "7F3AA67DD000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "AEB4C08FC47F86C475E9D3996DFE5E9B403ACEBF" }, { "b" : "7F3AA65D9000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "1E16CB57F699E215A2A8D4EFEF90883BC749B12D" }, { "b" : "7F3AA63CE000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "DF3219B89E86920E901BAC4A80AA60F2B6134588" }, { "b" : "7F3AA61B3000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "87783DF8A1058CD150F8886CB36340384093C18F" }, { "b" : "7F3AA5F72000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "1FE877BE52A424D0636AFD4D35BB330E41D6E0F3" }, { "b" : "7F3AA5D0E000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "A0E2D03FF5CF65937F4425D4EFD4D655243809EB" }, { "b" : "7F3AA5AFB000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "E07E186694852D8F69459C6AB28A53F8DA3CE3B6" }, { "b" : "7F3AA58F7000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "3364D4BF2113C4E8D17EF533867ECC99A53413D6" }, { "b" : "7F3AA56EE000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "73A8EADBC85860662B24850E71D4AFBE22C33359" }, { "b" : "7F3AA5464000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "59E742306A4EA2872E061ECCE92F35FADDA75357" }, { "b" : "7F3AA51C2000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "E5C159E415406AE79D21056D752BA949C408B5B1" }, { "b" : "7F3AA4F8F000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "7D15576E1F096614D360784E4A01A1F5FAF908C9" }, { "b" : "7F3AA4D79000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "481DB33C28D88E43DA6BED65E1A7599407D4D818" }, { "b" : "7F3AA4B71000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "9D9C958F1F4894AFEF6AECD90D1C430EA29AC34F" }, { "b" : "7F3AA4948000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "57E25072866B2D30CF02EBE7AE623B84F96FA700" }, { "b" : "7F3AA4739000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "F6F1B4E9F89B716C4A0BA5819BDFFAF4A13EFB91" }, { "b" : "7F3AA44EE000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "C60082E3BB78D0D42868D9B359B89BF66CE5A1A7" }, { "b" : "7F3AA4219000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "D9782BA023CAEC26B15D8676E3A5D07B55E121EF" }, { "b" : "7F3AA3FE1000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "7BDD51353D50310FFA1587E4AA01B40ABE32D582" } ] }} |
| Comments |
| Comment by Githook User [ 11/Jun/19 ] |
|
Author: {'name': 'Dianna Hohensee', 'email': 'dianna.hohensee@10gen.com', 'username': 'DiannaHohensee'}Message:
(cherry picked from commit 48d999c08304b6ede2a9d1f9d9db974b59fe97e2) |
| Comment by Githook User [ 12/Mar/19 ] |
|
Author: {'name': 'Dianna Hohensee', 'email': 'dianna.hohensee@10gen.com', 'username': 'DiannaHohensee'}Message: (cherry picked from commit 2be7f2677a40a863f336d2964f456c9d87ddc838)
(cherry picked from commit 9dafb7a3e3bafa463ab5951189b670965995dada) |
| Comment by Githook User [ 25/Feb/19 ] |
|
Author: {'name': 'Louis Williams', 'email': 'louis.williams@mongodb.com', 'username': 'louiswilliams'}Message: (cherry picked from commit 9dafb7a3e3bafa463ab5951189b670965995dada) |
| Comment by Githook User [ 18/Jan/19 ] |
|
Author: {'username': 'louiswilliams', 'email': 'louis.williams@mongodb.com', 'name': 'Louis Williams'}Message: |
| Comment by Louis Williams [ 16/Jan/19 ] |
|
The problem here is the static_cast to an unsigned 32 bit integer truncates the high-order bytes of tellg() when the file is larger than 4GB. In fact, this entire implementation uses 32-bit unsigned integers when it should be using 64-bit unsigned integers. |
| Comment by O [ 27/Dec/18 ] |
|
I finished this test and the conclusion is that non-unique indexes and indexes on arrays working on 4.0.4, but not on 4.1.6.
|
| Comment by O [ 26/Dec/18 ] |
|
I loaded the same database into a 4.0.4 instance and created an index (unique), and it worked. I'm testing now with a non unique index, and an index in an array. Soon as I have the results I will post it here. |
| Comment by O [ 26/Dec/18 ] |
|
I created a dump of this database and imported it. The same bug happens. |
| Comment by O [ 26/Dec/18 ] |
|
Since when moving a collection from a database to another some indexes are rebuilt, I did it as a test, and I got the following error: On the shell > db.runCommand({renameCollection: 'db1.col1', to: 'db2.col1'}) { "ok" : 0, "errmsg" : "BSONObj size: 1684301409 (0x64646661) is invalid. Size must be between 0 and 16793600(16MB)", "code" : 10334, "codeName" : "BSONObjectTooLarge" }On server logs
2018-12-26T10:13:01.739+0000 I COMMAND [conn79] command db2.tmpw7evh.renameCollection appName: "MongoDB Shell" command: renameCollection { renameCollection: "db1.col1", to: "db2.col1", lsid: { id: UUID("fa8f217f-63e7-4138-bf23-03fd55afaf60") }, $db: "admin" } numYields:0 ok:0 errMsg:"BSONObj size: 1684301409 (0x64646661) is invalid. Size must be between 0 and 16793600(16MB)" errName:BSONObjectTooLarge errCode:10334 reslen:185 locks:{ Global: { acquireCount: { r: 3, w: 6, W: 1 }}, Database: { acquireCount: { r: 1, w: 1, W: 1 }}, Collection: { acquireCount: { r: 1, w: 1 }} } protocol:op_msg 54798794ms |
| Comment by O [ 24/Dec/18 ] |
|
I finished importing the data into the collection with the indexes already created and the error didn't happen. |
| Comment by O [ 24/Dec/18 ] |
|
Thank you Kelsey, I will try to repeat the same procedure with the 4.0.4 version sooner as I can. Now, just for sake of completeness, I'm trying to insert the documents into the database with the indexes (one simple, one unique and one in an array) already created, since I'm more than half way through (about 70%), I will just wait to give you a more complete feedback of the bug. In case of failure, I will repeat the procedure with 4.0.4 and give you a feedback about it. |
| Comment by Kelsey Schubert [ 23/Dec/18 ] |
|
Hi ooo, Thanks for the report, we'll look into it. Please note that that the MongoDB 4.1.x series is a development branch and not suitable for production use. Until MongoDB 4.2 is released containing the work that has gone into the 4.1.x series, we recommend running the latest stable branch, MongoDB 4.0. For more information our on version numbering system see here. Kind regards, |
| Comment by O [ 23/Dec/18 ] |
|
The same bug also occurs when creating non-unique indexes. |