[SERVER-32372] Mongos crashes on bulk insert where size is slightly bigger than maxBsonObjectSize Created: 15/Dec/17  Updated: 30/Oct/23  Resolved: 18/Dec/17

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 3.6.0
Fix Version/s: 3.6.1, 3.7.1

Type: Bug Priority: Major - P3
Reporter: Yuriy [X] Assignee: Kaloian Manassiev
Resolution: Fixed Votes: 0
Labels: SWNA
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Java Source File mongos.crash.snippet.java     Text File stacktrace.87522.inserts.txt     Text File stacktrace.87800.inserts.txt    
Issue Links:
Backports
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v3.6
Steps To Reproduce:

1. Mongo sharded cluster 3.6.0 with 2 replicas.
2. Launch mongos.crash.snippet.java against mongos.
In snippet you can check number of inserts (NUMBER_OF_INSERTS), if it equals 87522 or higher mongos crashes each time you try to insertMany.
You get different stacktraces depends on number of inserts.

Sprint: Sharding 2018-01-01
Participants:
Case:

 Description   

We upgraded mongo sharded cluster from 3.4.10 to 3.6.0 version and mongoses started to crash from insertMany operation, if its size is bigger than maxBsonObjectSize.
It seems that it begins because of upping maxWriteBatchSize in 3.6.0 to 100000.
Got two different stacktraces:

2017-12-15T23:05:47.701+0300 E -        [NetworkInterfaceASIO-TaskExecutorPool-8-0] Assertion: 10334:BSONObj size: 16793791 (0x10040BF) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "test2" src/mongo/bson/bsonobj.cpp 101
2017-12-15T23:05:47.702+0300 F ASIO     [NetworkInterfaceASIO-TaskExecutorPool-8-0] Uncaught exception in NetworkInterfaceASIO IO worker thread of type: Location10334: BSONObj size: 16793791 (0x10040BF) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "test2"
2017-12-15T23:05:47.702+0300 F -        [NetworkInterfaceASIO-TaskExecutorPool-8-0] Fatal Assertion 28820 at src/mongo/executor/network_interface_asio.cpp 169
2017-12-15T23:05:47.702+0300 F -        [NetworkInterfaceASIO-TaskExecutorPool-8-0]
 
***aborting after fassert() failure
 
 
2017-12-15T23:05:47.705+0300 F -        [NetworkInterfaceASIO-TaskExecutorPool-8-0] Got signal: 6 (Aborted).
 
 0xbe3ab8edd1 0xbe3ab8dfe9 0xbe3ab8e4cd 0x7f2821716390 0x7f2821370428 0x7f282137202a 0xbe39fc6013 0xbe3a6610b4 0xbe3ac9c2b0 0x7f282170c6ba 0x7f28214423dd
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"BE39AC6000","o":"10C8DD1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"BE39AC6000","o":"10C7FE9"},{"b":"BE39AC6000","o":"10C84CD"},{"b":"7F2821705000","o":"11390"},{"b":"7F282133B000","o":"35428","s":"gsignal"},{"b":"7F282133B000","o":"3702A","s":"abort"},{"b":"BE39AC6000","o":"500013","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"BE39AC6000","o":"B9B0B4"},{"b":"BE39AC6000","o":"11D62B0"},{"b":"7F2821705000","o":"76BA"},{"b":"7F282133B000","o":"1073DD","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.1-rc0", "gitVersion" : "7ffeba9c365fdd3d1eb47ea841b0882a2f8fead7", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-93-generic", "version" : "#116-Ubuntu SMP Fri Aug 11 21:17:51 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "BE39AC6000", "elfType" : 3, "buildId" : "0A8C499C987308B3BCE669CF8458579C8CC67D42" }, { "b" : "7FFDCD162000", "elfType" : 3, "buildId" : "54890F8C663DC6CF6219ED314BEF17BC92A67C93" }, { "b" : "7F28228FA000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "E7196EFCC044747C45DA6705CBE6E3EB2CE5DAE2" }, { "b" : "7F2822691000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "DCF10134B91ED2139E3E8C72564668F5CDBA8522" }, { "b" : "7F282224D000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "1649272BE0CA9FA22F082DC86372B6C9959779B0" }, { "b" : "7F2822045000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7F2821E41000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7F2821B38000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7F2821922000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F2821705000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7F282133B000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7F2822B15000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" }, { "b" : "7F2821129000", "path" : "/lib/x86_64-linux-gnu/libnss_files.so.2", "elfType" : 3, "buildId" : "D8E2912D8CEAA52AAD781D76526BF205CF746752" }, { "b" : "7F2820F22000", "path" : "/lib/x86_64-linux-gnu/libnss_dns.so.2", "elfType" : 3, "buildId" : "D6921D29D0C848A24379504B2C6E6D45BB26D416" } ] }}
 mongos(_ZN5mongo15printStackTraceERSo+0x41) [0xbe3ab8edd1]
 mongos(+0x10C7FE9) [0xbe3ab8dfe9]
 mongos(+0x10C84CD) [0xbe3ab8e4cd]
 libpthread.so.0(+0x11390) [0x7f2821716390]
 libc.so.6(gsignal+0x38) [0x7f2821370428]
 libc.so.6(abort+0x16A) [0x7f282137202a]
 mongos(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0xbe39fc6013]
 mongos(+0xB9B0B4) [0xbe3a6610b4]
 mongos(+0x11D62B0) [0xbe3ac9c2b0]
 libpthread.so.0(+0x76BA) [0x7f282170c6ba]
 libc.so.6(clone+0x6D) [0x7f28214423dd]
-----  END BACKTRACE  -----

2017-12-15T23:17:00.272+0300 E -        [conn19] Assertion: 10334:BSONObj size: 16846689 (0x1010F61) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "test2" src/mongo/bson/bsonobj.cpp 101
2017-12-15T23:17:00.275+0300 F -        [conn19] Invariant failure _targeted.empty() src/mongo/s/write_ops/batch_write_op.cpp 227
2017-12-15T23:17:00.275+0300 F -        [conn19]
 
***aborting after invariant() failure
 
 
2017-12-15T23:17:00.285+0300 F -        [conn19] Got signal: 6 (Aborted).
 
 0x8cf24b9dd1 0x8cf24b8fe9 0x8cf24b94cd 0x7f011f3f0390 0x7f011f04a428 0x7f011f04c02a 0x8cf18f0d54 0x8cf1a2c969 0x8cf1a35154 0x8cf1a3fac6 0x8cf1a01770 0x8cf1e3036f 0x8cf1a1ff25 0x8cf1a20863 0x8cf1a20f49 0x8cf1942f55 0x8cf195f52a 0x8cf195b0d7 0x8cf195e321 0x8cf1dafa02 0x8cf1959f40 0x8cf195c482 0x8cf195cd7b 0x8cf195b15d 0x8cf195e321 0x8cf1daff65 0x8cf236a264 0x7f011f3e66ba 0x7f011f11c3dd
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"8CF13F1000","o":"10C8DD1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"8CF13F1000","o":"10C7FE9"},{"b":"8CF13F1000","o":"10C84CD"},{"b":"7F011F3DF000","o":"11390"},{"b":"7F011F015000","o":"35428","s":"gsignal"},{"b":"7F011F015000","o":"3702A","s":"abort"},{"b":"8CF13F1000","o":"4FFD54","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},{"b":"8CF13F1000","o":"63B969"},{"b":"8CF13F1000","o":"644154","s":"_ZN5mongo14BatchWriteExec12executeBatchEPNS_16OperationContextERNS_10NSTargeterERKNS_21BatchedCommandRequestEPNS_22BatchedCommandResponseEPNS_19BatchWriteExecStatsE"},{"b":"8CF13F1000","o":"64EAC6","s":"_ZN5mongo13ClusterWriter5writeEPNS_16OperationContextERKNS_21BatchedCommandRequestEPNS_19BatchWriteExecStatsEPNS_22BatchedCommandResponseE"},{"b":"8CF13F1000","o":"610770"},{"b":"8CF13F1000","o":"A3F36F","s":"_ZN5mongo7Command9publicRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE"},{"b":"8CF13F1000","o":"62EF25"},{"b":"8CF13F1000","o":"62F863"},{"b":"8CF13F1000","o":"62FF49","s":"_ZN5mongo8Strategy13clientCommandEPNS_16OperationContextERKNS_7MessageE"},{"b":"8CF13F1000","o":"551F55","s":"_ZN5mongo23ServiceEntryPointMongos13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"8CF13F1000","o":"56E52A","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"8CF13F1000","o":"56A0D7","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"8CF13F1000","o":"56D321"},{"b":"8CF13F1000","o":"9BEA02","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsE"},{"b":"8CF13F1000","o":"568F40","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS0_9OwnershipE"},{"b":"8CF13F1000","o":"56B482","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"8CF13F1000","o":"56BD7B","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"8CF13F1000","o":"56A15D","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"8CF13F1000","o":"56D321"},{"b":"8CF13F1000","o":"9BEF65"},{"b":"8CF13F1000","o":"F79264"},{"b":"7F011F3DF000","o":"76BA"},{"b":"7F011F015000","o":"1073DD","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.1-rc0", "gitVersion" : "7ffeba9c365fdd3d1eb47ea841b0882a2f8fead7", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-93-generic", "version" : "#116-Ubuntu SMP Fri Aug 11 21:17:51 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "8CF13F1000", "elfType" : 3, "buildId" : "0A8C499C987308B3BCE669CF8458579C8CC67D42" }, { "b" : "7FFC3E4CD000", "elfType" : 3, "buildId" : "54890F8C663DC6CF6219ED314BEF17BC92A67C93" }, { "b" : "7F01205D4000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "E7196EFCC044747C45DA6705CBE6E3EB2CE5DAE2" }, { "b" : "7F012036B000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "DCF10134B91ED2139E3E8C72564668F5CDBA8522" }, { "b" : "7F011FF27000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "1649272BE0CA9FA22F082DC86372B6C9959779B0" }, { "b" : "7F011FD1F000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7F011FB1B000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7F011F812000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7F011F5FC000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F011F3DF000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7F011F015000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7F01207EF000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" }, { "b" : "7F011EE03000", "path" : "/lib/x86_64-linux-gnu/libnss_files.so.2", "elfType" : 3, "buildId" : "D8E2912D8CEAA52AAD781D76526BF205CF746752" }, { "b" : "7F011EBFC000", "path" : "/lib/x86_64-linux-gnu/libnss_dns.so.2", "elfType" : 3, "buildId" : "D6921D29D0C848A24379504B2C6E6D45BB26D416" } ] }}
 mongos(_ZN5mongo15printStackTraceERSo+0x41) [0x8cf24b9dd1]
 mongos(+0x10C7FE9) [0x8cf24b8fe9]
 mongos(+0x10C84CD) [0x8cf24b94cd]
 libpthread.so.0(+0x11390) [0x7f011f3f0390]
 libc.so.6(gsignal+0x38) [0x7f011f04a428]
 libc.so.6(abort+0x16A) [0x7f011f04c02a]
 mongos(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0x0) [0x8cf18f0d54]
 mongos(+0x63B969) [0x8cf1a2c969]
 mongos(_ZN5mongo14BatchWriteExec12executeBatchEPNS_16OperationContextERNS_10NSTargeterERKNS_21BatchedCommandRequestEPNS_22BatchedCommandResponseEPNS_19BatchWriteExecStatsE+0x23B4) [0x8cf1a35154]
 mongos(_ZN5mongo13ClusterWriter5writeEPNS_16OperationContextERKNS_21BatchedCommandRequestEPNS_19BatchWriteExecStatsEPNS_22BatchedCommandResponseE+0x506) [0x8cf1a3fac6]
 mongos(+0x610770) [0x8cf1a01770]
 mongos(_ZN5mongo7Command9publicRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE+0x1F) [0x8cf1e3036f]
 mongos(+0x62EF25) [0x8cf1a1ff25]
 mongos(+0x62F863) [0x8cf1a20863]
 mongos(_ZN5mongo8Strategy13clientCommandEPNS_16OperationContextERKNS_7MessageE+0x59) [0x8cf1a20f49]
 mongos(_ZN5mongo23ServiceEntryPointMongos13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x5B5) [0x8cf1942f55]
 mongos(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xBA) [0x8cf195f52a]
 mongos(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x97) [0x8cf195b0d7]
 mongos(+0x56D321) [0x8cf195e321]
 mongos(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsE+0x1A2) [0x8cf1dafa02]
 mongos(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS0_9OwnershipE+0x150) [0x8cf1959f40]
 mongos(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xAF2) [0x8cf195c482]
 mongos(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x23B) [0x8cf195cd7b]
 mongos(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x11D) [0x8cf195b15d]
 mongos(+0x56D321) [0x8cf195e321]
 mongos(+0x9BEF65) [0x8cf1daff65]
 mongos(+0xF79264) [0x8cf236a264]
 libpthread.so.0(+0x76BA) [0x7f011f3e66ba]
 libc.so.6(clone+0x6D) [0x7f011f11c3dd]
-----  END BACKTRACE  -----



 Comments   
Comment by Scott Glajch [ 23/Aug/19 ]

Thanks Kaloian, yes it was a single update operation, and I don't have the full exact object being updated, but I have the current version of the object that we've trimmed.  I can confirm that the _id was not long.

I've logged a new bug here https://jira.mongodb.org/browse/SERVER-43021

Comment by Kaloian Manassiev [ 12/Aug/19 ]

Hi glajchs,

I can't see any new attachments, but since this ticket has already been released, would it be possible to create a new one and include the crash stack from the update operation?

Would it be possible to also include the update operation, which you performed? Specifically - was it update to a single document or multiple? Because update to a single document, no matter how large should only be returning the _id of that document and should not be able to exceed the response size (unless the size of the _id is extremely large).

Best regards,
-Kal.

Comment by Scott Glajch [ 12/Aug/19 ]

We have also just seen this (or something like this) on mongo version 3.6.6.  The only thing that seems different is the fact that I'm pretty sure we weren't doing a bulk insert, but rather an update.  What happened was we had an object that was approaching 16MB (when I checked the existing DB object on disk it was 15.99MB), and when we went to update it with just a bit more data, it crashed our first mongos server.  Then when the cache writer in our application retried the update 45 seconds later, it crashed our second mongos server.  I've attached both stack traces.

Obviously we don't want to be running with DB objects at or close to 16MB, so we fixed the object in question to not be as big, but even though this isn't something we have happening all the time, it does happen occasionally and we expect to need to run our production servers with the ability for 16MB objects to gracefully fail to save in the future.

Our version is technically 3.6.6-evg1, which is a custom build we have branched directly off of 3.6.6, which you can find here https://github.com/evergage/mongo/commits/v3.6.6-evg1.  The only difference is the last 3 commits you see there which just quiets some extra verbose metadata logging that was eating basically infinite log entries and we had to silence in order to run this in production.  Since the changes are so minor, hopefully that means that the stack trace line numbers and such are still usable for you.  Since then that bug (https://jira.mongodb.org/browse/SERVER-30841?filter=21888) has been fixed in 3.6.8, and assuming that it silenced all the things we silenced in our custom build (3 different files), then we might be able to get off of running a custom build in the future.

 

Please let me know if you'd rather we log a new bug for this if you deem it to be a separate issue, though I'm not sure we'll be able to reproduce the issue again (at least not at will).

Comment by Kaloian Manassiev [ 11/Jan/19 ]

Hi randar,

Based on when this issue was fixed (3.6.1) we do not expect it to be present in 3.6.5.

If you think you are experiencing similar crash in 3.5.6 can you please file a new SERVER ticket, specify the exact version, attach the mongos logs and (if possible) the repro steps you used?

Thanks in advance.

Best regards,
-Kal.

Comment by Randar Puust [ 11/Jan/19 ]

We are currently running Mongo version v3.6.5. It feels like we are having this exact same issue. I'm not clear if there was an attempt to fix it in 3.6.1 and it wasn't correctly fixed, or did it regress in 3.7.0 and finally fixed in 3.7.1. It's having a fairly big affect on us and we are trying to figure out if we should escalate an upgrade to Mongo 4.0. Do you think this issue was still there in 3.6.5?

Comment by Yuriy [X] [ 19/Dec/17 ]

Tested it from sources. All is ok! Thanks!

Comment by Githook User [ 18/Dec/17 ]

Author:

{'name': 'Kaloian Manassiev', 'email': 'kaloian.manassiev@mongodb.com', 'username': 'kaloianm'}

Message: SERVER-32372 Ensure per-shard write request size doesn't exceed the max allowed BSON size

(cherry picked from commit 85e1ed33ef2fc83e870124441eee7e036b8118a4)
Branch: v3.6
https://github.com/mongodb/mongo/commit/acf23ecc5b897dd2b8fcd973d6a765fea5557caf

Comment by Githook User [ 18/Dec/17 ]

Author:

{'name': 'Kaloian Manassiev', 'email': 'kaloian.manassiev@mongodb.com', 'username': 'kaloianm'}

Message: SERVER-32372 Ensure per-shard write request size doesn't exceed the max allowed BSON size
Branch: master
https://github.com/mongodb/mongo/commit/85e1ed33ef2fc83e870124441eee7e036b8118a4

Comment by Kaloian Manassiev [ 18/Dec/17 ]

Hi Ubus,

Thank you very much for your report and for your detailed investigation. We have figured out the cause for this problem and it is due to incorrect calculation of the resulting per-shard request size, which causes the maximum BSON size of 16MB to be exceeded.

We have a fix out for code review.

In the mean time in order to get unblocked, you can lower the size of the batches you are using.

Best regards,
-Kal.

Comment by Yuriy [X] [ 18/Dec/17 ]

Just tested it on 3.7.0-39-g4edcb81 mongos. Still same crash.

2017-12-18T15:23:34.348+0300 E -        [NetworkInterfaceASIO-TaskExecutorPool-1-0] Assertion: 10334:BSONObj size: 16793804 (0x10040CC) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "test3" src/mongo/bson/bsonobj.cpp 101
2017-12-18T15:23:34.349+0300 F ASIO     [NetworkInterfaceASIO-TaskExecutorPool-1-0] Uncaught exception in NetworkInterfaceASIO IO worker thread of type: Location10334: BSONObj size: 16793804 (0x10040CC) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "test3"
2017-12-18T15:23:34.349+0300 F -        [NetworkInterfaceASIO-TaskExecutorPool-1-0] Fatal Assertion 28820 at src/mongo/executor/network_interface_asio.cpp 169
2017-12-18T15:23:34.349+0300 F -        [NetworkInterfaceASIO-TaskExecutorPool-1-0]
 
***aborting after fassert() failure
 
 
2017-12-18T15:23:34.352+0300 F -        [NetworkInterfaceASIO-TaskExecutorPool-1-0] Got signal: 6 (Aborted).
 
 0x51f20c301 0x51f20b519 0x51f20b9fd 0x7f1f50507390 0x7f1f50161428 0x7f1f5016302a 0x51e60383d 0x51ecd98c4 0x51f3197e0 0x7f1f504fd6ba 0x7f1f502333dd
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"51E0FC000","o":"1110301","s":"_ZN5mongo15printStackTraceERSo"},{"b":"51E0FC000","o":"110F519"},{"b":"51E0FC000","o":"110F9FD"},{"b":"7F1F504F6000","o":"11390"},{"b":"7F1F5012C000","o":"35428","s":"gsignal"},{"b":"7F1F5012C000","o":"3702A","s":"abort"},{"b":"51E0FC000","o":"50783D","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"51E0FC000","o":"BDD8C4"},{"b":"51E0FC000","o":"121D7E0"},{"b":"7F1F504F6000","o":"76BA"},{"b":"7F1F5012C000","o":"1073DD","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.7.0-39-g4edcb81", "gitVersion" : "4edcb8195a1ad4ffa20b9840c323913894376cdb", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-93-generic", "version" : "#116-Ubuntu SMP Fri Aug 11 21:17:51 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "51E0FC000", "elfType" : 3, "buildId" : "21BBB1C7311CE9910D0FF0F07B756F76A17EF057" }, { "b" : "7FFD4939E000", "elfType" : 3, "buildId" : "54890F8C663DC6CF6219ED314BEF17BC92A67C93" }, { "b" : "7F1F516EB000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "E7196EFCC044747C45DA6705CBE6E3EB2CE5DAE2" }, { "b" : "7F1F51482000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "DCF10134B91ED2139E3E8C72564668F5CDBA8522" }, { "b" : "7F1F5103E000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "1649272BE0CA9FA22F082DC86372B6C9959779B0" }, { "b" : "7F1F50E36000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7F1F50C32000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7F1F50929000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7F1F50713000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F1F504F6000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7F1F5012C000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7F1F51906000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" }, { "b" : "7F1F4FF1A000", "path" : "/lib/x86_64-linux-gnu/libnss_files.so.2", "elfType" : 3, "buildId" : "D8E2912D8CEAA52AAD781D76526BF205CF746752" }, { "b" : "7F1F4FD13000", "path" : "/lib/x86_64-linux-gnu/libnss_dns.so.2", "elfType" : 3, "buildId" : "D6921D29D0C848A24379504B2C6E6D45BB26D416" } ] }}
 mongos(_ZN5mongo15printStackTraceERSo+0x41) [0x51f20c301]
 mongos(+0x110F519) [0x51f20b519]
 mongos(+0x110F9FD) [0x51f20b9fd]
 libpthread.so.0(+0x11390) [0x7f1f50507390]
 libc.so.6(gsignal+0x38) [0x7f1f50161428]
 libc.so.6(abort+0x16A) [0x7f1f5016302a]
 mongos(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x51e60383d]
 mongos(+0xBDD8C4) [0x51ecd98c4]
 mongos(+0x121D7E0) [0x51f3197e0]
 libpthread.so.0(+0x76BA) [0x7f1f504fd6ba]
 libc.so.6(clone+0x6D) [0x7f1f502333dd]
-----  END BACKTRACE  -----

Comment by Ramon Fernandez Marina [ 16/Dec/17 ]

Thanks for the detailed report Ubus. The stack traces you provided look as follows after running them through the symbolizer:

 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/util/stacktrace_posix.cpp:172:0: mongo::printStackTrace(std::ostream&)
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/util/signal_handlers_synchronous.cpp:180:0: mongo::(anonymous namespace)::printSignalAndBacktrace(int)
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/util/signal_handlers_synchronous.cpp:237:0: mongo::(anonymous namespace)::abruptQuit(int)
 ??:0:0: ??
 /build/glibc-bfm8X4/glibc-2.23/signal/../sysdeps/unix/sysv/linux/raise.c:54:0: gsignal
 /build/glibc-bfm8X4/glibc-2.23/stdlib/abort.c:89:0: abort
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/util/assert_util.cpp:148:0: mongo::fassertFailedWithLocation(int, char const*, unsigned int)
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/executor/network_interface_asio.cpp:169:0: mongo::executor::NetworkInterfaceASIO::startup()::{lambda()#1}::operator()() const [clone .constprop.370]
 /data/mci/c5d8f4120437056d97d16d15cf65c6ac/toolchain-builder/build-gcc-v2.sh-qR3/x86_64-mongodb-linux/libstdc++-v3/src/c++11/../../../../../gcc-5.4.0/libstdc++-v3/src/c++11/thread.cc:84:0: execute_native_thread_routine
 ??:0:0: start_thread
 /build/glibc-bfm8X4/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:109:0: clone

Not much there. The second one has more information:

 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/util/stacktrace_posix.cpp:172:0: mongo::printStackTrace(std::ostream&)
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/util/signal_handlers_synchronous.cpp:180:0: mongo::(anonymous namespace)::printSignalAndBacktrace(int)
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/util/signal_handlers_synchronous.cpp:237:0: mongo::(anonymous namespace)::abruptQuit(int)
 ??:0:0: ??
 /build/glibc-bfm8X4/glibc-2.23/signal/../sysdeps/unix/sysv/linux/raise.c:54:0: gsignal
 /build/glibc-bfm8X4/glibc-2.23/stdlib/abort.c:89:0: abort
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/util/assert_util.cpp:130:0: mongo::invariantFailed(char const*, char const*, unsigned int)
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/s/write_ops/batch_write_op.cpp:227:0: mongo::BatchWriteOp::~BatchWriteOp()
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/s/write_ops/batch_write_exec.cpp:94:0: mongo::BatchWriteExec::executeBatch(mongo::OperationContext*, mongo::NSTargeter&, mongo::BatchedCommandRequest const&, mongo::BatchedCommandResponse*, mongo::BatchWriteExecStats*)
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/s/commands/cluster_write.cpp:234:0: mongo::ClusterWriter::write(mongo::OperationContext*, mongo::BatchedCommandRequest const&, mongo::BatchWriteExecStats*, mongo::BatchedCommandResponse*)
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/s/commands/cluster_write_cmd.cpp:204:0: mongo::(anonymous namespace)::ClusterWriteCmd::enhancedRun(mongo::OperationContext*, mongo::OpMsgRequest const&, mongo::BSONObjBuilder&)
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/db/commands.cpp:354:0: mongo::Command::publicRun(mongo::OperationContext*, mongo::OpMsgRequest const&, mongo::BSONObjBuilder&)
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/s/commands/strategy.cpp:226:0: execCommandClient
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/s/commands/strategy.cpp:265:0: mongo::(anonymous namespace)::runCommand(mongo::OperationContext*, mongo::OpMsgRequest const&, mongo::BSONObjBuilder&&) [clone .constprop.537]
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/s/commands/strategy.cpp:423:0: mongo::Strategy::clientCommand(mongo::OperationContext*, mongo::Message const&)::{lambda()#1}::operator()() const
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/s/commands/strategy.cpp:434:0: mongo::Strategy::clientCommand(mongo::OperationContext*, mongo::Message const&)
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/s/service_entry_point_mongos.cpp:92:0: mongo::ServiceEntryPointMongos::handleRequest(mongo::OperationContext*, mongo::Message const&)
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/transport/service_state_machine.cpp:360:0: mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard)
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/transport/service_state_machine.cpp:419:0: mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/transport/service_state_machine.cpp:456:0: operator()
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1871:0: std::_Function_handler<void (), mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::ServiceStateMachine::Ownership)::{lambda()#1}>::_M_invoke(std::_Any_data const&)
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:2267:0: std::function<void ()>::operator()() const
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/transport/service_executor_synchronous.cpp:118:0: mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags)
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/transport/service_state_machine.cpp:459:0: mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::ServiceStateMachine::Ownership)
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/transport/service_state_machine.cpp:291:0: mongo::ServiceStateMachine::_sourceCallback(mongo::Status)
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/transport/service_state_machine.cpp:250:0: mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/transport/service_state_machine.cpp:416:0: mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/transport/service_state_machine.cpp:456:0: operator()
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1871:0: std::_Function_handler<void (), mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::ServiceStateMachine::Ownership)::{lambda()#1}>::_M_invoke(std::_Any_data const&)
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:2267:0: std::function<void ()>::operator()() const
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/transport/service_executor_synchronous.cpp:135:0: operator()
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1871:0: std::_Function_handler<void (), mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags)::{lambda()#1}>::_M_invoke(std::_Any_data const&)
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:2267:0: std::function<void ()>::operator()() const
 /data/mci/fc5afc27295e1a62d0669b196bd6f87a/src/src/mongo/transport/service_entry_point_utils.cpp:55:0: mongo::(anonymous namespace)::runFunc(void*)
 ??:0:0: start_thread
 /build/glibc-bfm8X4/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:109:0: clone

Comment by Yuriy [X] [ 15/Dec/17 ]

Ops, stacktraces in description got badly formatted shape.

Generated at Thu Feb 08 04:30:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.