Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-32372

Mongos crashes on bulk insert where size is slightly bigger than maxBsonObjectSize

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 3.6.1, 3.7.1
    • Affects Version/s: 3.6.0
    • Component/s: Sharding
    • Fully Compatible
    • ALL
    • v3.6
    • Hide

      1. Mongo sharded cluster 3.6.0 with 2 replicas.
      2. Launch mongos.crash.snippet.java against mongos.
      In snippet you can check number of inserts (NUMBER_OF_INSERTS), if it equals 87522 or higher mongos crashes each time you try to insertMany.
      You get different stacktraces depends on number of inserts.

      Show
      1. Mongo sharded cluster 3.6.0 with 2 replicas. 2. Launch mongos.crash.snippet.java against mongos. In snippet you can check number of inserts (NUMBER_OF_INSERTS), if it equals 87522 or higher mongos crashes each time you try to insertMany. You get different stacktraces depends on number of inserts.
    • Sharding 2018-01-01

      We upgraded mongo sharded cluster from 3.4.10 to 3.6.0 version and mongoses started to crash from insertMany operation, if its size is bigger than maxBsonObjectSize.
      It seems that it begins because of upping maxWriteBatchSize in 3.6.0 to 100000.
      Got two different stacktraces:

      2017-12-15T23:05:47.701+0300 E -        [NetworkInterfaceASIO-TaskExecutorPool-8-0] Assertion: 10334:BSONObj size: 16793791 (0x10040BF) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "test2" src/mongo/bson/bsonobj.cpp 101
      2017-12-15T23:05:47.702+0300 F ASIO     [NetworkInterfaceASIO-TaskExecutorPool-8-0] Uncaught exception in NetworkInterfaceASIO IO worker thread of type: Location10334: BSONObj size: 16793791 (0x10040BF) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "test2"
      2017-12-15T23:05:47.702+0300 F -        [NetworkInterfaceASIO-TaskExecutorPool-8-0] Fatal Assertion 28820 at src/mongo/executor/network_interface_asio.cpp 169
      2017-12-15T23:05:47.702+0300 F -        [NetworkInterfaceASIO-TaskExecutorPool-8-0]
      
      ***aborting after fassert() failure
      
      
      2017-12-15T23:05:47.705+0300 F -        [NetworkInterfaceASIO-TaskExecutorPool-8-0] Got signal: 6 (Aborted).
      
       0xbe3ab8edd1 0xbe3ab8dfe9 0xbe3ab8e4cd 0x7f2821716390 0x7f2821370428 0x7f282137202a 0xbe39fc6013 0xbe3a6610b4 0xbe3ac9c2b0 0x7f282170c6ba 0x7f28214423dd
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"BE39AC6000","o":"10C8DD1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"BE39AC6000","o":"10C7FE9"},{"b":"BE39AC6000","o":"10C84CD"},{"b":"7F2821705000","o":"11390"},{"b":"7F282133B000","o":"35428","s":"gsignal"},{"b":"7F282133B000","o":"3702A","s":"abort"},{"b":"BE39AC6000","o":"500013","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"BE39AC6000","o":"B9B0B4"},{"b":"BE39AC6000","o":"11D62B0"},{"b":"7F2821705000","o":"76BA"},{"b":"7F282133B000","o":"1073DD","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.1-rc0", "gitVersion" : "7ffeba9c365fdd3d1eb47ea841b0882a2f8fead7", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-93-generic", "version" : "#116-Ubuntu SMP Fri Aug 11 21:17:51 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "BE39AC6000", "elfType" : 3, "buildId" : "0A8C499C987308B3BCE669CF8458579C8CC67D42" }, { "b" : "7FFDCD162000", "elfType" : 3, "buildId" : "54890F8C663DC6CF6219ED314BEF17BC92A67C93" }, { "b" : "7F28228FA000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "E7196EFCC044747C45DA6705CBE6E3EB2CE5DAE2" }, { "b" : "7F2822691000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "DCF10134B91ED2139E3E8C72564668F5CDBA8522" }, { "b" : "7F282224D000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "1649272BE0CA9FA22F082DC86372B6C9959779B0" }, { "b" : "7F2822045000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7F2821E41000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7F2821B38000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7F2821922000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F2821705000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7F282133B000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7F2822B15000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" }, { "b" : "7F2821129000", "path" : "/lib/x86_64-linux-gnu/libnss_files.so.2", "elfType" : 3, "buildId" : "D8E2912D8CEAA52AAD781D76526BF205CF746752" }, { "b" : "7F2820F22000", "path" : "/lib/x86_64-linux-gnu/libnss_dns.so.2", "elfType" : 3, "buildId" : "D6921D29D0C848A24379504B2C6E6D45BB26D416" } ] }}
       mongos(_ZN5mongo15printStackTraceERSo+0x41) [0xbe3ab8edd1]
       mongos(+0x10C7FE9) [0xbe3ab8dfe9]
       mongos(+0x10C84CD) [0xbe3ab8e4cd]
       libpthread.so.0(+0x11390) [0x7f2821716390]
       libc.so.6(gsignal+0x38) [0x7f2821370428]
       libc.so.6(abort+0x16A) [0x7f282137202a]
       mongos(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0xbe39fc6013]
       mongos(+0xB9B0B4) [0xbe3a6610b4]
       mongos(+0x11D62B0) [0xbe3ac9c2b0]
       libpthread.so.0(+0x76BA) [0x7f282170c6ba]
       libc.so.6(clone+0x6D) [0x7f28214423dd]
      -----  END BACKTRACE  -----
      
      2017-12-15T23:17:00.272+0300 E -        [conn19] Assertion: 10334:BSONObj size: 16846689 (0x1010F61) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "test2" src/mongo/bson/bsonobj.cpp 101
      2017-12-15T23:17:00.275+0300 F -        [conn19] Invariant failure _targeted.empty() src/mongo/s/write_ops/batch_write_op.cpp 227
      2017-12-15T23:17:00.275+0300 F -        [conn19]
      
      ***aborting after invariant() failure
      
      
      2017-12-15T23:17:00.285+0300 F -        [conn19] Got signal: 6 (Aborted).
      
       0x8cf24b9dd1 0x8cf24b8fe9 0x8cf24b94cd 0x7f011f3f0390 0x7f011f04a428 0x7f011f04c02a 0x8cf18f0d54 0x8cf1a2c969 0x8cf1a35154 0x8cf1a3fac6 0x8cf1a01770 0x8cf1e3036f 0x8cf1a1ff25 0x8cf1a20863 0x8cf1a20f49 0x8cf1942f55 0x8cf195f52a 0x8cf195b0d7 0x8cf195e321 0x8cf1dafa02 0x8cf1959f40 0x8cf195c482 0x8cf195cd7b 0x8cf195b15d 0x8cf195e321 0x8cf1daff65 0x8cf236a264 0x7f011f3e66ba 0x7f011f11c3dd
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"8CF13F1000","o":"10C8DD1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"8CF13F1000","o":"10C7FE9"},{"b":"8CF13F1000","o":"10C84CD"},{"b":"7F011F3DF000","o":"11390"},{"b":"7F011F015000","o":"35428","s":"gsignal"},{"b":"7F011F015000","o":"3702A","s":"abort"},{"b":"8CF13F1000","o":"4FFD54","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},{"b":"8CF13F1000","o":"63B969"},{"b":"8CF13F1000","o":"644154","s":"_ZN5mongo14BatchWriteExec12executeBatchEPNS_16OperationContextERNS_10NSTargeterERKNS_21BatchedCommandRequestEPNS_22BatchedCommandResponseEPNS_19BatchWriteExecStatsE"},{"b":"8CF13F1000","o":"64EAC6","s":"_ZN5mongo13ClusterWriter5writeEPNS_16OperationContextERKNS_21BatchedCommandRequestEPNS_19BatchWriteExecStatsEPNS_22BatchedCommandResponseE"},{"b":"8CF13F1000","o":"610770"},{"b":"8CF13F1000","o":"A3F36F","s":"_ZN5mongo7Command9publicRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE"},{"b":"8CF13F1000","o":"62EF25"},{"b":"8CF13F1000","o":"62F863"},{"b":"8CF13F1000","o":"62FF49","s":"_ZN5mongo8Strategy13clientCommandEPNS_16OperationContextERKNS_7MessageE"},{"b":"8CF13F1000","o":"551F55","s":"_ZN5mongo23ServiceEntryPointMongos13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"8CF13F1000","o":"56E52A","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"8CF13F1000","o":"56A0D7","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"8CF13F1000","o":"56D321"},{"b":"8CF13F1000","o":"9BEA02","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsE"},{"b":"8CF13F1000","o":"568F40","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS0_9OwnershipE"},{"b":"8CF13F1000","o":"56B482","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"8CF13F1000","o":"56BD7B","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"8CF13F1000","o":"56A15D","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"8CF13F1000","o":"56D321"},{"b":"8CF13F1000","o":"9BEF65"},{"b":"8CF13F1000","o":"F79264"},{"b":"7F011F3DF000","o":"76BA"},{"b":"7F011F015000","o":"1073DD","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.1-rc0", "gitVersion" : "7ffeba9c365fdd3d1eb47ea841b0882a2f8fead7", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-93-generic", "version" : "#116-Ubuntu SMP Fri Aug 11 21:17:51 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "8CF13F1000", "elfType" : 3, "buildId" : "0A8C499C987308B3BCE669CF8458579C8CC67D42" }, { "b" : "7FFC3E4CD000", "elfType" : 3, "buildId" : "54890F8C663DC6CF6219ED314BEF17BC92A67C93" }, { "b" : "7F01205D4000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "E7196EFCC044747C45DA6705CBE6E3EB2CE5DAE2" }, { "b" : "7F012036B000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "DCF10134B91ED2139E3E8C72564668F5CDBA8522" }, { "b" : "7F011FF27000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "1649272BE0CA9FA22F082DC86372B6C9959779B0" }, { "b" : "7F011FD1F000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7F011FB1B000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7F011F812000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7F011F5FC000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F011F3DF000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7F011F015000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7F01207EF000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" }, { "b" : "7F011EE03000", "path" : "/lib/x86_64-linux-gnu/libnss_files.so.2", "elfType" : 3, "buildId" : "D8E2912D8CEAA52AAD781D76526BF205CF746752" }, { "b" : "7F011EBFC000", "path" : "/lib/x86_64-linux-gnu/libnss_dns.so.2", "elfType" : 3, "buildId" : "D6921D29D0C848A24379504B2C6E6D45BB26D416" } ] }}
       mongos(_ZN5mongo15printStackTraceERSo+0x41) [0x8cf24b9dd1]
       mongos(+0x10C7FE9) [0x8cf24b8fe9]
       mongos(+0x10C84CD) [0x8cf24b94cd]
       libpthread.so.0(+0x11390) [0x7f011f3f0390]
       libc.so.6(gsignal+0x38) [0x7f011f04a428]
       libc.so.6(abort+0x16A) [0x7f011f04c02a]
       mongos(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0x0) [0x8cf18f0d54]
       mongos(+0x63B969) [0x8cf1a2c969]
       mongos(_ZN5mongo14BatchWriteExec12executeBatchEPNS_16OperationContextERNS_10NSTargeterERKNS_21BatchedCommandRequestEPNS_22BatchedCommandResponseEPNS_19BatchWriteExecStatsE+0x23B4) [0x8cf1a35154]
       mongos(_ZN5mongo13ClusterWriter5writeEPNS_16OperationContextERKNS_21BatchedCommandRequestEPNS_19BatchWriteExecStatsEPNS_22BatchedCommandResponseE+0x506) [0x8cf1a3fac6]
       mongos(+0x610770) [0x8cf1a01770]
       mongos(_ZN5mongo7Command9publicRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE+0x1F) [0x8cf1e3036f]
       mongos(+0x62EF25) [0x8cf1a1ff25]
       mongos(+0x62F863) [0x8cf1a20863]
       mongos(_ZN5mongo8Strategy13clientCommandEPNS_16OperationContextERKNS_7MessageE+0x59) [0x8cf1a20f49]
       mongos(_ZN5mongo23ServiceEntryPointMongos13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x5B5) [0x8cf1942f55]
       mongos(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xBA) [0x8cf195f52a]
       mongos(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x97) [0x8cf195b0d7]
       mongos(+0x56D321) [0x8cf195e321]
       mongos(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsE+0x1A2) [0x8cf1dafa02]
       mongos(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS0_9OwnershipE+0x150) [0x8cf1959f40]
       mongos(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xAF2) [0x8cf195c482]
       mongos(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x23B) [0x8cf195cd7b]
       mongos(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x11D) [0x8cf195b15d]
       mongos(+0x56D321) [0x8cf195e321]
       mongos(+0x9BEF65) [0x8cf1daff65]
       mongos(+0xF79264) [0x8cf236a264]
       libpthread.so.0(+0x76BA) [0x7f011f3e66ba]
       libc.so.6(clone+0x6D) [0x7f011f11c3dd]
      -----  END BACKTRACE  -----
      

        1. stacktrace.87800.inserts.txt
          8 kB
        2. stacktrace.87522.inserts.txt
          4 kB
        3. mongos.crash.snippet.java
          0.5 kB

            Assignee:
            kaloian.manassiev@mongodb.com Kaloian Manassiev
            Reporter:
            Ubus Yuriy [X]
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: