Uploaded image for project: 'MongoDB Database Tools'
  1. MongoDB Database Tools
  2. TOOLS-1075

Different behaviour between 3.2 and 2.6 on json file import

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • v3.2, v2.6

      Importing this the attached file `posts.json` through mongoimport 2.6 parses each line as individual documents while 3.2 parses the full file as one single document, that in this case causes the operation to fail.

      On 3.2

      mongoimport  -d blog -c posts --drop posts.json                                                                           [11:08:06]
      2016-02-19T11:08:16.565-0500	connected to: localhost
      2016-02-19T11:08:16.565-0500	dropping: blog.posts
      2016-02-19T11:08:17.102-0500	Failed: lost connection to server
      2016-02-19T11:08:17.102-0500	imported 0 documents
      

      On 2.6

      mongoimport -d blog -c posts --drop posts.json                                                                              [11:06:51]
      connected to: 127.0.0.1
      2016-02-19T11:06:55.509-0500 dropping: blog.posts
      2016-02-19T11:06:58.472-0500 		Progress: 17822182/35553830	50%
      2016-02-19T11:06:58.472-0500 			500	166/second
      2016-02-19T11:06:59.953-0500 check 9 1000
      2016-02-19T11:06:59.953-0500 imported 1000 objects
      

      I've also added a slight change to the file by transforming it into a json array (attached file a.json) also get different behaviour:

      On 3.2

      mongoimport --jsonArray -d blog -c posts --drop a.json                                                                                  [11:08:30]
      2016-02-19T11:08:40.167-0500	connected to: localhost
      2016-02-19T11:08:40.167-0500	dropping: blog.posts
      2016-02-19T11:08:40.708-0500	Failed: lost connection to server
      2016-02-19T11:08:40.708-0500	imported 0 documents
      

      On 2.6

      mongoimport --jsonArray -d blog -c posts --drop a.json                                                                                  [11:05:45]
      connected to: 127.0.0.1
      2016-02-19T11:06:47.067-0500 dropping: blog.posts
      2016-02-19T11:06:50.484-0500 		Progress: 21501846/35554833	60%
      2016-02-19T11:06:50.484-0500 			600	200/second
      2016-02-19T11:06:51.452-0500 check 9 1000
      2016-02-19T11:06:51.452-0500 imported 1000 objects
      

      More importantly than that this is causing a massert log entry on the server:

      2016-02-19T11:08:40.701-0500 I -        [conn22] Assertion: 10334:BSONObj size: 33609065 (0x200D569) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "posts"
      2016-02-19T11:08:40.706-0500 I CONTROL  [conn22]
       0x103b30119 0x103ae6523 0x103ad6195 0x103ad5ffa 0x10327b1b8 0x103471360 0x1032c9d5d 0x10355499f 0x10355136a 0x10325864f 0x103af289e 0x103af2041 0x7fff8cac2c13 0x7fff8cac2b90 0x7fff8cac0375
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"10324E000","o":"8E2119","s":"_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE"},{"b":"10324E000","o":"898523","s":"_ZN5mongo10logContextEPKc"},{"b":"10324E000","o":"888195","s":"_ZN5mongo11msgassertedEiPKc"},{"b":"10324E000","o":"887FFA","s":"_ZN5mongo11msgassertedEiRKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEE"},{"b":"10324E000","o":"2D1B8","s":"_ZNK5mongo7BSONObj14_assertInvalidEv"},{"b":"10324E000","o":"223360","s":"_ZN5mongo9DbMessage9nextJsObjEv"},{"b":"10324E000","o":"7BD5D","s":"_ZN5mongo12QueryMessageC2ERNS_9DbMessageE"},{"b":"10324E000","o":"30699F","s":"_ZN5mongoL15receivedCommandEPNS_16OperationContextERKNS_15NamespaceStringERNS_6ClientERNS_10DbResponseERNS_7MessageE"},{"b":"10324E000","o":"30336A","s":"_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE"},{"b":"10324E000","o":"A64F","s":"_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE"},{"b":"10324E000","o":"8A489E","s":"_ZN5mongo17PortMessageServer17handleIncomingMsgEPv"},{"b":"10324E000","o":"8A4041","s":"_ZNSt3__114__thread_proxyINS_5tupleIJNS_6__bindIPFPvS3_EJPN5mongo12_GLOBAL__N_124MessagingPortWithHandlerEEEEEEEEES3_S3_"},{"b":"7FFF8CABF000","o":"3C13","s":"_pthread_body"},{"b":"7FFF8CABF000","o":"3B90","s":"_pthread_body"},{"b":"7FFF8CABF000","o":"1375","s":"thread_start"}],"processInfo":{ "mongodbVersion" : "3.2.3", "gitVersion" : "b326ba837cf6f49d65c2f85e1b70f6f31ece7937", "compiledModules" : [], "uname" : { "sysname" : "Darwin", "release" : "15.3.0", "version" : "Darwin Kernel Version 15.3.0: Thu Dec 10 18:40:58 PST 2015; root:xnu-3248.30.4~1/RELEASE_X86_64", "machine" : "x86_64" }, "somap" : [ { "path" : "/Users/norberto/sandbox/essentials_training_test/mongodb-osx-x86_64-3.2.3/bin/./mongod", "machType" : 2, "b" : "10324E000", "vmaddr" : "100000000", "buildId" : "8A20E1DCBF543EE3AE6F68D820D4C1B6" }, { "path" : "/usr/lib/libSystem.B.dylib", "machType" : 6, "b" : "7FFF8BF82000", "vmaddr" : "7FFF8B7AE000", "buildId" : "5A4257EF31453BB387A40D2404A9462D" }, { "path" : "/usr/lib/libc++.1.dylib", "machType" : 6, "b" : "7FFF90C83000", "vmaddr" : "7FFF904AF000", "buildId" : "8FC3D139805534989AC56467CB7F4D14" }, { "path" : "/usr/lib/system/libcache.dylib", "machType" : 6, "b" : "7FFF92991000", "vmaddr" : "7FFF921BD000", "buildId" : "6B245C0AF3EA383BA5425B0D0456A41B" }, { "path" : "/usr/lib/system/libcommonCrypto.dylib", "machType" : 6, "b" : "7FFF81795000", "vmaddr" : "7FFF80FC1000", "buildId" : "766BC3F541F33315BABC72718A98EA92" }, { "path" : "/usr/lib/system/libcompiler_rt.dylib", "machType" : 6, "b" : "7FFF862CA000", "vmaddr" : "7FFF85AF6000", "buildId" : "D3C4AB4023B43BC68C385B8758D14E80" }, { "path" : "/usr/lib/system/libcopyfile.dylib", "machType" : 6, "b" : "7FFF887B8000", "vmaddr" : "7FFF87FE4000", "buildId" : "F51332690B22388CA57C079667B6291E" }, { "path" : "/usr/lib/system/libcorecrypto.dylib", "machType" : 6, "b" : "7FFF8D8D4000", "vmaddr" : "7FFF8D100000", "buildId" : "C6BD205F4ECE37EEBCABA76F39CDCFFA" }, { "path" : "/usr/lib/system/libdispatch.dylib", "machType" : 6, "b" : "7FFF90269000", "vmaddr" : "7FFF8FA95000", "buildId" : "324C91892AF33356847F6F4CE1C6E901" }, { "path" : "/usr/lib/system/libdyld.dylib", "machType" : 6, "b" : "7FFF946D5000", "vmaddr" : "7FFF93F01000", "buildId" : "AA629043C6F632FE8007E3478E99ACA7" }, { "path" : "/usr/lib/system/libkeymgr.dylib", "machType" : 6, "b" : "7FFF81781000", "vmaddr" : "7FFF80FAD000", "buildId" : "09397E0160663179A50C2CE666FDA929" }, { "path" : "/usr/lib/system/liblaunch.dylib", "machType" : 6, "b" : "7FFF81EFF000", "vmaddr" : "7FFF8172B000", "buildId" : "EDF719D6D2BB38DD8C944272BEFDA2CD" }, { "path" : "/usr/lib/system/libmacho.dylib", "machType" : 6, "b" : "7FFF816A6000", "vmaddr" : "7FFF80ED2000", "buildId" : "CB745E1F48853F96B38B2093DF488FD5" }, { "path" : "/usr/lib/system/libquarantine.dylib", "machType" : 6, "b" : "7FFF8C95E000", "vmaddr" : "7FFF8C18A000", "buildId" : "163CF63A74553D1FAE578C4475A9204C" }, { "path" : "/usr/lib/system/libremovefile.dylib", "machType" : 6, "b" : "7FFF887AC000", "vmaddr" : "7FFF87FD8000", "buildId" : "B8D1A5FCCFD53AAB8A1014DDC129710A" }, { "path" : "/usr/lib/system/libsystem_asl.dylib", "machType" : 6, "b" : "7FFF85578000", "vmaddr" : "7FFF84DA4000", "buildId" : "9B500E4EE462321E828E5524DC984C1B" }, { "path" : "/usr/lib/system/libsystem_blocks.dylib", "machType" : 6, "b" : "7FFF85BC6000", "vmaddr" : "7FFF853F2000", "buildId" : "49D423297DE9341392C3A473A7E9CF35" }, { "path" : "/usr/lib/system/libsystem_c.dylib", "machType" : 6, "b" : "7FFF901D3000", "vmaddr" : "7FFF8F9FF000", "buildId" : "EAB38A6C86713B13B50090EC1B912063" }, { "path" : "/usr/lib/system/libsystem_configuration.dylib", "machType" : 6, "b" : "7FFF90E1B000", "vmaddr" : "7FFF90647000", "buildId" : "5FD7907036CC3D02BEA7BB5D2AE97D5D" }, { "path" : "/usr/lib/system/libsystem_coreservices.dylib", "machType" : 6, "b" : "7FFF909A0000", "vmaddr" : "7FFF901CC000", "buildId" : "1B3F5AFCFFCD3ECB8B9A5538366FB20D" }, { "path" : "/usr/lib/system/libsystem_coretls.dylib", "machType" : 6, "b" : "7FFF8ED78000", "vmaddr" : "7FFF8E5A4000", "buildId" : "75C97D880A633093AE83DE33EB7405CE" }, { "path" : "/usr/lib/system/libsystem_dnssd.dylib", "machType" : 6, "b" : "7FFF8DB17000", "vmaddr" : "7FFF8D343000", "buildId" : "945B5FB1DA913D45A961A8FAD53C1E7E" }, { "path" : "/usr/lib/system/libsystem_info.dylib", "machType" : 6, "b" : "7FFF8DB20000", "vmaddr" : "7FFF8D34C000", "buildId" : "6513635B4ADE3B45BF63ED7AC565B0C9" }, { "path" : "/usr/lib/system/libsystem_kernel.dylib", "machType" : 6, "b" : "7FFF87EDF000", "vmaddr" : "7FFF8770B000", "buildId" : "9CEB6C3B1CAF3C32A9FD93BC72CBCEA1" }, { "path" : "/usr/lib/system/libsystem_m.dylib", "machType" : 6, "b" : "7FFF82487000", "vmaddr" : "7FFF81CB3000", "buildId" : "26655445CA97321EB221801CB378D1AA" }, { "path" : "/usr/lib/system/libsystem_malloc.dylib", "machType" : 6, "b" : "7FFF830DD000", "vmaddr" : "7FFF82909000", "buildId" : "9EECAB18F02534C48E327EFFA6720EFC" }, { "path" : "/usr/lib/system/libsystem_network.dylib", "machType" : 6, "b" : "7FFF85C30000", "vmaddr" : "7FFF8545C000", "buildId" : "865FE79AA22D3733A14FFC7B37F3AECD" }, { "path" : "/usr/lib/system/libsystem_networkextension.dylib", "machType" : 6, "b" : "7FFF8E14C000", "vmaddr" : "7FFF8D978000", "buildId" : "DC8A102ABF0231A4891465C34DF6B592" }, { "path" : "/usr/lib/system/libsystem_notify.dylib", "machType" : 6, "b" : "7FFF862FD000", "vmaddr" : "7FFF85B29000", "buildId" : "243FADE1255A3B788033F336CD64B817" }, { "path" : "/usr/lib/system/libsystem_platform.dylib", "machType" : 6, "b" : "7FFF85D1F000", "vmaddr" : "7FFF8554B000", "buildId" : "D3A27E107F083603ACC87A92B2C04BAB" }, { "path" : "/usr/lib/system/libsystem_pthread.dylib", "machType" : 6, "b" : "7FFF8CABF000", "vmaddr" : "7FFF8C2EB000", "buildId" : "327CECD0B88131538FCC4FD4818B7F16" }, { "path" : "/usr/lib/system/libsystem_sandbox.dylib", "machType" : 6, "b" : "7FFF887C1000", "vmaddr" : "7FFF87FED000", "buildId" : "3E0036AFFC6433528DA46B550C2C2562" }, { "path" : "/usr/lib/system/libsystem_secinit.dylib", "machType" : 6, "b" : "7FFF85D7C000", "vmaddr" : "7FFF855A8000", "buildId" : "FD6ECF2C148932CA981B9045B5EB1FAA" }, { "path" : "/usr/lib/system/libsystem_trace.dylib", "machType" : 6, "b" : "7FFF81750000", "vmaddr" : "7FFF80F7C000", "buildId" : "B485369FE3A1319E998C89AAF606079E" }, { "path" : "/usr/lib/system/libunc.dylib", "machType" : 6, "b" : "7FFF81ACE000", "vmaddr" : "7FFF812FA000", "buildId" : "1D0F8265F0263CBD93D3F8DF14FFCE68" }, { "path" : "/usr/lib/system/libunwind.dylib", "machType" : 6, "b" : "7FFF92E07000", "vmaddr" : "7FFF92633000", "buildId" : "124E0F0523503774A32C7F5BF38EDE73" }, { "path" : "/usr/lib/system/libxpc.dylib", "machType" : 6, "b" : "7FFF808B4000", "vmaddr" : "7FFF800E0000", "buildId" : "61AB46109304354C9E9BD57198AE9866" }, { "path" : "/usr/lib/libobjc.A.dylib", "machType" : 6, "b" : "7FFF9137C000", "vmaddr" : "7FFF90BA8000", "buildId" : "9F45830DF1D53CDF94611A5477ED7D1E" }, { "path" : "/usr/lib/libauto.dylib", "machType" : 6, "b" : "7FFF847EF000", "vmaddr" : "7FFF8401B000", "buildId" : "999E610F41FC32A3ADCA5EC049B65DFB" }, { "path" : "/usr/lib/libc++abi.dylib", "machType" : 6, "b" : "7FFF8E098000", "vmaddr" : "7FFF8D8C4000", "buildId" : "DCCC81773D0935BC97842A04FEC4C71B" }, { "path" : "/usr/lib/libDiagnosticMessagesClient.dylib", "machType" : 6, "b" : "7FFF879BE000", "vmaddr" : "7FFF871EA000", "buildId" : "4243B6B421E9355B9C5A95A216233B96" } ] }}
       mongod(_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE+0x39) [0x103b30119]
       mongod(_ZN5mongo10logContextEPKc+0x163) [0x103ae6523]
       mongod(_ZN5mongo11msgassertedEiPKc+0x195) [0x103ad6195]
       mongod(_ZN5mongo11msgassertedEiRKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEE+0x1A) [0x103ad5ffa]
       mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x398) [0x10327b1b8]
       mongod(_ZN5mongo9DbMessage9nextJsObjEv+0xA0) [0x103471360]
       mongod(_ZN5mongo12QueryMessageC2ERNS_9DbMessageE+0x5D) [0x1032c9d5d]
       mongod(_ZN5mongoL15receivedCommandEPNS_16OperationContextERKNS_15NamespaceStringERNS_6ClientERNS_10DbResponseERNS_7MessageE+0x8F) [0x10355499f]
       mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x7EA) [0x10355136a]
       mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0xFF) [0x10325864f]
       mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x27E) [0x103af289e]
       mongod(_ZNSt3__114__thread_proxyINS_5tupleIJNS_6__bindIPFPvS3_EJPN5mongo12_GLOBAL__N_124MessagingPortWithHandlerEEEEEEEEES3_S3_+0x61) [0x103af2041]
       libsystem_pthread.dylib(_pthread_body+0x83) [0x7fff8cac2c13]
       libsystem_pthread.dylib(_pthread_body+0x0) [0x7fff8cac2b90]
       libsystem_pthread.dylib(thread_start+0xD) [0x7fff8cac0375]
      -----  END BACKTRACE  -----
      2016-02-19T11:08:40.708-0500 I NETWORK  [conn22] AssertionException handling request, closing client connection: 10334 BSONObj size: 33609065 (0x200D569) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "posts"
      2016-02-19T11:08:40.725-0500 I NETWORK  [conn21] end connection 127.0.0.1:58280 (0 connections now open)
      2016-02-19T11:13:58.692-0500 W NETWORK  [HostnameCanonicalizationWorker] Failed to obtain name info for: [ (10.4.128.115, "nodename nor servname provided, or not known"), (10.4.128.115, "nodename nor servname provided, or not known") ]
      

      N.

        1. a.json
          33.91 MB
          Norberto Fernando Rocha Leite
        2. posts.json
          33.91 MB
          Norberto Fernando Rocha Leite

            Assignee:
            gabriel.russell@mongodb.com Gabriel Russell (Inactive)
            Reporter:
            norberto.leite Norberto Fernando Rocha Leite (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: