[SERVER-1083] mongoimport: option to support an array of json objects Created: 04/May/10  Updated: 12/Jul/16  Resolved: 14/Jun/10

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 1.5.1
Fix Version/s: 1.5.3

Type: Improvement Priority: Major - P3
Reporter: Alex Sharp Assignee: Mathias Stearn
Resolution: Done Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

os x


Participants:

 Description   

Many public API's (twitter, for example) return a collection of json objects for a request for a collection of resources. The mongoimport tool only understands newline-delimited json objects in a file. In other words, it only understands one json object at a time. It seems like mongoimport should know how to understand these collections natively. For example, trying the following will result in an error:

curl http://api.twitter.com//1/statuses/user_timeline.json?screen_name=noradio > noradio.twitter.json
mongoimport -d twitter -c noradio --file=noradio.twitter.json



 Comments   
Comment by Eliot Horowitz (Inactive) [ 01/Sep/10 ]

@maxime if you want to increase the limit, can you open a new case. this one shouldn't be modified anymore

Comment by Maxime Bizon [ 26/Aug/10 ]

Hi Mathias,

I just hit the array size limit. The generated data comes from a perl script, where it is handy to put everything in a array or hash, and at the end JSON::encode the whole thing.

I can certainly put the loop in the perl script to generate one line per entry, but it doesn't seem natural.

Any chance this limit could be raised ?

Thanks,

Comment by Mathias Stearn [ 14/Jun/10 ]

Right now the array must be <4MB. Let me know if that is an issue.

Comment by auto [ 14/Jun/10 ]

Author:

{'login': 'RedBeard0531', 'name': 'Mathias Stearn', 'email': 'redbeard0531@gmail.com'}

Message: test for --jsonArray SERVER-1083
http://github.com/mongodb/mongo/commit/b52051be9340e0ddf4b2d0d75339647654a47d9d

Comment by auto [ 14/Jun/10 ]

Author:

{'login': 'RedBeard0531', 'name': 'Mathias Stearn', 'email': 'redbeard0531@gmail.com'}

Message: --jsonArray for mongoexport SERVER-1083
http://github.com/mongodb/mongo/commit/78cb3cfc87981128b50211ba9efd6496ceb7b17c

Comment by auto [ 14/Jun/10 ]

Author:

{'login': 'RedBeard0531', 'name': 'Mathias Stearn', 'email': 'redbeard0531@gmail.com'}

Message: --jsonArray for mongoimport SERVER-1083
http://github.com/mongodb/mongo/commit/dc95ed05762010809d000354a6a9a251c39066fd

Generated at Thu Feb 08 02:56:00 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.