[SERVER-22885] Unable to find some of filed in document which were imported by mondoimport. Created: 28/Feb/16 Updated: 11/Mar/16 Resolved: 11/Mar/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Querying |
| Affects Version/s: | 3.2.1 |
| Fix Version/s: | None |
| Type: | Question | Priority: | Major - P3 |
| Reporter: | Chaedoo Jun [X] | Assignee: | Kelsey Schubert |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Participants: | |||||||||
| Description |
|
I am testing MongoDB to find out if it would be a solution to automate a task. To achieve this task, I have to import around 20,000 xml files in to MondoDB collection which I think it's really suite for MondodB. I tested with mongo v3.2 and mongo compass to verify most possible queries what I need to use. While I am testing after import 2 xml files after I converted to json format, found some of queries are returned with no values but some of them are OK. Following is successful case;
Following case has returned no values
I have tested same case with Mongo compass but result are same. All of data in this case is test data so I have attached a sample json file here so please advise me why I could not get any data when I query with CardSerialNumber ? |
| Comments |
| Comment by Kelsey Schubert [ 11/Mar/16 ] | ||||||||||||||||||||||||||||||
|
SERVER-6439 describes modifying the behavior of MongoDB to not allow duplicate fields at the same level. I believe this ticket encompasses your concerns about how MongoDB handles insertion of data with repeated field names. Please watch SERVER-6439 for updates, and feel free to vote for it. Additionally, I'd like to clarify that in your example you are not seeing b1 values written into the document because the shell helper is treating {b: {x: 1, y: 1}} as insert options. Please see the two following examples using db.collection.insert():
This inserts a single document that contains two subdocuments a1 and b1.
This does a bulk insert and creates two documents: one has a subdocument a1, and the other has a subdocument b1. For guidance about schema design in MongoDB, please consider posting on the mongodb-users group describing your use case. Kind regards, | ||||||||||||||||||||||||||||||
| Comment by Chaedoo Jun [X] [ 10/Mar/16 ] | ||||||||||||||||||||||||||||||
|
It totally make sense for me example {"x":1,"x":2}case but ,please correct me if I am wrong, when I tested as below , it still treated as same field name if I use dot notation like a.x and a.y.
Even it did not write b1 values into document and no error or warning displayed ... ( I assume it should threw error message saying duplicate field name or similar way ) | ||||||||||||||||||||||||||||||
| Comment by Chaedoo Jun [X] [ 10/Mar/16 ] | ||||||||||||||||||||||||||||||
|
Thanks for your kind answer and now it makes sense for me. It is disappointing I could not adapt MongoDB in to my first project.. but I will find another task to use MongoDB. | ||||||||||||||||||||||||||||||
| Comment by Kelsey Schubert [ 10/Mar/16 ] | ||||||||||||||||||||||||||||||
|
Thank you for the clarifying information. I understand what is happening now, and can successfully reproduce it on my end. The behavior that you are observing is the result of having more than one field with the same name in a single document. As our documentation mentions while BSON documents may have more than one field with the same name, most MongoDB interfaces represent MongoDB with a structure (e.g. a hash table) that does not support duplicate field names. The issue that you are observing is that if field names are set twice in the same document, mongo shell and Compass may display different values for this field. Furthermore, if the values displayed by the shell and Compass differ, only the values displayed by the shell are used to determine whether a document should be returned as part of a query. As a simple example please consider the following json
And import it using mongoimport. The mongo shell will display the following results
However, in Compass this same document will display that x equals 2. As shown above, a query on this equality match (x = 2) will not return any results. This behavior explains the why you are seeing "No document found" when you query using the values in Compass for some documents. To resolve this issue on your side, I would recommend modifying your file conversion method to ensure that there are no duplicate field names in your documents. Thank you, | ||||||||||||||||||||||||||||||
| Comment by Chaedoo Jun [X] [ 10/Mar/16 ] | ||||||||||||||||||||||||||||||
|
Thanks for your support but I am sorry that I don't get what you said "expected behavior" . I have tested with customized document which I uploaded for the first time, it actually returned vales as I expected but other files that I converted from xml to JSON, it returns no values even though it has imported with no error. As attached image named " before_apply.jpg" there is a record with card_id but when I click apply, it returned no document as attached image "after_apply.jpg" Would you please explain why it show different result ? | ||||||||||||||||||||||||||||||
| Comment by Kelsey Schubert [ 09/Mar/16 ] | ||||||||||||||||||||||||||||||
|
Thank you for uploading these documents. Unfortunately, I have been unable reproduce the issue you describe where some documents cannot be found. The image you have privately uploaded shows expected behavior. The document containing the subdocument with the matching CardSerialNumber is returned in its entirety including many fields not related to the card number. For more information, please review our documentation on querying an array of embedded documents. From my investigation, I do not see anything to indicate a bug in the MongoDB server. For MongoDB-related support discussion please post on the mongodb-users group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-users group. | ||||||||||||||||||||||||||||||
| Comment by Chaedoo Jun [X] [ 06/Mar/16 ] | ||||||||||||||||||||||||||||||
|
any progress ?? | ||||||||||||||||||||||||||||||
| Comment by Chaedoo Jun [X] [ 29/Feb/16 ] | ||||||||||||||||||||||||||||||
|
Also all of data what I sent you were from test system but please DO NOT SHARE or DISTRIBUTE any other places or materials. | ||||||||||||||||||||||||||||||
| Comment by Chaedoo Jun [X] [ 29/Feb/16 ] | ||||||||||||||||||||||||||||||
|
Hi Thomas. Thanks for your feedback. By the way... I found wired result of query when I test with other imported file ( same kind of file I converted from other test platform). It gave me result of query included all other fields which are not matching with my query condition. I will upload additional json files ( AC20120302113030.json, AC20120302113730.json). ) then you will get many fields returned not related with the card number. | ||||||||||||||||||||||||||||||
| Comment by Kelsey Schubert [ 29/Feb/16 ] | ||||||||||||||||||||||||||||||
|
I used mongoimport to import the sample json file, and was able to successfully locate it using the following query:
So we can continue to investigate can you please provide your complete dataset? I have created a secure upload portal here. Files you upload will only be visible to MongoDB employees investigating this behavior. Thank you, | ||||||||||||||||||||||||||||||
| Comment by Chaedoo Jun [X] [ 28/Feb/16 ] | ||||||||||||||||||||||||||||||
|
I just capture compass screen because it's easy to show my symptom. |