[CDRIVER-2128] Support revised Extended JSON spec Created: 17/Apr/17  Updated: 08/Jan/18  Resolved: 11/Jul/17

Status: Closed
Project: C Driver
Component/s: json, libbson
Affects Version/s: 1.7.0
Fix Version/s: 1.7.0

Type: Improvement Priority: Major - P3
Reporter: A. Jesse Jiryu Davis Assignee: A. Jesse Jiryu Davis
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by PHPC-926 Implement Extended JSON spec Closed
is depended on by DRIVERS-351 Implement Extended JSON Spec Implementing
Related
related to CDRIVER-2223 JSON reader cannot parse integers for... Closed
related to CDRIVER-2225 BSON corpus Decimal128 parse error te... Closed
related to PHPC-1067 BSON document produces driver segfault Closed
related to CDRIVER-2208 Introduce new API for Extended JSON s... Closed
related to PHPC-926 Implement Extended JSON spec Closed
related to PHPC-954 Revert toExtendedJSON() function Closed
related to PHPC-828 Use canonical extended JSON for Times... Closed
is related to CDRIVER-1947 Improve libbson compatibility with th... Closed
is related to CDRIVER-2168 Escape JSON strings for regex, symbol... Closed

 Description   

We're rethinking the MongoDB Extended JSON spec. libbson is among the first of our drivers to complete an implementation; luckily we haven't released it yet. Disable or revert the Extended JSON changes in libbson.



 Comments   
Comment by Ramon Fernandez Marina [ 12/Sep/17 ]

Author:

{'username': u'ajdavis', 'name': u'A. Jesse Jiryu Davis', 'email': u'jesse@mongodb.com'}

Message:CDRIVER-2128 prohibit extra keys in $dbPointer
Branch:master
https://github.com/mongodb/libbson/commit/ba7dd45313fdfef61d4f813930818e8493f96d99

Comment by Ramon Fernandez Marina [ 12/Sep/17 ]

Author:

{'username': u'ajdavis', 'name': u'A. Jesse Jiryu Davis', 'email': u'jesse@mongodb.com'}

Message:CDRIVER-2128 $regex now requires $options
Branch:master
https://github.com/mongodb/libbson/commit/38caceeec3b4a38512511d8ed56c59b6e13ba61a

Comment by Ramon Fernandez Marina [ 12/Sep/17 ]

Author:

{'username': u'ajdavis', 'name': u'A. Jesse Jiryu Davis', 'email': u'jesse@mongodb.com'}

Message:CDRIVER-2128 merge Decimal128 and BSON Corpus tests
Branch:master
https://github.com/mongodb/libbson/commit/d7406f5dc010f4eb223c9eef8c144aa6efc33504

Comment by Ramon Fernandez Marina [ 12/Sep/17 ]

Author:

{'username': u'ajdavis', 'name': u'A. Jesse Jiryu Davis', 'email': u'jesse@mongodb.com'}

Message:CDRIVER-2128 drop old

{"$date": 12345}

JSON syntax

Extended JSON spec prohibits it, only

{"$date": "2017-06-07T12:34:45"}

or {"$date": {"$numberLong": "12345"}} are allowed.
Branch:master
https://github.com/mongodb/libbson/commit/933fd98107d11e2f9e516c229c7eb23dad941812

Comment by Ramon Fernandez Marina [ 12/Sep/17 ]

Author:

{'username': u'ajdavis', 'name': u'A. Jesse Jiryu Davis', 'email': u'jesse@mongodb.com'}

Message:CDRIVER-2128 bug in BSON Corpus test
Branch:master
https://github.com/mongodb/libbson/commit/0cd3dc1f7bcd8a08db1067b9c4bd83e007daaec4

Comment by Ramon Fernandez Marina [ 12/Sep/17 ]

Author:

{'username': u'ajdavis', 'name': u'A. Jesse Jiryu Davis', 'email': u'jesse@mongodb.com'}

Message:CDRIVER-2128 update JSON tests

Implement the "degenerate_extjson" portion of the tests.
Branch:master
https://github.com/mongodb/libbson/commit/a4353590df13e105a235325b34d4a5271826a4cb

Comment by Ramon Fernandez Marina [ 12/Sep/17 ]

Author:

{'username': u'ajdavis', 'name': u'A. Jesse Jiryu Davis', 'email': u'jesse@mongodb.com'}

Message:CDRIVER-2128 test ISO8601 date formatting
Branch:master
https://github.com/mongodb/libbson/commit/cdb73fdf4b2f783b655ed19c9de622632b197100

Comment by Ramon Fernandez Marina [ 12/Sep/17 ]

Author:

{'username': u'ajdavis', 'name': u'A. Jesse Jiryu Davis', 'email': u'jesse@mongodb.com'}

Message:CDRIVER-2128 avoid int overflow when parsing JSON
Branch:master
https://github.com/mongodb/libbson/commit/d48d106c100a32b6808afbf11557bd222315bdc0

Comment by Ramon Fernandez Marina [ 12/Sep/17 ]

Author:

{'username': u'ajdavis', 'name': u'A. Jesse Jiryu Davis', 'email': u'jesse@mongodb.com'}

Message:CDRIVER-2128 update JSON tests
Branch:master
https://github.com/mongodb/libbson/commit/63b71beff4760733879bc90343e9a8e8fa33bdea

Comment by Ramon Fernandez Marina [ 12/Sep/17 ]

Author:

{'username': u'ajdavis', 'name': u'A. Jesse Jiryu Davis', 'email': u'jesse@mongodb.com'}

Message:CDRIVER-2128 update BSON Corpus tests
Branch:master
https://github.com/mongodb/libbson/commit/980546ac70cd7ded7a09be9c7173ca0135843eb7

Comment by Ramon Fernandez Marina [ 12/Sep/17 ]

Author:

{'username': u'ajdavis', 'name': u'A. Jesse Jiryu Davis', 'email': u'jesse@mongodb.com'}

Message:CDRIVER-2128 revert $timestamp-parsing change
Branch:master
https://github.com/mongodb/libbson/commit/f87dc5160b66e3611521eac2fb1287e24e31a682

Comment by Ramon Fernandez Marina [ 12/Sep/17 ]

Author:

{'username': u'ajdavis', 'name': u'A. Jesse Jiryu Davis', 'email': u'jesse@mongodb.com'}

Message:CDRIVER-2128 revert $timestamp-generation change
Branch:master
https://github.com/mongodb/libbson/commit/6389f1f8c3796a5054c683fac0b2b7c014a8d3da

Comment by Ramon Fernandez Marina [ 12/Sep/17 ]

Author:

{'username': u'ajdavis', 'name': u'A. Jesse Jiryu Davis', 'email': u'jesse@mongodb.com'}

Message:CDRIVER-2128 wrap NaN in $numberDouble
Branch:master
https://github.com/mongodb/libbson/commit/820f104ad76a08b3a3525b760fc51c8312442673

Comment by Ramon Fernandez Marina [ 12/Sep/17 ]

Author:

{'username': u'ajdavis', 'name': u'A. Jesse Jiryu Davis', 'email': u'jesse@mongodb.com'}

Message:CDRIVER-2128 code style
Branch:master
https://github.com/mongodb/libbson/commit/713f5a185f185d6453b603a42a1d0d4293de2ab8

Comment by Ramon Fernandez Marina [ 12/Sep/17 ]

Author:

{'username': u'ajdavis', 'name': u'A. Jesse Jiryu Davis', 'email': u'jesse@mongodb.com'}

Message:CDRIVER-2128 generate ISO8601 JSON datetimes
Branch:master
https://github.com/mongodb/libbson/commit/3cc35e92e98c440cfa0e33a7907f907e405aa90d

Comment by Ramon Fernandez Marina [ 12/Sep/17 ]

Author:

{'username': u'ajdavis', 'name': u'A. Jesse Jiryu Davis', 'email': u'jesse@mongodb.com'}

Message:CDRIVER-2128 parse legacy $regex
Branch:master
https://github.com/mongodb/libbson/commit/f510e319fadcdfcf79a4a14f7209273913b78f4c

Comment by Ramon Fernandez Marina [ 12/Sep/17 ]

Author:

{'username': u'ajdavis', 'name': u'A. Jesse Jiryu Davis', 'email': u'jesse@mongodb.com'}

Message:CDRIVER-2128 test parsing legacy $regex
Branch:master
https://github.com/mongodb/libbson/commit/ccb24e31551760d72f1b7935f6c58dc7853d0629

Comment by Ramon Fernandez Marina [ 12/Sep/17 ]

Author:

{'username': u'ajdavis', 'name': u'A. Jesse Jiryu Davis', 'email': u'jesse@mongodb.com'}

Message:CDRIVER-2128 use $regularExpression, not $regex

A change in the Extended JSON Spec.
Branch:master
https://github.com/mongodb/libbson/commit/9755d9f64fc0a309362e786333e79976face5df4

Comment by Ramon Fernandez Marina [ 12/Sep/17 ]

Author:

{'username': u'ajdavis', 'name': u'A. Jesse Jiryu Davis', 'email': u'jesse@mongodb.com'}

Message:CDRIVER-2128 update BSON corpus tests
Branch:master
https://github.com/mongodb/libbson/commit/365ffc80090829602737b267dc97fc5f591b3600

Comment by Ramon Fernandez Marina [ 12/Sep/17 ]

Author:

{'username': u'ajdavis', 'name': u'A. Jesse Jiryu Davis', 'email': u'jesse@mongodb.com'}

Message:CDRIVER-2128 format Binary as $binary:

{base64: ..., subType: ...}

Branch:master
https://github.com/mongodb/libbson/commit/9866ca7e94f4b32f0c8e047ff80ef91ffb5833fb

Comment by Ramon Fernandez Marina [ 12/Sep/17 ]

Author:

{'username': u'ajdavis', 'name': u'A. Jesse Jiryu Davis', 'email': u'jesse@mongodb.com'}

Message:CDRIVER-2128 test legacy $binary parsing
Branch:master
https://github.com/mongodb/libbson/commit/6147635f8c9e2265eb360124b1ce6860082fede9

Comment by Ramon Fernandez Marina [ 12/Sep/17 ]

Author:

{'username': u'ajdavis', 'name': u'A. Jesse Jiryu Davis', 'email': u'jesse@mongodb.com'}

Message:CDRIVER-2128 typo in NEWS
Branch:master
https://github.com/mongodb/libbson/commit/ad4d7c28fa8e1b2b92df17cffd20ba4351006640

Comment by Ramon Fernandez Marina [ 12/Sep/17 ]

Author:

{'username': u'ajdavis', 'name': u'A. Jesse Jiryu Davis', 'email': u'jesse@mongodb.com'}

Message:CDRIVER-2128 ensure regex options are sorted

Ensure that the regex option flags "ilmsux" are stored in sorted order,
even when passed to bson_append_regex.
Branch:master
https://github.com/mongodb/libbson/commit/f9c179bbc0fe6049a73a46abac2fc3e27c832393

Comment by Ramon Fernandez Marina [ 12/Sep/17 ]

Author:

{'username': u'ajdavis', 'name': u'A. Jesse Jiryu Davis', 'email': u'jesse@mongodb.com'}

Message:CDRIVER-2128 ensure regex options are sorted

Ensure that the regex option flags "ilmsux" are stored in sorted order,
even when passed to bson_append_regex.
Branch:r1.7
https://github.com/mongodb/libbson/commit/ff66b1907c3e2bf608980c289381a0e3cf2afbf2

Comment by A. Jesse Jiryu Davis [ 11/Jul/17 ]

CDRIVER-2128 ensure regex options are sorted.

Master: https://github.com/mongodb/libbson/commit/f9c179bbc0fe6049a73a46abac2fc3e27c832393

r1.7: https://github.com/mongodb/libbson/commit/ff66b1907c3e2bf608980c289381a0e3cf2afbf2

Comment by Githook User [ 11/Jul/17 ]

Author:

{u'username': u'jmikola', u'name': u'Jeremy Mikola', u'email': u'jmikola@gmail.com'}

Message: Expect failure in regex flag test pending CDRIVER-2128
Branch: master
https://github.com/mongodb/mongo-php-driver/commit/6c5bfafd9c4140c092249fa92365cc198d938cf6

Comment by Githook User [ 11/Jul/17 ]

Author:

{u'username': u'jmikola', u'name': u'Jeremy Mikola', u'email': u'jmikola@gmail.com'}

Message: PHPC-977: Update tests for expected JSON output from libbson

Extended JSON output for libbson changed in CDRIVER-2128.
Branch: master
https://github.com/mongodb/mongo-php-driver/commit/2761072d2447ccd92a5ead8b763613da9e2f83e5

Comment by Jeremy Mikola [ 10/Jul/17 ]

jesse: I believe I found one more outstanding error with 1.7.0-rc0.

mongodb/specifications#161 includes the following test case in source/bson-corpus/tests/regex.json:

{
     "description": "flags not alphabetized",
     "degenerate_bson": "100000000B6100616263006D69780000",
     "canonical_bson": "100000000B610061626300696D780000",
     "canonical_extjson": "{\"a\" : {\"$regularExpression\" : { \"pattern\": \"abc\", \"options\" : \"imx\"}}}",
     "degenerate_extjson": "{\"a\" : {\"$regularExpression\" : { \"pattern\": \"abc\", \"options\" : \"mix\"}}}"
},

Using the included bsonview utility, we have the following representations of canonical_bson and degenerate_bson, respectively:

$ echo "100000000B610061626300696D780000" | ./bsonview -x
 10000000 0b "a" 00 "abc" 00 "imx" 00 00
 
$ echo "100000000B6100616263006D69780000" | ./bsonview -x
 10000000 0b "a" 00 "abc" 00 "mix" 00 00

The guide for BSON corpus spec tests includes the following:

  • for dEJ input (if it exists):
    • json_to_bson(dEJ) = cB (unless lossy)

Since this isn't a lossy test, we should expect the degenerate extended JSON string to produce canonical BSON. Based on my testing, bson_init_from_json() fails to alphabetize the regex flags and produces the degenerate BSON instead.

Comment by A. Jesse Jiryu Davis [ 27/Jun/17 ]

I've completed the main changes. We now also parse and generate $regularExpression, while still parsing the old $regex format. Still todo: parse and generate this new Binary format:

{"x" : { "$binary" : {"hex" : "", "subType" : "00"}}}

We'll maintain the code that parses the old format:

{"x" : { "$binary" : "", "$type" : "00"}}

I hear from Jeff that the Java driver will follow the same path.

Comment by A. Jesse Jiryu Davis [ 18/Jun/17 ]

Blocked waiting for Jeff to propose a change to $regex syntax.

Comment by A. Jesse Jiryu Davis [ 17/Jun/17 ]

We've caught up to the current Extended JSON Spec and its tests. I'm keeping this open until I hear what jeff.yemin@mongodb.com wants to change about $regex parsing and generation.

Comment by A. Jesse Jiryu Davis [ 05/May/17 ]

Blocked on revised Extended JSON spec.

Generated at Wed Feb 07 21:14:16 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.