[CDRIVER-1370] Replace libyajl with jsonsl Created: 11/Jul/16  Updated: 22/May/18  Resolved: 13/Dec/16

Status: Closed
Project: C Driver
Component/s: json, libbson
Affects Version/s: None
Fix Version/s: 1.6.0

Type: Improvement Priority: Major - P3
Reporter: A. Jesse Jiryu Davis Assignee: A. Jesse Jiryu Davis
Resolution: Done Votes: 0
Labels: intern2016
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by CDRIVER-994 "[{$numberLong: '1'}]" is BSON-encode... Closed
is depended on by CDRIVER-1942 JSON can be parsed despite stray "}" ... Closed
is depended on by CDRIVER-1472 NaN handling difference between C and... Closed
is depended on by CDRIVER-1913 Parse "$code" extended JSON Closed
Related
is related to CDRIVER-597 Support building without requiring ve... Closed

 Description   

libyajl's unmaintained and overly complex for our purposes – there are a ton of JSON parsing bugs open in libbson that are very hard to fix with libyajl. We currently vendor libyajl and we've patched it to make it perform better when parsing a stream of documents, but we can't use that patch on Debian, due to Debian's policy against vendoring third-party libraries that are available as Debian packages. This puts us in a Catch-22: we can't fix upstream since upstream isn't maintained, but we can't use our patch since Debian prohibits vendoring libyajl there.

We'll vendor a different JSON-parsing library that is not already packaged for Debian, that presents a simpler interface better suited to our needs, and which already performs well when parsing a stream of documents. I'd thought JSMN was good but it's not actually incremental – not suitable for parsing a very large stream of one or more JSON objects. We'll use jsonsl. It's MIT licensed, actually designed for incremental parsing with callbacks, seems complex but good, not in Debian (which is good news).

Local benchmarks show between 1% and 15% gain on the JSON portions of the Driver Benchmark Suite with the first draft of a JSONSL port.

Status:

  • Done: libbson ported to jsonsl, passes all tests, plus a bit of additional testing for areas that seemed risky to me as I did the port
  • Done: test with bad escape sequences, I think jsonsl_util_unescape's "len" might be invalid if "err" is set.
  • Done: escape sequences in keys, test and implement.
  • Done: functions like static int _bson_json_read_null (void *_ctx) can be updated with void returns and take a bson_json_reader_t, const char pointer, and ssize_t, so I can delete a bunch of casts
  • Done: were top-level arrays allowed with previous implementation?
  • Done: large amount of cleanup and additional error checks with tests for those missing checks
  • Done: perf improvement by setting call_UESCAPE and handling escapes inline, instead of on whole keys and string values?
  • Done: the naked malloc and free in jsonsl need to be patched to use the BSON macros


 Comments   
Comment by Githook User [ 22/May/18 ]

Author:

{'username': 'jmikola', 'name': 'Jeremy Mikola', 'email': 'jmikola@gmail.com'}

Message: Remove YAJL include path for Windows builds

YAJL was removed in libbson 1.6.0 (CDRIVER-1370).
Branch: master
https://github.com/mongodb/mongo-php-driver/commit/c613391ddbc4cbfc722a7ac6f6d87557d5dda47c

Comment by Githook User [ 07/Apr/17 ]

Author:

{'username': 'ajdavis', 'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com'}

Message: CDRIVER-1370 avoid bare malloc/free in JSONSL (reapplied)

Re-applied after reverting our changes to jsonsl.h and jsonsl.c.
Originally #da366bb2.
Branch: master
https://github.com/mongodb/libbson/commit/d222f6f628e2752d178b2b40ffbf2bb15dc282c2

Comment by Githook User [ 07/Apr/17 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-1370 avoid bare malloc/free in JSONSL (reapplied)

Re-applied after reverting our changes to jsonsl.h and jsonsl.c.
Originally #da366bb2.
Branch: master
https://github.com/mongodb/libbson/commit/d222f6f628e2752d178b2b40ffbf2bb15dc282c2

Comment by Githook User [ 02/Feb/17 ]

Author:

{u'username': u'jmikola', u'name': u'Jeremy Mikola', u'email': u'jmikola@gmail.com'}

Message: PHPC-892: Bump libbson to 1.6.0

Includes CDRIVER-1370, which replaces yajl with jsonsl.
Branch: master
https://github.com/mongodb/mongo-php-driver/commit/13b82a5357d2238c5eaa9a48c9ec025b3e26fb67

Comment by Githook User [ 13/Dec/16 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-1370 avoid bare malloc/free in JSONSL
Branch: master
https://github.com/mongodb/libbson/commit/da366bb24db363604e6ca7f9a3cb8261de236130

Comment by Githook User [ 07/Dec/16 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-1370 test JSON with final "{}"
Branch: master
https://github.com/mongodb/libbson/commit/323ab1b147751698f7bc480404600849718fe530

Comment by Githook User [ 07/Dec/16 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-1370 note about a failed optimization
Branch: master
https://github.com/mongodb/libbson/commit/cb3be078f885ca83f0d7dd9593b7b923f79eafb3

Comment by Githook User [ 07/Dec/16 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-1370 dead code in JSONSL port
Branch: master
https://github.com/mongodb/libbson/commit/4559af8006721bff37440f81953e1bcae4a01dea

Comment by Githook User [ 07/Dec/16 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-1370 use proper JSONSL destructor
Branch: master
https://github.com/mongodb/libbson/commit/478e6ca897a233c4547515b47f5cf15ac76241b8

Comment by Githook User [ 07/Dec/16 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-1370 handle corrupt UTF-8 in JSON keys
Branch: master
https://github.com/mongodb/libbson/commit/943d0079ada1a27e59ae4e471e923eab7f0ca07a

Comment by Githook User [ 07/Dec/16 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-1370 cleanups for JSONSL port
Branch: master
https://github.com/mongodb/libbson/commit/7329c32c44341382c7030856f8c2fd14bc0f74a7

Comment by Githook User [ 07/Dec/16 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-1370 decode escapes in JSON keys
Branch: master
https://github.com/mongodb/libbson/commit/a12e1ea82a53f0eea7cc32c33cc37c8f82ed73b5

Comment by Githook User [ 07/Dec/16 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-1370 handle bad JSON escapes
Branch: master
https://github.com/mongodb/libbson/commit/58063f9019b78ed44253a5c167ffed34f4e08dc1

Comment by Githook User [ 07/Dec/16 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-1370 replace yajl with jsonsl
Branch: master
https://github.com/mongodb/libbson/commit/f508b26a0d748171626cc7b2299031f98d3dd8e7

Generated at Wed Feb 07 21:12:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.