The JSON parser currently reads directly from the JSON string buffer, converting to BSON as it goes using recursive descent. This ticket is about adding a Tokenizer as an initial stage before doing the conversion to BSON. This may improve performance in some cases, since we could potentially reduce the number of string comparisons needed to determine which "$<keyword>" we are using, or which "<constructor>()" we are using for extended JSON types. This also may facilitate better error messages.

The fact that the JSON parser doesn't Tokenize as a separate stage also means that we cannot use parseNumberFromString, which is currently our standard way to read numbers out of strings, but requires the length of the number to be known beforehand.