When using the $http stage to fetch results from an API that returns valid JSON, but the response is a string, we need a way to serialize that JSON as a string to an object.

For example, the OpenAI API returns the results of a query to the LLM like this:

{
  "message": {
    "content": "{\"sentiment_score\":0, \"reasoning\": \"The text is neutral without any sentiment expressed regarding streaming data.\", \"streaming_score\":0 }"
  }
}

This is valid JSON but we need a way to serialize this output to an object. In python for example one could write code like:

mydict = json.loads(message.content) 

And mydict would be a dictionary (an object) serialized from the string with the (for example) keys 'sentiment_score', 'reasoning', and 'streaming_score' from above.

We need that same capability in ASP so that the results of a call like this can be serialized into the pipeline and downstream stages can work with the results.

$convert doesn't work on objects, and $jsontoDocument has been deprecated so we lack a tool to take a string and return an object.

Note the string has escapes for double quotes in the string. The solution should handle this case.

If the data can't be converted, then send to DLQ.

 
We will use the new field that will be in the config.
parseResponseFields will be optional and take in an array of strings. i.e. ["content", "nested.other"].
Being verbose to avoid confusion this feature could apply to the payload being sent to the destination server.Normal usage would could look like:

 $https: { as: "response", config: { parseResponseFields: ["content"] } }

 
If the https destination server returns a JSON Object payload that looks like

{"content": "{\"a\": 0, \"b\": \"foobar\"}"}

 
result

 { ...rest of doc, "response": { "content": { "a": 0, "b": "foobar" } } }

 
If the https destination server returns a JSON Array payload that looks like

[ {"content": "{\"a\": 0, \"b\": \"foobar\"}"}, {"content": "{\"c\": 1, \"d\": \"idk\"}"} ]

 
result

 { ...rest of doc, "response": [ { "content": { "a": 0, "b": "foobar" } }, { "content": { "c": 1, "d": "idk" } } ] }