CODESYS Forge - CODESYS Forge / Talk / Engineering 🇬🇧: JSONByteArrayWriter string result incorrect order

Hello everybody!

We are currently working on implementing a call to an API that tracks data sent to it through a JSON body.
Versions: CODESYS V3.5 SP19 Patch 7 using IIOT Libraries SL license, specifically JSON Utilities SL version 1.13.0.0.

For simplicity, let's say we want to construct the following JSON body for our request (simplified to just 1 key-value pair in each of the array objects to keep things shorter, but we'd have multiple fields in each in reality):

{
    "id": "machine_1",
    "readings": [
        {
            "temp_mid": 153.8
        },
        {
            "temp_mid": 98.3
        }
    ],
    "status": "RUNNING"
}

However, our "readings" data is added over time, not in creation, so calls to the builder would be somewhat out of order. In a simplified manner, our calls would look like this:

jsonBuilder(pJsonData := JsonData, diRootObj => rootJsonIndex);
// rootJsonIndex = 0.

// Set "id" at root JSON object.
wsValue := "machine_1";
jsonBuilder.SetKeyWithValue("id", wsValue, diParentIndex := rootJsonIndex);
// JSONData KEY: diParentIndex = 0 & diIndex = 1, VALUE: diParentIndex = 1 & diIndex = 2.

// ... Time passes, we get our first reading to add ...

// The "readings" array does not exist yet, so we make it first at root JSON object before adding our first reading.
readingsJsonArrayIndex := jsonBuilder.SetKeyWithArray("readings", diParentIndex := rootJsonIndex);
// JSONData KEY: diParentIndex = 0 & diIndex = 3, VALUE: diParentIndex = 3 & diIndex = 4.

// Then add the first reading, for which we first have to make an object.
readingsJsonArrayObjectIndex := jsonBuilder.SetObject(diParentIndex := readingsJsonArrayIndex);
// JSONData OBJECT: diParentIndex = 4 & diIndex = 5.

// And then we add the field(s) in the object for the first reading.
// NOTE the use of `readingsJsonArrayObjectIndex` so we set the field(s) in the first object in the array.
rValue := 153.8;
jsonBuilder.SetKeyWithValue("temp_mid", wsValue, diParentIndex := readingsJsonArrayObjectIndex);
// JSONData KEY: diParentIndex = 5 & diIndex = 6, VALUE: diParentIndex = 6 & diIndex = 7.

// ... Time passes, we want to add another root-JSON-level field ...

// Set "status" at root JSON object.
wsValue := "RUNNING";
jsonBuilder.SetKeyWithValue("status", wsValue, diParentIndex := rootJsonIndex);
// JSONData KEY: diParentIndex = 0 & diIndex = 8, VALUE: diParentIndex = 8 & diIndex = 9.

// ... Time passes, we want to add a second reading ...

// "readings" array exists, so add the second reading, but first we have to make another object.
// NOTE the re-use of the earlier stored `readingsJsonArrayIndex`, so we add the new object to that previously created array.
readingsJsonArrayObjectIndex := jsonBuilder.SetObject(diParentIndex := readingsJsonArrayIndex);
// JSONData OBJECT: diParentIndex = 4 & diIndex = 10.

// And then we add the field(s) in the object for the first reading.
// NOTE the use of `readingsJsonArrayObjectIndex` so we set the field(s) in the SECOND(, new) object in the array.
rValue := 98.3;
jsonBuilder.SetKeyWithValue("temp_mid", wsValue, diParentIndex := rootJsonIndex);
// JSONData KEY: diParentIndex = 10 & diIndex = 11, VALUE: diParentIndex = 11 & diIndex = 12.

// ... Finally, we want to send to the API, so we must convert the builder to data the client accepts as a body ...

xTestWriteToJsonArray := TRUE;

jsonArrayWriter(
    xExecute    := xTestWriteToJsonArray,
    pwData      := ADR(jsonDataString),
    udiSize     := SIZEOF(jsonDataString),
    jsonData    := JsonData^
);

IF jsonArrayWriter.xDone THEN
    // The writer was successful, the `jsonDataString` VAR should now contain a WString copy of the JSON object from the builder.           
    xTestWriteToJsonArray := FALSE;
END_IF

We see that the JsonData STRUCT is correctly organised (in terms of the diParentIndex & diIndex set by it for each JsonElement) as we expect, as outlined in my comments in the simplified code above.
However, as soon as we pass it to the jsonArrayWriter (which is a JSON.JSONByteArrayWriter), the resulting jsonDataString does not match our expectations, instead coming out like this:

{
    "id": "machine_1",
    "readings": [
        {
            "temp_mid": 153.8
        }
]
}
{
    "temp_mid": 98.3
}
"status": "RUNNING"

The behaviour of the JSONByteArrayWriter thus seems to be the problem here. It does not seem to correctly process the children of JsonElements that are ARRAYs when additions to them are done broken up by additions to the lower level JSON OBJECT they are a part of. We have confirmed this by changing the order of the calls in our example to add both readings before adding the "status", in which case we get our expected outcome. However, in reality this is not possible for us - additions to any of the JSON's objects, arrays or array objects may happen at any time after other fields elsewhere in the JSON have been added.

Does anyone know a way around this, a fix, or knows a solution we simply have not found?

This same phenomenon was also noted on this forum by user @ryusoup at the end of 2023 (https://forge.codesys.com/forge/talk/Engineering/thread/c45929e2f1/#e27f) and user @mtho in early 2024 (https://forge.codesys.com/forge/talk/Engineering/thread/cd1bb450db/#1292) but both topics received no activity beyond both users' opening posts.
Looking through the release notes of all the versions of the JSON Utilities SL library, I also did not see any remarks on resolutions of bugs in this vain.

Thanks in advance for the assistance!

Last edit: brouwyka 6 days ago

brouwyka - 6 days ago

Additional note: When doing all additions to a JSON array one after another without additions outside the array inbetween (and no more additions to the array after an item outside the array has been added) - which is not a viable solution for us as we might add items to any of the arrays at any time, but we tested regardless to be thorough - we get the following result:

{ "id": "machine_1", "readings": [ { "temp_mid": 153.8 }, { "temp_mid": 98.3 } ], "status": "RUNNING" }

This mostly matches our expected outcome, aside from the JSONByteArrayWriter not adding indenting before closing square brackets ("]") at all. That doesn't matter for JSON of course, as it's not sensitive to indenting, but it's still a bug regardless.
Also, it adds an unnecessary newline between items in the array (at least when having an array of objects, we haven't tested with arrays of primitives).

I also want to reiterate that this is not a viable workaround for the issue for us - we have to be able to add to any array at any time, which should be possible according to the docs, and we are 100% sure indexes are correctly being managed and built up by the JSONBuilder and our wrapper (to track the index of each array & the latest item added to each exactly to make use of the documented possibility to add anything anywhere in the JSON at any time as long as you keep track of the returned indexes) around it.

Last edit: brouwyka 6 days ago
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

See the example below. You can insert key/value pairs as required

At first the JSON string will be filled with:
"{$"Key1$": $"Value0$", $"Key2$": [{$"Key3$": 1, $"Key4$": 2}]}"

With xAdd, you can insert items, which will result in:

"{$"Key1$": $"Value0$", $"Key2$": [{$"Key3$": 1, $"Key4$": 2},{$"Key3$": 3, $"Key4$": 4}]}"

VAR
    factory : JSON.JSONDataFactory;
    eDataFactoryError : FBF.ERROR;
    pJsonData : POINTER TO JSON.JSONData := factory.Create(eError => eDataFactoryError);
    fb_JBuilder : JSON.JSONBuilder;
    wsValue : WSTRING;
    diRootIndex, diObjectIndex1, diObjectIndex2, diObjectIndex3 : DINT;
    iValue : INT;
    jsonArrayWriter : JSON.JSONByteArrayWriter;
    wsJsonData : WSTRING(1000);
    xFirst : BOOL := TRUE;
    xWrite: BOOL;
    xAdd: BOOL;
END_VAR

IF xFirst THEN
    fb_JBuilder.Reset();
    fb_JBuilder(pJsonData := pJsonData, diRootObj => diRootIndex);
    wsValue := "Value0";
    fb_JBuilder.SetKeyWithValue("Key1", wsValue, diParentIndex := diRootIndex);
    diObjectIndex1 := fb_JBuilder.SetKeyWithArray(wsKey := "Key2", diParentIndex := diRootIndex);
    diObjectIndex2 := fb_JBuilder.SetObject(diParentIndex := diObjectIndex1);
    iValue := 1;
    fb_JBuilder.SetKeyWithValue("Key3", iValue, diParentIndex := diObjectIndex2);
    iValue := 2;
    fb_JBuilder.SetKeyWithValue("Key4", iValue, diParentIndex := diObjectIndex2);
    xFirst := FALSE;
    xWrite := TRUE;
END_IF
IF xAdd THEN
    xAdd := FALSE;
    diObjectIndex3 := fb_JBuilder.SetObject(diParentIndex := diObjectIndex1);
    iValue := 3;
    fb_JBuilder.SetKeyWithValue("Key3", iValue, diParentIndex := diObjectIndex3);
    iValue := 4;
    fb_JBuilder.SetKeyWithValue("Key4", iValue, diParentIndex := diObjectIndex3);
    xWrite := TRUE;
END_IF
IF xWrite THEN
    xWrite := FALSE;
    jsonArrayWriter(
        xExecute := TRUE,
        wsLinebreak := "",
        pwData := ADR(wsJsonData),
        udiSize := SIZEOF(wsJsonData),
        jsonData := pJsonData^,
        xAsyncMode := FALSE
    );
END_IF
IF jsonArrayWriter.xDone OR jsonArrayWriter.xError THEN
    jsonArrayWriter(xExecute := FALSE, pwData := ADR(wsJsonData), udiSize := SIZEOF(wsJsonData), jsonData := pJsonData^, xAsyncMode := FALSE);
END_IF

brouwyka - 1 day ago

Hi @TimvH,

As discussed in our email contact, your example does not actually test/reproduce the bug I am describing:
Adding to the JSON builder on later cycles works fine.

Your example works because you do the following:
1. You finish the first cycle (xFirst) with adding an object to the array with 2 fields;
2. When the next cycle is triggered (xAdd), you immediately start with adding another object to that array.

This works fine as the last thing you did before #2 was the addition of the array and an object to it.

To reproduce the bug you should be doing the following instead, as I showed with the code I shared with my first post:
1. Create the array - save the index that is returned;
2. Add an item, either a primitive or an object, to the array;
3. Add an item outside the array (anything: a primitive, a new array, a nested object, etc);
4. Add another item to the array using the index you saved at #1.

You will now see, as I shared in my initial post, that the second item is placed completely outside the root JSON object.
This also happens to anything else you try to add after step #4: everything after this point will be added outside the root JSON object: the JSON is completely broken.

Important to note is that this not only happens with arrays, but also with nested JSON objects.
Once you add something outside of a nested JSON object, you can no longer add anything to that nested JSON object, as that causes the exact same bug.
This also applies to arrays of objects, so if in your test you had tried adding a new key-value pair to the first nested object in your array after you created the second nested object, you would also run into this bug.

It seems that the JSONByteArrayWriter (I haven't tested the other writers in the JSON Utilities SL library, so I don't know if they suffer from the same problem) simply does not handle any JSON fields that add brackets (so arrays with "[" & "]" and nested objects with "{" & "}") well, and closes them prematurely instead of checking if any later JSONElements in the JSONData's array belong to any of these bracketed fields.
After reviewing the objects & functions of the JSON Utilities SL library, my guess is that either the JSONByteArrayWriter linearly goes through the array of JSONElements in the JSONData and only checks the diParentIndex of each JSONElement in direct ascending order, OR if it does use the JSONElement.GetChildren(); method, that this method is either broken and doesn't give all children correctly to the writer.
Neither explains why everything completely breaks, and you cannot even add to the root JSON object anymore, however, so there is probably more than just that going wrong in the writer.

To me, after a full week of testing and attempting workarounds, this seems like a bug in the library that needs to be fixed by Codesys, as I cannot see anything wrong in the JSONData constructed by the JSONBuilder - this seems purely a problem in the writer.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Forums

Help

JSONByteArrayWriter string result incorrect order

Forums

Help

JSONByteArrayWriter string result incorrect order document.SUBSCRIPTION_OPTIONS = { "thing": "topic", "subscribed": false, "url": "subscribe", "icon": { "css": "fa fa-envelope-o" } };

JSONByteArrayWriter string result incorrect order