Welcome to our new forum
All users of the legacy CODESYS Forums, please create a new account at account.codesys.com. But make sure to use the same E-Mail address as in the old Forum. Then your posts will be matched. Close

http data in JSON format - encoding

damian177
2022-01-27
2022-01-28
1 2 > >> (Page 1 of 2)
  • damian177

    damian177 - 2022-01-27

    Hi,
    I have a problem.
    I get data from http get request in JSON format. When I use http_sys.hTTPClient_0.httpResult.wsContent (WSTRING) everything with chars (UTF-8) is ok. But for the needs JSON Parse I must use http_sys.hTTPClient_0.httpResult.sContent (STRING) data and in this case I have problem with Polish characters.
    After parse I create variables to print on HMI(webvisu):
    description:= STRING_TO_WSTRING(NewJSONObj.description.AsString);

    And the result is that I have no Polish characters. How resolve this problem ?

     

    Last edit: damian177 2022-01-27
  • dFx

    dFx - 2022-01-27

    The issue is using string_to function assumes that every char in your string is coded on 8 bits, while UTF8 is on 16 bits. So you are interpreting splitted chars.

    I don't know why you "need" to use .sContent to parse, but this could be a way to solve it.

    Also you may also just interpret a byte array (string and wsstring are no more than byte arrays) as a wstring using union for instance.

     

    Last edit: dFx 2022-01-27
    • damian177

      damian177 - 2022-01-27

      When I use .wsContent teh JSON Parse doesn't work.

      In JSON struct we not have a WSTRING data , only STRING. Please find in attachement.
      I use PRO_JSON library:
      https://forge.codesys.com/lib/pro-json/home/Home/

       

      Last edit: damian177 2022-01-27
  • damian177

    damian177 - 2022-01-27

    Someone tried Parse JSON from WSTRING by PRO_JSON library ?

     
  • tvm

    tvm - 2022-01-27

    Hi

    I created the PRO_JSON library. It's designed to use STRING types, but the functions refer to the strings like this:

    JSONString:     POINTER TO ARRAY[1..GPL_JSON.MAX_JSON_STRING] OF ASCII;
    

    where ASCII is just an enumeration of all ASCII characters (type BYTE).

    the code uses the array index, not pointers, to iterate over all the characters.

    So, I haven't tried this out, but it might be as simple as changing the ASCII enumeration to WORD type. That would change the array elements to 16 bit, and since WSTRING and STRING are the same for the first 127 characters anyway, anywhere the enumerations are used in the code would be the same.

    let me know if that works, I'm not sure if there's a way to allow for both WSTRING and STRING types, but I'll put it on the list of things to think about for future versions

    Tim

     
    • damian177

      damian177 - 2022-01-27

      In library is:

      {attribute 'displaymode':='hex'}JSONString:     POINTER TO ARRAY[1..GPL_JSON.MAX_JSON_STRING] OF BYTE;  //pointer to JSON string
      

      I should change to:

      {attribute 'displaymode':='hex'}JSONString:     POINTER TO ARRAY[1..GPL_JSON.MAX_JSON_STRING] OF WORD;  //pointer to JSON string
      

      Ho change it in library function ? It is no edited ...

       
      • tvm

        tvm - 2022-01-27

        Yes.
        You should be able to open the .library file and make the changes and then save it to your own library repository

         
  • tvm

    tvm - 2022-01-27

    Sorry, I just realize that I only implemented the ASCII enumeration in version 1.0.0.12, which I haven't released yet. In the version 1.0.0.11, the JSON strings are referred to as:

    {attribute 'displaymode':='hex'}JSONString:     POINTER TO ARRAY[1..GPL_JSON.MAX_JSON_STRING] OF BYTE;  //pointer to JSON string
    

    which might mean you just have to change the BYTE to a WORD

     
  • tvm

    tvm - 2022-01-27

    Thinking about it a bit more, there's probably some changes that would need to be made to the JSONVAR function block as well. Line 110 of JSON_TO_STRUCT is where the value from the JSON string is written to the local variable. It uses the JSONVAR.AsString method. It takes a STRING type, which in turn uses the STRING_TO_JSONVALUE function to guess at what type the value is.
    I think all of those points would also have to be changed.

     

    Last edit: tvm 2022-01-27
  • damian177

    damian177 - 2022-01-27

    For tests I will change only BYTE to WORD like you said, but in my STRUCT's (NewJSONObj) I won't use JSONVAR only WSTRING. And calling ParseJSON would be:

    ParseJSON(
            Execute:= EXECUTE_JSON_DATA,
            JSONString:= ADR(http_sys.hTTPClient_0.httpResult.wsContent),              
            JSONStringSize:= SIZEOF(http_sys.hTTPClient_0.httpResult.wsContent),     
            JSONVars:= ADR(NewJSONObj),
            NumberOfVars:= SIZEOF(NewJSONObj) / SIZEOF(WSTRING)
    );
    

    What do you think ? it enought ?

     

    Last edit: damian177 2022-01-27
    • tvm

      tvm - 2022-01-27

      I do not think this will work, because JSON_TO_STRUCT is expecting the JSONVars input to consist of only variables of JSONVAR type. It doesn't matter if they are a STRUCT or an ARRAY on the outside, but within the function block they are all addressed as an ARRAY OF JSONVAR.
      The more I look at it, I think it would require a substantial rewrite. Basically everything would have to be changed to WSTRING internally.
      Which might be something to look at in the future, but it's not a quick fix.

       
  • damian177

    damian177 - 2022-01-28

    I rewrited all Your library changing all STRING datatypes to WSTRING, and change functiona like LEN, DELETE etc to WLEN, WDELETE etc ... and partially works.

    It looks like that finde the tag's but it have a problem witch data lenght. For example it is my test data in WSTRING JSON:

    {"id":25,"Tag":"RRWWQQAAQQ","description":"ΕΕšΔ˜Γ³Ε›uΔ…QΓ³Ε›Εš","comment":"test1"}
    

    After parsing I have good data but not all length - please find in attachement ...
    Any idea what should i change ?

     
    • dFx

      dFx - 2022-01-28

      Feels like end of string detection isn't right. Maybe the lib was testing a byte, where you should test a word to NULL when using WSTRING.

       
      • tvm

        tvm - 2022-01-28

        line 41 of JSON_TO_STRUCT limits the loop to either finding a null character, or the total string size. Did you change the ASCII enumeration to WORD?

         
  • damian177

    damian177 - 2022-01-28

    You talking about this function ?:

    FUNCTION WSTRING_TO_JSONVALUE
    VAR_INPUT
        In_WString:     WSTRING;
        Out_JSON:       POINTER TO JSONVAR_16;
    END_VAR
    VAR
        TempBool:   BOOL;
        TempReal:   REAL;
        TempInt:    DINT;
        TempString: WSTRING(255);
    END_VAR
    
    
    
    IF (Out_JSON^.ValueType = JSONVALUETYPE_16.json_null) THEN
    //if the type is not explicitly configured, make a guess at the type
        IF ISINTEGER_16(ADR(In_WString)) THEN
            Out_JSON^.ValueType:= JSONVALUETYPE_16.json_integer;
        ELSIF ISNUMBER_16(ADR(In_WString)) THEN
            Out_JSON^.ValueType:= JSONVALUETYPE_16.json_number;
        ELSIF ISBOOLEAN_16(ADR(In_WString)) THEN
            Out_JSON^.ValueType:= JSONVALUETYPE_16.json_boolean;
        ELSIF (In_WString = "null") THEN
            Out_JSON^.ValueType:= JSONVALUETYPE_16.json_null;
        ELSE
            Out_JSON^.ValueType:= JSONVALUETYPE_16.json_wstring;
        END_IF
    END_IF
    
    CASE Out_JSON^.ValueType OF
    JSONVALUETYPE_16.json_null:
        Out_JSON^.Null:= "null";
    JSONVALUETYPE_16.json_boolean:
        StrToUpperA(pString:= ADR(In_WString));
        IF (In_WString = "TRUE") THEN
            TempBool:= TRUE;
        ELSE
            TempBool:= FALSE;
        END_IF
        Out_JSON^.Boolean:= TempBool;
    JSONVALUETYPE_16.json_integer:
        TempInt:= WSTRING_TO_DINT(In_WString);
        Out_JSON^.Integer:= TempInt;    
    JSONVALUETYPE_16.json_number:
        TempReal:= WSTRING_TO_REAL(In_WString);
        Out_JSON^.Number:= TempReal;
    JSONVALUETYPE_16.json_wstring:
        Out_JSON^.CharString:= In_WString;
    END_CASE
    
     
  • damian177

    damian177 - 2022-01-28

    When Iuse HMIVarAsWString variable is better, but still not full values in variables

     

    Last edit: damian177 2022-01-28
  • tvm

    tvm - 2022-01-28

    Try increasing these values in GPL_JSON

        MAX_NAME_SIZE:              UINT:= 20;      //max string length of JSON name
        MAX_VALUE_SIZE:             UINT:= 20;      //max value size in bytes (all values are this size in memory)
    

    These were basically only to limit memory usage. For example, MAX_VALUE_SIZE is the number of bytes that each JSONVAR uses for internal variable storage, regardless of type. So if all your JSON variables are booleans or numbers, it could probably be 4 bytes. In your case, with a lot of strings, it needs to be longer.

     
  • damian177

    damian177 - 2022-01-28

    I increased this values but without effect.
    Actually it is a number of WORD :

    NameStringPtr:      POINTER TO ARRAY[1..GPL_JSON_16.MAX_VAR_NAME] OF WORD;
    

    OR

    {attribute 'monitoring':='call'}PROPERTY AsWString : WSTRING(GPL_JSON_16.MAX_VALUE_SIZE)
    
     
  • tvm

    tvm - 2022-01-28

    Did you increase the values in the library, or in your project? Because I've noticed in the past that if I change a value in a parameter list in the library, it doesn't change in the project unless I do it there as well

     
  • damian177

    damian177 - 2022-01-28

    I do not can edit library. In my project create structs, fb, fun with with a "_ 16" note to the original name.
    My global variables : GPL_JSON_16 and ther I chcnege value.

     
    • tvm

      tvm - 2022-01-28

      OK, what about in JSONVAR

      _InternalValue: ARRAY[1..GPL_JSON.MAX_VALUE_SIZE+1] OF BYTE;
      

      should that also be changed to a WORD, or is it big enough anyway?

       
  • damian177

    damian177 - 2022-01-28

    According with described above logic, I created JSONVAR_16 and change like this:

    {attribute 'pack_mode' := '4'}
    {attribute 'reflection'}
    {attribute 'symbol' := 'none'}
    FUNCTION_BLOCK JSONVAR_16
    VAR_INPUT
        Names:          ARRAY[1..GPL_JSON_16.MAX_LEVELS] OF JSONVARNAME_16; //variable path as an array. [1] is variable name, [2..x] are higher level structures   
    END_VAR
    VAR
        {attribute 'instance-path'}{attribute 'noinit'}
        {attribute 'hide'}_VarNameFull:         WSTRING(GPL_JSON_16.MAX_VAR_NAME);                  //JSON name uses the variable name
        {attribute 'hide'}_ValueType:           JSONVALUETYPE_16:= JSONVALUETYPE_16.json_null;      //type of variable contained in the value field, also implicity defines size of local var
        {attribute 'hide'}_InternalValue:       ARRAY[1..GPL_JSON_16.MAX_VALUE_SIZE+1] OF WORD; //value stored internally as an array of bytes
        {attribute 'hide'}_Size:                UINT;                                       //size of internal value in bytes, automatically determined when any of the type properties are set
        {attribute 'hide'}_ApplicationLevel:    INT;                                        //array position which points to the application
    END_VAR
    VAR_OUTPUT
        {attribute 'symbol' := 'read'}HMIVarAsWString:  WSTRING(GPL_JSON_16.MAX_VALUE_SIZE); //this output is updated every time AsString property is set, for use with external HMI through sumbol configuration
    END_VAR
    VAR_STAT
        {attribute 'hide'}PrevApplicationLevel: REFERENCE TO INT;       //highest variable name in previous variable
        {attribute 'hide'}PrevNameArray:        REFERENCE TO ARRAY[1..GPL_JSON_16.MAX_LEVELS] OF JSONVARNAME_16;//path of the previous variable
    END_VAR
    
     
    • tvm

      tvm - 2022-01-28

      OK, I have another idea.
      Take a look in JSON_TO_STRUCT.Value, line 65-70

              //copy buffer to temporary value as a string
              SysMemCpy(
                  pDest:= ADR(NameValue[obj_level].Value), 
                  pSrc:= ADR(JSONString^[value_start]), 
                  udiCount:= MIN((value_stop+1) - value_start, GPL_JSON.MAX_VALUE_SIZE-1)
              );
      

      this copies bytes from the JSON string into a temporary buffer containing the name and value. The NAMEVALUE buffer should be fine, but the udiCount input to SysMemCpy is in bytes, and should be doubled, I think

              //copy buffer to temporary value as a string
              SysMemCpy(
                  pDest:= ADR(NameValue[obj_level].Value), 
                  pSrc:= ADR(JSONString^[value_start]), 
                  udiCount:= MIN(((value_stop+1) - value_start) * 2, GPL_JSON.MAX_VALUE_SIZE-1)
              );
      
       
  • damian177

    damian177 - 2022-01-28

    it's better with this change. In HMIVarAsWStrin I have all data from variable, but in AsWString is still to small.
    Please find in attachement.

     

    Last edit: damian177 2022-01-28
    • tvm

      tvm - 2022-01-28

      What about property JSONVAR.CharString?

      In your new function WSTRING_TO_JSONVALUE, you still have this line. Did CharString get changed to WSTRING type?

      JSONVALUETYPE_16.json_wstring:
          Out_JSON^.CharString:= In_WString;
      
       
1 2 > >> (Page 1 of 2)

Log in to post a comment.