Welcome to our new forum
All users of the legacy CODESYS Forums, please create a new account at account.codesys.com. But make sure to use the same E-Mail address as in the old Forum. Then your posts will be matched. Close

CSV Writer Speed

2020-01-31
2020-02-11
  • vernon.laurence

    vernon.laurence - 2020-01-31

    Hello - There are a large number of data points which need to be added to a CSV file. There is an indexing routine which cycles through all of the data points in the array and uses the CSV.AddString FB to add the strings to the buffer. When the buffer write is complete the items are saved to file using the CSV.WriteFile FB. Everything works as expected. The problem is that the process is extremely slow. The system takes 5 minutes to complete 5000 data points. The customer is requesting 45,000 data points.

    I have taken the following steps to increased speed:
    1. Change the write to a separate task with a cyclic time of 2ms
    2. Changed the logic to only perform the WriteFile after adding 100 data points (cannot do all of the data points at once due to buffer size)
    3. Tried to perform an CSV.AddString during every cycle without paying attention to the done from the FB. This did not report the data correctly.

    Attached is the code I am using to process the data. Any advice on a different, faster approach would be much appreciated. Thank you for your time.

    FUNCTION_BLOCK TestDataProcessorLREAL
    VAR_INPUT
       bExecute: BOOL;
       InputArray             :    ARRAY[0..50000] OF LREAL;
       uiSampleNumber         :   UINT;
    END_VAR
    VAR_IN_OUT
       cCSVWriter      : CSV.CSVWriter;
    END_VAR
    VAR_OUTPUT
       bDone: BOOL;
    END_VAR
    VAR
       i: UINT;
       sTempString          : STRING;
       csvAddTempString       : CSV.AddSTRING;
       csvWriteFile         : CSV.WriteFile;
       bcsvAddStringExecute   :   BOOL;
       bcsvWriteFileExecute   : BOOL;
    END_VAR
    IF bExecute AND NOT bDone THEN
       bcsvAddStringExecute := TRUE;
       
       //Convert the input data to string to be stored by the CSV writer
       sTempString  := LREAL_TO_STRING(InputArray[i]);
       //Check to see if the previous string Add is complete, and there is not a file write requested and then reset the add string bit and index ahead 1
       IF csvAddTempString.xDone AND NOT bcsvWriteFileExecute THEN
          bcsvAddStringExecute :=FALSE;
          i:=i+1;
       END_IF
       //Only call the Write file every 100 calls of the add string FB, this was done to increase speed
       IF (i MOD 100) = 0 AND i >0 AND csvAddTempString.xDone THEN
          bcsvWriteFileExecute :=TRUE;
       END_IF
       
    //When the write file is complete, turn off the Execute bit
       IF csvWriteFile.xDone THEN
          bcsvWriteFileExecute := FALSE;
       END_IF
       
    //These calls put the data points into the buffer and then write it to the file 
    CSVAddTempSTRING(sInput:=sTempString,rCSVWriter:=cCSVWriter,xExecute:=bcsvAddStringExecute);
    csvWriteFile(xExecute:=csvAddTempString.xDone,rCSVWriter:=cCSVWriter);
    END_IF
    //Reset the index and stop execution if 
    IF NOT bExecute THEN
       bcsvAddStringExecute :=FALSE;
       i := 0;
       bDone := FALSE;
    END_IF
    //Set the DONE bit when the end of array index is reached
    IF i >= uiSampleNumber AND bExecute THEN
       bDone := TRUE;
    END_IF
    
     
  • sgronchi

    sgronchi - 2020-02-04

    You are inserting one point per task cycle, so if you have a 5 ms cycle time, it will take 250 seconds (4 minutes and 10 seconds) to insert 50k points.
    How about a FOR loop calling CSVAddTempSTRING 100 times in a single cycle task? I would bet you will speed up operations by a similar factor (down to about 3 seconds).
    Furthermore, IIRC VAR_INPUT mandates a copy of array is made upon each invocation. Try passing the array as VAR_IN_OUT CONSTANT to avoid copies.

     
  • vernon.laurence

    vernon.laurence - 2020-02-05

    Gronchi - Thank you for the response. Unfortunately it does not work to call the AddString FB (instance name is CSVAddTempString) repeatedly. I tried this method in the beginning. You have to wait for the Done bit of the FB before updating the input and calling it again. Otherwise it does not finish writing the data to the buffer and a lot of data goes missing.

    You are correct that there is a better way to pass the data around: a pass by reference would probably be best. However, there is no memory shortage so I haven't tackled this yet.

     
  • eschwellinger

    eschwellinger - 2020-02-05

    Hi,
    on which plc is this?

    BR
    Edwin

     
  • dFx

    dFx - 2020-02-05

    I don't get this :

    bcsvWriteFileExecute :=TRUE;
    
    bcsvWriteFileExecute :=FALSE;
    
    csvWriteFile(xExecute:=csvAddTempString.xDone,rCSVWriter:=cCSVWriter);
    

    Feels like it is still writing every cycle (beging that CSVAddTempSTRING is a synchronous fb).

     
  • vernon.laurence

    vernon.laurence - 2020-02-05

    dFx hat geschrieben:
    I don't get this :

    bcsvWriteFileExecute :=TRUE;
    
    bcsvWriteFileExecute :=FALSE;
    
    csvWriteFile(xExecute:=csvAddTempString.xDone,rCSVWriter:=cCSVWriter);
    

    Feels like it is still writing every cycle (beging that CSVAddTempSTRING is a synchronous fb).

    That was another attempt to speed it up, although probably not to much effect. In the beginning I was trying to stay away from a state machine as that would inherently add one more scan to each change of state. However this became hard to manage, so i re-wrote the code in a CASE statement. It is much easier to follow now.... with one major change, I am concatenating a series of the data points into a single string and manually adding the commas. In this way, the WriteFile still has each data point separated by a comma but significantly reduces the number of AddString calls that need to be made. The limit on this is the CONCAT function can only take 255 characters at one time. It is much faster now, but still not fast enough. What I probably need to do is allocate an area of memory which is as large as the buffer for the CSV writer and fill that completely, then write to file. This should be fast enough.

    IF  NOT bExecute  THEN
       step_number := 0;
       bDone := FALSE;
    END_IF
       CASE step_number OF
          0: 
             i:=0;
             IF bExecute THEN
                step_number := 10;
             END_IF
          10:
             i_step := i + 19;
             sTempString := '';
             FOR i := i TO i_step DO
                sTempString  := Concat(STempString, Concat(LREAL_TO_STRING(InputArray[i]), ','));
             END_FOR
             sTempString  := Concat(STempString, LREAL_TO_STRING(InputArray[i]));
             step_number := 20;
          20:
             bcsvAddStringExecute := TRUE;
             IF csvAddTempString.xDone THEN
                bcsvAddStringExecute := FALSE;
                step_number := 30;
             END_IF
          30:
             IF (i MOD 100) = 0 THEN
                bcsvWriteFileExecute :=TRUE;
                step_number := 40;
             ELSE
                step_number := 10;
             END_IF
          40:
             IF csvWriteFile.xDone THEN
                bcsvWriteFileExecute :=FALSE;
                IF i >= uiSampleNumber THEN
                   bDone := TRUE;
                   ELSE
                   step_number := 10;   
                END_IF
             END_IF
          
             
          
       END_CASE
       
       
       
       
    //These calls put the data points into the buffer and then write it to the file 
    CSVAddTempSTRING(sInput:=sTempString,rCSVWriter:=cCSVWriter,xExecute:=bcsvAddStringExecute);
    csvWriteFile(xExecute:=bcsvWriteFileExecute,rCSVWriter:=cCSVWriter);
    
     
  • vernon.laurence

    vernon.laurence - 2020-02-05

    Edwin Schwellinger hat geschrieben:
    Hi,
    on which plc is this?
    BR
    Edwin

    This is on a Parker Automation Controller.

     
  • dFx

    dFx - 2020-02-11

    Strings are no more than byte arrays ending with a 16#0 value. To reduce your treatment, just manage by yourself the length and offset for new string write inside a byte buffer. OSCAT lib may help for this.

    Also you are still writing the csv format (with comas), you could aswell handle the cariage return and line feed.

    Then, last but not least, you could try opening the file once, and then only writing the new data to it before closing it once done. This could be way more efficient thant opening and closing the file at each write.

     

Log in to post a comment.