CSV Writer Speed

2020-02-27T15:05:52.703000Z

Strings are no more than byte arrays ending with a 16#0 value. To reduce your treatment, just manage by yourself the length and offset for new string write inside a byte buffer. OSCAT lib may help for this.

Also you are still writing the csv format (with comas), you could aswell handle the cariage return and line feed.

Then, last but not least, you could try opening the file once, and then only writing the new data to it before closing it once done. This could be way more efficient thant opening and closing the file at each write.

CSV Writer Speed

2020-02-27T15:05:51.988000Z

Edwin Schwellinger hat geschrieben:
Hi,
on which plc is this?
BR
Edwin

This is on a Parker Automation Controller.

CSV Writer Speed

2020-02-27T15:05:50.924000Z

dFx hat geschrieben:
I don't get this :

bcsvWriteFileExecute :=TRUE;

bcsvWriteFileExecute :=FALSE;

csvWriteFile(xExecute:=csvAddTempString.xDone,rCSVWriter:=cCSVWriter);

Feels like it is still writing every cycle (beging that CSVAddTempSTRING is a synchronous fb).

That was another attempt to speed it up, although probably not to much effect. In the beginning I was trying to stay away from a state machine as that would inherently add one more scan to each change of state. However this became hard to manage, so i re-wrote the code in a CASE statement. It is much easier to follow now.... with one major change, I am concatenating a series of the data points into a single string and manually adding the commas. In this way, the WriteFile still has each data point separated by a comma but significantly reduces the number of AddString calls that need to be made. The limit on this is the CONCAT function can only take 255 characters at one time. It is much faster now, but still not fast enough. What I probably need to do is allocate an area of memory which is as large as the buffer for the CSV writer and fill that completely, then write to file. This should be fast enough.

IF  NOT bExecute  THEN
   step_number := 0;
   bDone := FALSE;
END_IF
   CASE step_number OF
      0: 
         i:=0;
         IF bExecute THEN
            step_number := 10;
         END_IF
      10:
         i_step := i + 19;
         sTempString := '';
         FOR i := i TO i_step DO
            sTempString  := Concat(STempString, Concat(LREAL_TO_STRING(InputArray[i]), ','));
         END_FOR
         sTempString  := Concat(STempString, LREAL_TO_STRING(InputArray[i]));
         step_number := 20;
      20:
         bcsvAddStringExecute := TRUE;
         IF csvAddTempString.xDone THEN
            bcsvAddStringExecute := FALSE;
            step_number := 30;
         END_IF
      30:
         IF (i MOD 100) = 0 THEN
            bcsvWriteFileExecute :=TRUE;
            step_number := 40;
         ELSE
            step_number := 10;
         END_IF
      40:
         IF csvWriteFile.xDone THEN
            bcsvWriteFileExecute :=FALSE;
            IF i >= uiSampleNumber THEN
               bDone := TRUE;
               ELSE
               step_number := 10;   
            END_IF
         END_IF
      
         
      
   END_CASE
   
   
   
   
//These calls put the data points into the buffer and then write it to the file 
CSVAddTempSTRING(sInput:=sTempString,rCSVWriter:=cCSVWriter,xExecute:=bcsvAddStringExecute);
csvWriteFile(xExecute:=bcsvWriteFileExecute,rCSVWriter:=cCSVWriter);

CSV Writer Speed

2020-02-27T15:05:48.777000Z

I don't get this :

bcsvWriteFileExecute :=TRUE;

bcsvWriteFileExecute :=FALSE;

csvWriteFile(xExecute:=csvAddTempString.xDone,rCSVWriter:=cCSVWriter);

Feels like it is still writing every cycle (beging that CSVAddTempSTRING is a synchronous fb).

CSV Writer Speed

2020-02-27T15:05:48.146000Z

Hi,
on which plc is this?

BR
Edwin

CSV Writer Speed

2020-02-27T15:05:47.529000Z

Gronchi - Thank you for the response. Unfortunately it does not work to call the AddString FB (instance name is CSVAddTempString) repeatedly. I tried this method in the beginning. You have to wait for the Done bit of the FB before updating the input and calling it again. Otherwise it does not finish writing the data to the buffer and a lot of data goes missing.

You are correct that there is a better way to pass the data around: a pass by reference would probably be best. However, there is no memory shortage so I haven't tackled this yet.

CSV Writer Speed

2020-02-27T15:05:45.802000Z

You are inserting one point per task cycle, so if you have a 5 ms cycle time, it will take 250 seconds (4 minutes and 10 seconds) to insert 50k points.
How about a FOR loop calling CSVAddTempSTRING 100 times in a single cycle task? I would bet you will speed up operations by a similar factor (down to about 3 seconds).
Furthermore, IIRC VAR_INPUT mandates a copy of array is made upon each invocation. Try passing the array as VAR_IN_OUT CONSTANT to avoid copies.

CSV Writer Speed

2020-02-27T15:05:45.096000Z

Hello - There are a large number of data points which need to be added to a CSV file. There is an indexing routine which cycles through all of the data points in the array and uses the CSV.AddString FB to add the strings to the buffer. When the buffer write is complete the items are saved to file using the CSV.WriteFile FB. Everything works as expected. The problem is that the process is extremely slow. The system takes 5 minutes to complete 5000 data points. The customer is requesting 45,000 data points.

I have taken the following steps to increased speed:
1. Change the write to a separate task with a cyclic time of 2ms
2. Changed the logic to only perform the WriteFile after adding 100 data points (cannot do all of the data points at once due to buffer size)
3. Tried to perform an CSV.AddString during every cycle without paying attention to the done from the FB. This did not report the data correctly.

Attached is the code I am using to process the data. Any advice on a different, faster approach would be much appreciated. Thank you for your time.

FUNCTION_BLOCK TestDataProcessorLREAL
VAR_INPUT
   bExecute: BOOL;
   InputArray             :    ARRAY[0..50000] OF LREAL;
   uiSampleNumber         :   UINT;
END_VAR
VAR_IN_OUT
   cCSVWriter      : CSV.CSVWriter;
END_VAR
VAR_OUTPUT
   bDone: BOOL;
END_VAR
VAR
   i: UINT;
   sTempString          : STRING;
   csvAddTempString       : CSV.AddSTRING;
   csvWriteFile         : CSV.WriteFile;
   bcsvAddStringExecute   :   BOOL;
   bcsvWriteFileExecute   : BOOL;
END_VAR
IF bExecute AND NOT bDone THEN
   bcsvAddStringExecute := TRUE;
   
   //Convert the input data to string to be stored by the CSV writer
   sTempString  := LREAL_TO_STRING(InputArray[i]);
   //Check to see if the previous string Add is complete, and there is not a file write requested and then reset the add string bit and index ahead 1
   IF csvAddTempString.xDone AND NOT bcsvWriteFileExecute THEN
      bcsvAddStringExecute :=FALSE;
      i:=i+1;
   END_IF
   //Only call the Write file every 100 calls of the add string FB, this was done to increase speed
   IF (i MOD 100) = 0 AND i >0 AND csvAddTempString.xDone THEN
      bcsvWriteFileExecute :=TRUE;
   END_IF
   
//When the write file is complete, turn off the Execute bit
   IF csvWriteFile.xDone THEN
      bcsvWriteFileExecute := FALSE;
   END_IF
   
//These calls put the data points into the buffer and then write it to the file 
CSVAddTempSTRING(sInput:=sTempString,rCSVWriter:=cCSVWriter,xExecute:=bcsvAddStringExecute);
csvWriteFile(xExecute:=csvAddTempString.xDone,rCSVWriter:=cCSVWriter);
END_IF
//Reset the index and stop execution if 
IF NOT bExecute THEN
   bcsvAddStringExecute :=FALSE;
   i := 0;
   bDone := FALSE;
END_IF
//Set the DONE bit when the end of array index is reached
IF i >= uiSampleNumber AND bExecute THEN
   bDone := TRUE;
END_IF

Recent posts to CSV Writer Speed

CSV Writer Speed

CSV Writer Speed

CSV Writer Speed

CSV Writer Speed

CSV Writer Speed

CSV Writer Speed

CSV Writer Speed

CSV Writer Speed