Hi everyone,
I am trying to save data to disk directly in csv format, the reason is that I want the operator to extract the external USB drive without need to reconstruct the file from binary to csv, I am using the SysFileAsynch23 in C3.5 SP6 .
There is a small problem with linux OS (this is an EPEC HMI) where the file handling system call does not write data immediately to disk but place it on a buffer and then write to file when the buffer is full or the file handler is closed.
This result in a problem when the machine (this is installed on a mobile machine) suddenly lose power so all the data placed in this local buffer and not yet wrote to disk will be lost forever.
In the same talking I don`t want to physically write to file every time I write a sample (open file handler, write and close) so not so short the lifetime of the memory.
The way I have adjusted to this is using a local buffer (local to C3.5 SP6) made of retain memory, so data will always kept in the application software.
Once the buffer is full (100 samples) I will open/write/close to file and everything works smothly.
The only problem I have is that, in order to write directly to csv, I need to use a buffer (array of string) of fixed length, where the length of the string has to allow for the longest string possible (i.e. I want to record a pressure as 4 digits 0-9999psi but when the pressure i 0 I will need to record 0000 which is wasting 3 extra bytes).
On the long run this will result in a big waste of memory.
I had an idea of using a buffer of struct to memorize the samples, then when this is full, to write to disk using a for loop where I reconstruct each sample in a string value and write it to disk, then I close the file handler to physically write data to memory.
The problem in this case is that, using the Asynch function, each sample will take at least one cycletime, so with a low cycletime of 20ms and writing 100 sample, writing the buffer to disk will take up to 2 seconds!!
Is there any possible alternative to save both in time and space but still writing directly in string format and not in binary??
Thank you to everyone who will share.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Something to try with your async method is to put that part of the program on its own task. I ran into the same issue you did where it would take a really long time to write to file because of the cycle time. We ended up drastically reducing that time by putting that part of the program on its own task and setting that task type to "Status". When we were ready to execute the program we would change the status variable to true and that task would freewheel until you were done and set the variable to false. Make sure you set the priority value > than more important tasks so that it doesn't lock up the system. Maybe this type of setup would also work for you.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi everyone,
I am trying to save data to disk directly in csv format, the reason is that I want the operator to extract the external USB drive without need to reconstruct the file from binary to csv, I am using the SysFileAsynch23 in C3.5 SP6 .
There is a small problem with linux OS (this is an EPEC HMI) where the file handling system call does not write data immediately to disk but place it on a buffer and then write to file when the buffer is full or the file handler is closed.
This result in a problem when the machine (this is installed on a mobile machine) suddenly lose power so all the data placed in this local buffer and not yet wrote to disk will be lost forever.
In the same talking I don`t want to physically write to file every time I write a sample (open file handler, write and close) so not so short the lifetime of the memory.
The way I have adjusted to this is using a local buffer (local to C3.5 SP6) made of retain memory, so data will always kept in the application software.
Once the buffer is full (100 samples) I will open/write/close to file and everything works smothly.
The only problem I have is that, in order to write directly to csv, I need to use a buffer (array of string) of fixed length, where the length of the string has to allow for the longest string possible (i.e. I want to record a pressure as 4 digits 0-9999psi but when the pressure i 0 I will need to record 0000 which is wasting 3 extra bytes).
On the long run this will result in a big waste of memory.
I had an idea of using a buffer of struct to memorize the samples, then when this is full, to write to disk using a for loop where I reconstruct each sample in a string value and write it to disk, then I close the file handler to physically write data to memory.
The problem in this case is that, using the Asynch function, each sample will take at least one cycletime, so with a low cycletime of 20ms and writing 100 sample, writing the buffer to disk will take up to 2 seconds!!
Is there any possible alternative to save both in time and space but still writing directly in string format and not in binary??
Thank you to everyone who will share.
Something to try with your async method is to put that part of the program on its own task. I ran into the same issue you did where it would take a really long time to write to file because of the cycle time. We ended up drastically reducing that time by putting that part of the program on its own task and setting that task type to "Status". When we were ready to execute the program we would change the status variable to true and that task would freewheel until you were done and set the variable to false. Make sure you set the priority value > than more important tasks so that it doesn't lock up the system. Maybe this type of setup would also work for you.