AC Datalog on a RPi 3 buffer trouble

pruwetbe
2020-10-05
2020-10-21
  • pruwetbe

    pruwetbe - 2020-10-05

    I configured recently the AC Datalog on my home heating management application that will run on a RPi 3.

    My idea is to store some values every 5 min for Analog values and use the event logging for Boolean ...
    This Datalog generate a lot of red message saying that the buffer is full at each data log in the message log on the target.
    What can I do to avoid this?
    Csv file parameter has a stand buffer size of 2000 bytes and even triggering the write at 50% rather than 75% does not help.

    Thank you for any idea to solve this.

     
  • Morberis

    Morberis - 2020-10-06

    I'm sorry I can't seem to recreate this, or haven't yet. I am using a Rpi 3 B+ 1gb ram with a 32gb class 10 sdhc card that has 16gb free. I'm using the standard buffer size of 2000 bytes, write is triggered at 75%. I do have i2c and spi enabled. Also the latest codesys package for the Rpi but only version 10 of rasbian buster.

    How many analog variables and of what type are you recording? You might need to increase the buffer size.

    Right now I'm logging 10 real variables and 10 boolean variables.

     

    Last edit: Morberis 2020-10-06
  • pruwetbe

    pruwetbe - 2020-10-07

    Hello
    Thanks for your feedback .
    My first version of Datalog config was logging 4 real and 2 Boolean but I discovered that the HVAC library I use for pump and valve management has its own Datalog configuration that I cannot deactivate so the number is quite higher.

    In the second version, I configured two Datalog channels, one event and one cyclic
    I Created the whole IO list of my future app From excel sheet and put all real value I a cyclic 5min ( that is triggered every 15s now for testing purpose). I have 43 real values logged.
    I put all Boolean and HVAC stuff in the event channel ( so recorded at startup and then on change)
    The recording itself seems working properly for the cyclic data channel, but event channel was recorded only at startup.
    I mean that the file Date and size on the RPi did not change... ( could be because there was not enough change to reach the buffer limit triggering a write on the disk)
    Unfortunately, the runtime was stopped after half an hour due to exception ...
    So data( if recorded in buffer ) where lost...
    I think that the error messages were already present with the first version but the number or error message was smaller.
    I mean that it seems that this message come multiple times with roughly same timestamp.

    Do you know how to export journal of error to a text file?
    I am running on a RPi 3+ with SD only ... I will check the exact rasbian distribution but it was installed in beginning of this year.

    More details by mail this evening...

     
    • pruwetbe

      pruwetbe - 2020-10-07

      Hello

      here is the archive of my testing project combining investigation on
      datalog / trends / traces.

      J’ai lié 1 fichier à ce message :
      Gestion chauffage H3 V1.0.projectarchive
      https://we.tl/t-JRoG39HdQC(128 Mo)WeTransferhttps://we.tl/t-JRoG39HdQC
      Mozilla Thunderbird https://www.thunderbird.net permet de partager
      facilement des fichiers volumineux.

      i also copy the log file.

      Thanks for your help.

      Le 07-10-20 à 08:13, pruwetbe a écrit :

      Hello
      Thanks for your feedback .
      My first version of Datalog config was logging 4 real and 2 Boolean
      but I discovered that the HVAC library I use for pump and valve
      management has its own Datalog configuration that I cannot deactivate
      so the number is quite higher.

      In the second version, I configured two Datalog channels, one event
      and one cyclic
      I Created the whole IO list of my future app From excel sheet and put
      all real value I a cyclic 5min ( that is triggered every 15s now for
      testing purpose). I have 43 real values logged.
      I put all Boolean and HVAC stuff in the event channel ( so recorded at
      startup and then on change)
      The recording itself seems working properly for the cyclic data
      channel, but event channel was recorded only at startup.
      I mean that the file Date and size on the RPi did not change... (
      could be because there was not enough change to reach the buffer limit
      triggering a write on the disk)
      Unfortunately, the runtime was stopped after half an hour due to
      exception ...
      So data( if recorded in buffer ) where lost...
      I think that the error messages were already present with the first
      version but the number or error message was smaller.
      I mean that it seems that this message come multiple times with
      roughly same timestamp.

      Do you know how to export journal of error to a text file?
      I am running on a RPi 3+ with SD only ... I will check the exact
      rasbian distribution but it was installed in beginning of this year.

      More details by mail this evening...


      AC Datalog on a RPi 3 buffer trouble
      https://forge.codesys.com/forge/talk/Engineering/thread/a4c6c530c5/?limit=25#a41f


      Sent from forge.codesys.com because you indicated interest in
      https://forge.codesys.com/forge/talk/Engineering/

      To unsubscribe from further messages, please visit
      https://forge.codesys.com/auth/subscriptions/

       
  • Morberis

    Morberis - 2020-10-09

    Your buffer needs to be bigger. I just made it 20000 instead of 2000 for both and it worked fine. So you'll need to play around with that.

     
    • pruwetbe

      pruwetbe - 2020-10-09

      Good news , some times the simplist idea are working the best...
      So the preset size of the buffer is really too small...
      I will try this tomorrow morning !
      Thanks a lot...
      Now I have the licences RPi and HVAC working on a dongle so I can test without having my runtime stopping after 30 min...
      I will check the options and macros, thanks for the indications.

      Pierre RUWET

      Le 9 oct. 2020 à 19:44, Morberis forge@codesys.com a écrit :

      
      Your buffer needs to be bigger. I just made it 20000 instead of 2000 for both and it worked fine. So you'll need to play around with that.

      AC Datalog on a RPi 3 buffer trouble

      Sent from forge.codesys.com because you indicated interest in https://forge.codesys.com/forge/talk/Engineering/

      To unsubscribe from further messages, please visit https://forge.codesys.com/auth/subscriptions/

       
      👍
      1
  • pruwetbe

    pruwetbe - 2020-10-11

    Hello Morberis,
    after get datalog running without error, the next step was to check how to create flat CSV (one tag per column with common time stamp sampling) sothat I can use the nice and easy free tools (datplot) to create plot.
    After some search on pythons,that was finally more easy than I thought.
    Basically, Pandas library give very powerfull function for resampling time series.
    If you are interested, I can share such python script.

    The more difficult part was on codesys side again. For a reason, I did not catch, up to now, the hashfile containing the Tag list with tag ID and format stopped to be updated by the datalog configuration.
    I tried a lot of different changes in the datalog definition to force this file to be updated.
    The crazy stuff is that the log file indicate that the file is created with success but when looking to the RPI datalog directory, the old hashfile stayed unchanged.
    Even when i finally deactivate the hash file option, the CSV file never include again the tags property.
    The datalog config txt file in the config subdirectory follow perfectly the configuration done and the logging itself is working but it is difficult to find the ID when the hashfile is not updated...
    I finally tried after so many unsuccessfull trial to start from an old backup and could get one hashfile again but when updating my codesys program and datalog code with the code from the last version, i got again no creation the datalog.

    Any idea?

     

    Last edit: pruwetbe 2020-10-11
  • pruwetbe

    pruwetbe - 2020-10-21

    Hello Morberis,
    The only solution I found to solve this strange problem was to remove completely all the Datalog definition from my codesys project and then recreate them from scratch and now it is working normally.
    From now, each time I will change something in my software, I will check if hashfile are properly updated before to go further step.
    Thanks for your help on these subject, now I can go back to my main subject, continue to develop the app with simulation and be able to évaluation the perfect function with trace, trends and Datalog.

     

Log in to post a comment.