Transferring Large CSV File to the Cloud

dgirard
2021-11-29
2022-01-03
  • dgirard - 2021-11-29

    Good day

    We need to transfer 100 MB CSV files stored locally on the controller up to the Cloud through our REST API. What would be the preferred method to do this using Codesys? We are exploring the IIoT Library SL from the Codesys Store at the moment. We can adapt the REST API to best match the Codesys implementation.

    Can we directly upload a CSV to our API using HTTP PUT with the WebClient SL from the IIoT library? Are there better alternative we should consider?

    Any ideas are much appreciated.

    Thank you

     
  • i-campbell

    i-campbell - 2021-11-29

    100Mb is relatively large, but I can't see why it wouldn't support it though. Just edit the WEB_CLIENT.Param.g_udiMaxRequestSize to something like 112000000.

     
  • nothinrandom - 2022-01-03

    @dgirard,

    Could you provide more details such as your cloud provider, which service of your cloud provider is going to store this CSV file, etc? With more details, you might not even need the IIoT library at all to make this transfer. For example, let's take AWS as cloud provider with S3 is the storage solution. You just need to follow these steps:

    1. Install onto the device:
       1. AWS CLI client (windows/linux) 
       2. AWS simple system manager (linux is preferred).  This is to provide the short live cloud access token
    2. Attach S3 put permission to the IAM role that is used for the device
    3. In Codesys, use syscommand to execute a s3 put (e.g. `aws s3 cp /path/to/myFile.csv s3://bucketName/path/myFile.csv)` 
    

    Google Cloud and Microsoft Azure have similar solutions, so you could do the same there.

     

Log in to post a comment.