On a Schneider M262, when monitoring CPU load with SchedGetProcessorLoad, it seems to run at around 25-45% during most tasks, occasionally jumping up to about 60% with some visualizations when loading.
I have some very small trends that I am only storing about 7 days data on, but multiple times now when viewing trends my PLC has gone into a Processor Load Watchdog fault.
Sometimes it works no drama, running back to the start of the 7 days, changing span, etc.
Then sometimes, like just now, all I tried to do was change the span to 1hr, and it crashed.
Is there something that can be done about this? Why would looking at a trend cause such a high load? What are my options for avoiding this???
I have a similar situation. I'm not using the built in trends, I created my own library for that, but I am using M262 PLCs with web visu.
I haven't been able to narrow down the problem, but I have seen the same thing with Processor Load watchdog at seemingly random times. My best guess at the moment is something to do with the interaction between file access and the visu, but that's still just a guess.
I've also seen it happen a few times where the PLC stays running , but the http server stops working and only the https works. I'll sometimes see in the logs '/usr/visu failed'. I'm not 100% sure that this is the same problem.
A few questions/things to check:
1. What browser are you using? Does this happen with HMIs or just computer browsers? I know there's issues with the older firmware on the Schneider HMISTW panels which would cause the M262 web server to stop working. We actually stopped using Schneider HMIs, we've been pretty happy with the Phoenix WP series.
2. Which M262 model?
3. What firmware on the M262?
4. Any other file access functions working?
5. What's the interval and priority on your tasks?
6. What's the update rate and default communication buffer size of your visualization? https://faq.codesys.com/pages/viewpage.action?pageId=112525371
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
(Sorry for the late reply, Codesys Forge was down for a few days?)
It did seem to me like some kind of file accessing issue... That sucks though, as I guess for now I just need to remove the trends because the plant that it is running can't afford to stop as it may not be noticed and has the potential to destroy a lot of monies worth of product. I was talking to the client about getting another PLC so that we could split the process & the visu/file handling, that way at least if there is a crash the process can continue in the background. However this is really something you shouldn't have to spend money on when the product should be able to handle the process.
To answer your questions:
1. We are using the Schneider HMI's, however, I have had the crash happen when looking at the trend from my PC's browser, so I'm not sure that matters. Last time it crashed on Chrome, but I'm quite sure I have had it crash on Edge too.
2. TM262L10MESE8T
3. Firmware V5.1.7.11
4. I am creating and editing CSV files (creating reports) but when this is happening, it is only about once an hour, and it doesn't seem to line up at all with the crashes as far as I can tell.
5. I have the following Tasks:
- MAST (all code) 50ms, Priority 15
- IO_Scanning (no code, only used to scan bus), 300ms, Priority 30
- AlarmManagerTask, 200ms, Priority 20
- TrendRecordingTask, 1000ms, Priority 25
- VISU_TASK, 10ms, Priority 10
6. Update rate 40ms, Buffer size 50000 - Reading your attachment I see I might have all these Visualization values a bit low... They used to be higher but as you mentioned about the Schneider HMI's they are just so slow updating pages - I just kept lowering these values, it seemed like it helped but maybe I need to raise them back up...
If you have any suggestions or guides on what all the rates work well at I am all ears, It is probably not much more than guess work for me atm.
The other thing I did when trying to speed up the visuals was lower the Controllers monitoring interval to 100ms, which I believe also increases Load... Perhaps this is something I need to revert?
I guess the issue is as I said, regardless if these values are any good, the Load is really not that high until that one out of tenth time you go to check a trend...
I have also raised a case with Schneider (Although in the past I have found this forum much more helpful) - I'll see where that gets me.
Ben
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I had an issue a couple years ago with the HMISTW panels using HMI FW version 1.0, where all the web visus could stop working (both touch screen and computer). Upgrading to 1.1 fixed it, but I always thought it was more likely a server issue. Which is the only reason I bring it up here, it might not have anything to do with anything.
I'm using the M262 L10 and L20, and I've seen it on both. I have one job with an M35, and I've never seen it there. But we don't specialize in any one particular industry--all my jobs are custom, so while I use the same libraries, every project is very different.
Most of my task cycles are something like:
-MAST: 200mS (15)
-File task: 50-100mS (20) (I've experimented with different speeds and priorities)
-Network task: 20-100mS, depending on the network I'm using (various priorities, also depending on the network)
-Visu task: 200mS (lowest priority 31)
I've been using an update rate of 200mS
I did the same as you at one point, increasing the VISU_TASK as well as the update rate. I've found a visu task of >200mS to be too slow even with the Phoenix HMIs, but <100mS I started to see processor load go way up
I also have the same thing with the load usually being good, until that one condition in two weeks when it crashes.
One thing we've theorized about is the quality of the network connection. Two reasons:
-We have one job where the CAT6 cabling between the PLC and the nearest switch (for the record, by another contractor) is almost 150m long, and very poor workmanship. We've noticed a lot of crashes with that particular installation. (similar to what you're describing) But it's hard to consistently reproduce.
-We had a problem for years on another job, where the PLC would crash at random times. We eventually traced it back to a network scan that the IT department was doing every couple weeks (scanning for viruses, compromised computers, etc). The scan was hammering the PLC so hard that it just couldn't handle all the requests. Schneider updated the M262 firmware a couple times to address this, and I think it's fixed, but I also set up the router to block requests from the scanning computer, mostly because I had spent way too much time working on it and I just wanted it to go away.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Wow, looks like you've had a lot of experience dealing with these kinds of things!
I believe the HMI's are the latest firmware, but I do not have one on hand to check. I'm quite certain I remember needing to upgrade them towards the beginning of the project for something though. I will confirm.
Hmm the M35, well I don't need motion so that seems a little overkill but it is interesting to know.... Perhaps it is something I can share with Schneider if needed, perhaps there are differences they will be able to work out themselves.
OK, thanks for those scan times, I might look at increasing those times again, and I might also look at increasing the MAST time too, it really doesn't need to be all that fast.
Hmm, interesting on the network connection. The last crash I was connected remotely using a VPN, however I have had it crash on my test bench with a 1 meter lead connected directly too, so I'm not so sure....
It sounds like my process is a lot simpler than many of yours, it is just out on orchids, so there are no IT things happening or special processes - It is just critical that it stay online as it is for the Refrigeration of long stored fruit.
The code & visuals are all very simple, I would say some of the simplest code I have written, except for the only "complicated" part being the CSV file reporting & some emailing.
This trend thing seems to be the last thing creating issues. Which is a real pain as I don't think of trends as being something that should be so difficult, I've worked with Trends on Citect for many years and never had any troubles, it definitely seems like an M262 problem....
Thanks
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
We have feedback from R&D that this issue has already been reported globally and they are working on resolving it. It should be fixed with the integration of CoDeSys V3.5.SP19 in Machine Expert V2.2 planned for December 2023.
This is the reference number for the fix/change request. PEP1059101R: JMT-M262-AUS-Processor Load Watchdog when loading trend in WebVisu
In order to avoid the crash, we would recommend to load a small quantity of data in the trend. Maybe only 15 min and 1 min.
Using a faster CPU may also be an option. In any case, please ensure you are using the latest version of ESME (V2.1) and have upgraded the firmware accordingly.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks for the update. Your original post said that the fault happens "when viewing trends". Has it ever happened when the visu is not open, or is it always while viewing a chart?
I created my own library for trending, but it still uses the XY Chart. I have no idea if the built in trends use the same elements as the generic XY Chart, but it seems reasonable. My trends reference an ARRAY[1..1440] OF REAL, which seems to work just fine under normal conditions. But I wonder if the crashes I've seen are related.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Not that I am aware of, I mean, I have had my fair share of crashes haha! I think I have seen this one when I was testing some other type of code before but this is the only consistent fault I have, it definitely only seems to happen when looking through trends.
I looked into a similar idea but I don't know if it is the type of processor I have, the size of the existing project already or perhaps even the way I was trying do it, but I just didn't have anywhere near enough room to store that much retained data. I could have created a non-retainable array, but that seems to defeat the purpose for us mostly, we need the history even thorugh a power flick.
Thanks
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
On a Schneider M262, when monitoring CPU load with SchedGetProcessorLoad, it seems to run at around 25-45% during most tasks, occasionally jumping up to about 60% with some visualizations when loading.
I have some very small trends that I am only storing about 7 days data on, but multiple times now when viewing trends my PLC has gone into a Processor Load Watchdog fault.
Sometimes it works no drama, running back to the start of the 7 days, changing span, etc.
Then sometimes, like just now, all I tried to do was change the span to 1hr, and it crashed.
Is there something that can be done about this? Why would looking at a trend cause such a high load? What are my options for avoiding this???
I have a similar situation. I'm not using the built in trends, I created my own library for that, but I am using M262 PLCs with web visu.
I haven't been able to narrow down the problem, but I have seen the same thing with Processor Load watchdog at seemingly random times. My best guess at the moment is something to do with the interaction between file access and the visu, but that's still just a guess.
I've also seen it happen a few times where the PLC stays running , but the http server stops working and only the https works. I'll sometimes see in the logs '/usr/visu failed'. I'm not 100% sure that this is the same problem.
A few questions/things to check:
1. What browser are you using? Does this happen with HMIs or just computer browsers? I know there's issues with the older firmware on the Schneider HMISTW panels which would cause the M262 web server to stop working. We actually stopped using Schneider HMIs, we've been pretty happy with the Phoenix WP series.
2. Which M262 model?
3. What firmware on the M262?
4. Any other file access functions working?
5. What's the interval and priority on your tasks?
6. What's the update rate and default communication buffer size of your visualization? https://faq.codesys.com/pages/viewpage.action?pageId=112525371
Hi tvm, I appreciate your help, once again.
(Sorry for the late reply, Codesys Forge was down for a few days?)
It did seem to me like some kind of file accessing issue... That sucks though, as I guess for now I just need to remove the trends because the plant that it is running can't afford to stop as it may not be noticed and has the potential to destroy a lot of monies worth of product. I was talking to the client about getting another PLC so that we could split the process & the visu/file handling, that way at least if there is a crash the process can continue in the background. However this is really something you shouldn't have to spend money on when the product should be able to handle the process.
To answer your questions:
1. We are using the Schneider HMI's, however, I have had the crash happen when looking at the trend from my PC's browser, so I'm not sure that matters. Last time it crashed on Chrome, but I'm quite sure I have had it crash on Edge too.
2. TM262L10MESE8T
3. Firmware V5.1.7.11
4. I am creating and editing CSV files (creating reports) but when this is happening, it is only about once an hour, and it doesn't seem to line up at all with the crashes as far as I can tell.
5. I have the following Tasks:
- MAST (all code) 50ms, Priority 15
- IO_Scanning (no code, only used to scan bus), 300ms, Priority 30
- AlarmManagerTask, 200ms, Priority 20
- TrendRecordingTask, 1000ms, Priority 25
- VISU_TASK, 10ms, Priority 10
6. Update rate 40ms, Buffer size 50000 - Reading your attachment I see I might have all these Visualization values a bit low... They used to be higher but as you mentioned about the Schneider HMI's they are just so slow updating pages - I just kept lowering these values, it seemed like it helped but maybe I need to raise them back up...
If you have any suggestions or guides on what all the rates work well at I am all ears, It is probably not much more than guess work for me atm.
The other thing I did when trying to speed up the visuals was lower the Controllers monitoring interval to 100ms, which I believe also increases Load... Perhaps this is something I need to revert?
I guess the issue is as I said, regardless if these values are any good, the Load is really not that high until that one out of tenth time you go to check a trend...
I have also raised a case with Schneider (Although in the past I have found this forum much more helpful) - I'll see where that gets me.
Ben
OK, a couple ideas:
-MAST: 200mS (15)
-File task: 50-100mS (20) (I've experimented with different speeds and priorities)
-Network task: 20-100mS, depending on the network I'm using (various priorities, also depending on the network)
-Visu task: 200mS (lowest priority 31)
-We have one job where the CAT6 cabling between the PLC and the nearest switch (for the record, by another contractor) is almost 150m long, and very poor workmanship. We've noticed a lot of crashes with that particular installation. (similar to what you're describing) But it's hard to consistently reproduce.
-We had a problem for years on another job, where the PLC would crash at random times. We eventually traced it back to a network scan that the IT department was doing every couple weeks (scanning for viruses, compromised computers, etc). The scan was hammering the PLC so hard that it just couldn't handle all the requests. Schneider updated the M262 firmware a couple times to address this, and I think it's fixed, but I also set up the router to block requests from the scanning computer, mostly because I had spent way too much time working on it and I just wanted it to go away.
Wow, looks like you've had a lot of experience dealing with these kinds of things!
I believe the HMI's are the latest firmware, but I do not have one on hand to check. I'm quite certain I remember needing to upgrade them towards the beginning of the project for something though. I will confirm.
Hmm the M35, well I don't need motion so that seems a little overkill but it is interesting to know.... Perhaps it is something I can share with Schneider if needed, perhaps there are differences they will be able to work out themselves.
OK, thanks for those scan times, I might look at increasing those times again, and I might also look at increasing the MAST time too, it really doesn't need to be all that fast.
Hmm, interesting on the network connection. The last crash I was connected remotely using a VPN, however I have had it crash on my test bench with a 1 meter lead connected directly too, so I'm not so sure....
It sounds like my process is a lot simpler than many of yours, it is just out on orchids, so there are no IT things happening or special processes - It is just critical that it stay online as it is for the Refrigeration of long stored fruit.
The code & visuals are all very simple, I would say some of the simplest code I have written, except for the only "complicated" part being the CSV file reporting & some emailing.
This trend thing seems to be the last thing creating issues. Which is a real pain as I don't think of trends as being something that should be so difficult, I've worked with Trends on Citect for many years and never had any troubles, it definitely seems like an M262 problem....
Thanks
From Schneider:
We have feedback from R&D that this issue has already been reported globally and they are working on resolving it. It should be fixed with the integration of CoDeSys V3.5.SP19 in Machine Expert V2.2 planned for December 2023.
This is the reference number for the fix/change request. PEP1059101R: JMT-M262-AUS-Processor Load Watchdog when loading trend in WebVisu
In order to avoid the crash, we would recommend to load a small quantity of data in the trend. Maybe only 15 min and 1 min.
Using a faster CPU may also be an option. In any case, please ensure you are using the latest version of ESME (V2.1) and have upgraded the firmware accordingly.
Thanks for the update. Your original post said that the fault happens "when viewing trends". Has it ever happened when the visu is not open, or is it always while viewing a chart?
I created my own library for trending, but it still uses the XY Chart. I have no idea if the built in trends use the same elements as the generic XY Chart, but it seems reasonable. My trends reference an ARRAY[1..1440] OF REAL, which seems to work just fine under normal conditions. But I wonder if the crashes I've seen are related.
G'day tvm.
Not that I am aware of, I mean, I have had my fair share of crashes haha! I think I have seen this one when I was testing some other type of code before but this is the only consistent fault I have, it definitely only seems to happen when looking through trends.
I looked into a similar idea but I don't know if it is the type of processor I have, the size of the existing project already or perhaps even the way I was trying do it, but I just didn't have anywhere near enough room to store that much retained data. I could have created a non-retainable array, but that seems to defeat the purpose for us mostly, we need the history even thorugh a power flick.
Thanks