Micrologix 1100 issues

Before I start, I'll introduce myself. I'm a lab nerd who has some IT background and I've been roped into trying to help figure out what's going on with a PLC...which until about 3 months ago, I didn't know anything about...I still don't know much. I did manage to figure out how to get a copy of RSLogix Micro and RSLinx Classic going on my laptop and I've used it to download a program to the Micrologix 1100. I don't know anything about programming PLCs but I'm willing to learn whatever I can!

What I've got is a Micrologix 1100 that keeps ending up in fault and someone keeps clearing the fault before I can get to it to find out what's going on. Well today I was able to get the fault. The display read "Major Error Code=0022h" The PLC is being used with SCADA to control 2 pumps and receive some flow information from a radio and pass it along to SCADA. When it hangs up, the pumps stop and the flow data from the radio also hangs.

This keeps happening but not in a set frequency. I've attached the program for you to look over. I've got 2 of the Micrologix 1100s, one on my desk to play with and one online in the field.

Thanks in advance for your patience and time!

Tom

From the manual

0022 WATCHDOG TIMER
EXPIRED, SEE S:3
The program scan time exceeded the
watchdog timeout value (S:3H).
Non-Recoverable
• Determine if the program is caught in a loop
and correct the problem.
• Increase the watchdog timeout value in the
status file

Also, if your ML1100 is running FRN 6, you might want to update to FRN 9 or reduce the frequency of the Ethernet port communications packets (especially SCADA broadcast).

FRN == firmware revision number (since OP isn't a PLC person)

Error code 22 is a watchdog timeout. I see nothing in the programming that should cause a watchdog timeout to be likely.

About that error: In a PLC, the scan is by its very nature an infinite loop, and the CPU keeps an eye on the time to complete each pass. If the program finishes within the amount of time allowed by the watchdog time limit (2500ms in your file) no fault occurs. If you were to do something that caused the cycle to hang up, the watchdog timer will be triggered and the CPU will shut down.

You are using an STI (selectable timed interrupt) to calculate a flow total. The STI routine causes the CPU to drop everything and run file 7 exactly once per second. If you were doing that once every 3ms, then that could be the root cause of a watchdog. Once every 1000ms should be just fine. Make sure nothing is writing to that STI time setpoint...

There are MSG instructions which can add to the overhead of the CPU and slow things down a tiny bit...still nothing too odd about any of the logic. The serial port is set up for half duplex master (to allow multi-drop RS232 comms to your radio remotes). That protocol selection might add some small insignificant burden to the PLC overhead (execution between scans) above more "normal" settings but shouldn't be causing a problem.

The ML1100 series B is probably scanning that logic in under 5ms at almost all times...I see nothing glaringly wrong with the code. I'd be looking at external stuff or perhaps a piece of faulty hardware.

Make sure any loads connected to the I/O of the base unit are not causing a surge. Make sure the power supply to the CPU is solid. If you find nothing wrong there, try a new PLC or swap it with the other one and see if the problem follows the hardware.

Your ML 1100 is running Firmware Revision 5 (OS FRN 5); if Paul's recommendations will not solve the issue I'd suggest ControlFlash upgrading the firmware to at least FRN 10 (lowest available).

See 'Corrected Anomalies in Firmware Revision 6' in http://literature.rockwellautomation.com/idc/groups/literature/documents/rn/1763-rn003_-en-p.pdf

Download 1763 FRN 10 at http://compatibility.rockwellautoma...rumb=112&refSoft=1&toggleState=&versions=3943

Download ControlFLASH software at http://compatibility.rockwellautoma...umb=112&refSoft=1&toggleState=&versions=55875

ControlFLASH User Manual at:http://literature.rockwellautomation.com/idc/groups/literature/documents/um/1756-um105_-en-e.pdf

dmargineau said:

Your ML 1100 is running Firmware Revision 5 (OS FRN 5); if Paul's recommendations will not solve the issue I'd suggest ControlFlash upgrading the firmware to at least FRN 10 (lowest available).

Click to expand...

Let's start here. I'm guessing the program I uploaded indicated that the ML 1100 is running FRN 5. I just checked the unit online and it is running FRN 14. Could the mismatch between the program and the unit possibly be the cause?

OkiePC said:

Error code 22 is a watchdog timeout. I see nothing in the programming that should cause a watchdog timeout to be likely.

About that error: In a PLC, the scan is by its very nature an infinite loop, and the CPU keeps an eye on the time to complete each pass. If the program finishes within the amount of time allowed by the watchdog time limit (2500ms in your file) no fault occurs. If you were to do something that caused the cycle to hang up, the watchdog timer will be triggered and the CPU will shut down.

Click to expand...

That all makes sens, even to me as someone with almost no knowledge of PLCs.

OkiePC said:

You are using an STI (selectable timed interrupt) to calculate a flow total. The STI routine causes the CPU to drop everything and run file 7 exactly once per second. If you were doing that once every 3ms, then that could be the root cause of a watchdog. Once every 1000ms should be just fine. Make sure nothing is writing to that STI time setpoint...

Click to expand...

How would I determine if something is writing to that STI time setpoint?

OkiePC said:

There are MSG instructions which can add to the overhead of the CPU and slow things down a tiny bit...still nothing too odd about any of the logic. The serial port is set up for half duplex master (to allow multi-drop RS232 comms to your radio remotes). That protocol selection might add some small insignificant burden to the PLC overhead (execution between scans) above more "normal" settings but shouldn't be causing a problem.

Click to expand...

Could a failed, late, or damaged transmission from the other radio (sending the flow information) cause the error?

OkiePC said:

The ML1100 series B is probably scanning that logic in under 5ms at almost all times...I see nothing glaringly wrong with the code. I'd be looking at external stuff or perhaps a piece of faulty hardware.

Make sure any loads connected to the I/O of the base unit are not causing a surge. Make sure the power supply to the CPU is solid. If you find nothing wrong there, try a new PLC or swap it with the other one and see if the problem follows the hardware.

Click to expand...

The ML1100 we use is running on AC power, 120 V to be specific. What's got me is that this is the 2nd ML1100 to act up in under 2 years. I think the one that's online now was installed the end of 2015 maybe early 2016. One other thing to mention...the unit that is online now lost program about 2 months ago. I reloaded the program and it ran fine until last week when it lost program again. I reloaded again and 3 days later it gave this 22h error.

I do have a brand new ML1100 with the program already loaded on it that I could swap out. Though, if something external is causing the issue, I'd much rather figure that out first.

I've attached a few pics of the ML1100 in it's cabinet, just in case that is of any help.

Thanks!
Tom

H2OTom said:

Let's start here. I'm guessing the program I uploaded indicated that the ML 1100 is running FRN 5. I just checked the unit online and it is running FRN 14. Could the mismatch between the program and the unit possibly be the cause?

Click to expand...

No; the currently running application is what matters only; it would be helpful though to post the latest (apparently faulty) copy instead of a years old one; since you are running the 'Micro' edition of RSLogix 500, Upload and Save As, archive and post it here again; how else did you 'check the unit online'?

dmargineau said:

No; the currently running application is what matters only; it would be helpful though to post the latest (apparently faulty) copy instead of a years old one; since you are running the 'Micro' edition of RSLogix 500, Upload and Save As, archive and post it here again; how else did you 'check the unit online'?

Click to expand...

Here's the file. As far as what I've done to check it, .
I really don't know enough to even be dangerous here...well maybe a little.
I've been using the laptop connected via a crossover ethernet cable to access it using the RSLogix Micro. What can I do to check the unit online?

Thanks!

Does your plant have a lot of controls? Drives? Things like that? If there is a lot of noise on your power grid, that might be causing at least some of the problems you outline if you don't have some sort of filter conditioning the power going to the Micro

It definitely is a Channel 0 communications issue; Channel 0 (DF1 Half Duplex Master) Channel Status counters from your latest application state 5842 Retried Messages, 469 Undelivered Messages, 996 Duplicate Messages Received and 1008 Bad Packets Received.

There are six MSG instructions (MG:11.0, MG:11.1, MG:11.2, MG:11.3, MG:11.4 and MG:11.7) triggered at the same time for no longer than a second at ten seconds intervals.

Two other ones (MG:11.5 and MG:11.6) are triggered at all times when the previous 'batch' is not Enabled and when a bit without a coil (B3:0/7) is true.

There are hundreds (160+) of words of data Read/Written between the ML1100 and the radio modem(?) and any hiccup could stall the program scan to the point that the Watchdog time can be exceeded.

Unlatching the .EN MSG bit when .ER or .DN are true does not do affect the B3:0/6 or B3:0/7 command bits; B3:0/6 is ON for a second every 10 seconds no matter if the controlled MSGs are successful or not while B3:0/7 is on all the time (unless externally written).

This data exchange logic approach needs to be 'cascaded' as to only one MSG should be enabled at any time especially when the communications are serviced serially via a radio modem link; moreover, any MSG failure should address the 'command' bits' states too instead of merely 'unlatching' the MSG instructions themselves.

You might want to approach someone with more experience with 'old school' programming Serial communications data transfer between radio modems and low tier PLCs.

I didn't analyze the logic driving the messages. What dmargineau found is worth digging into. I would not think it could hang the overall scan time, but if you have comms service selection bits turned off, then perhaps the overhead trying to all the messages handled is killing the CPU.

In a micrologix (since you can't edit messages online at all) and in general, I like to trigger my messages with a state engine. One message at a time, set the channel (and/or message timeouts) to a reasonably short value and only advance to the next mutually exclusive state when the previous message is DoNe or ERrorred.

On ethernet comms, I will manipulate the TO bit with my own message timer to avoid setting the channel timeout to a very low value, but with serial comms, I don't manipulate the message bits...I just set the timeouts in each message instruction to something like 3 to 5 seconds. Even with a repeater radio at 4800 baud, most messages of up to 100 integers complete in under three seconds in the radios I have dealt with. Where there are repeaters, I usually set the timeout to 5 seconds. The idea being that if the message is going to fail, let the logic move on quickly and not bog down the whole cycle.

With a state engine in charge of the message logic, if I want to implement retries or skip over nodes that are out of service, I can manipulate the state logic with online edits and leave the message logic alone.

dmargineau said:

It definitely is a Channel 0 communications issue; Channel 0 (DF1 Half Duplex Master) Channel Status counters from your latest application state 5842 Retried Messages, 469 Undelivered Messages, 996 Duplicate Messages Received and 1008 Bad Packets Received.

There are six MSG instructions (MG:11.0, MG:11.1, MG:11.2, MG:11.3, MG:11.4 and MG:11.7) triggered at the same time for no longer than a second at ten seconds intervals.

Two other ones (MG:11.5 and MG:11.6) are triggered at all times when the previous 'batch' is not Enabled and when a bit without a coil (B3:0/7) is true.

There are hundreds (160+) of words of data Read/Written between the ML1100 and the radio modem(?) and any hiccup could stall the program scan to the point that the Watchdog time can be exceeded.

Unlatching the .EN MSG bit when .ER or .DN are true does not do affect the B3:0/6 or B3:0/7 command bits; B3:0/6 is ON for a second every 10 seconds no matter if the controlled MSGs are successful or not while B3:0/7 is on all the time (unless externally written).

This data exchange logic approach needs to be 'cascaded' as to only one MSG should be enabled at any time especially when the communications are serviced serially via a radio modem link; moreover, any MSG failure should address the 'command' bits' states too instead of merely 'unlatching' the MSG instructions themselves.

You might want to approach someone with more experience with 'old school' programming Serial communications data transfer between radio modems and low tier PLCs.

Click to expand...

The radio communications has been what I've suspected all along. I really don't know enough about this stuff. Thank you for taking the time to look at it for me! Can you tell me where I'd find those channel status counters?

OkiePC said:

I didn't analyze the logic driving the messages. What dmargineau found is worth digging into. I would not think it could hang the overall scan time, but if you have comms service selection bits turned off, then perhaps the overhead trying to all the messages handled is killing the CPU.

Click to expand...

Much of this is all so far over my head. I'd like to dig into it but I don't know where to start.

H2OTom said:

Can you tell me where I'd find those channel status counters?

Click to expand...

In your application's Project 'tree'/Controller/Channel Configuration/Channel Status/Channel 0.

Channel 1 (Ethernet) is clear of errors so SCADA reads are probably not the fault cause.

As for Channel 0, there are lots of errors; you could Clear the counters and start with a 'clean state', however, in your case, since running RSLogix 500 Micro you will not be able to do so unless downloading to the CPU a 'blank' comms counters file; that's where RSLogix 500 Standard Edition will be required as to be able to monitor and edit the application while Online with the controller.

As for the MSG data transfer since there are only two 'external' devices communicating on Channel 0 there should be only four MSG instructions; one Read and one Write for each node thus the addresses where the data is read into or written from will need to be contiguous.

As Paul suggested, a State Engine handling the 'one-at-a-time' triggering of the explicit messaging would be preferred.

There is need for significant editing to be performed in order to implement the proper communication data transfer logic; this will need to be performed Online as to be able to monitor the system functionality while modifying its programming code; again, someone experienced in PLC programming and system communications functionality will probably be required for the mitigation of this issue.

Please don't take it personally...This is an 'exact science' only up until the point you turn the Offline programmed system ON...There is quite some 'trial and error' required work from that moment on when expecting to achieve the intended functionality...Experience does not have a substitute...