ControlLogix Issue

Hi,

I have a weird issue with a ControlLogix processor and was wondering if anyone else has had the following issue. There is a PLC-5 which communicates to the ControlLogix via DH+ using a MSG instruction. This has been working without issue for over 7 years. However, recently it was noticed that the area which was being written to by the PLC-5 was being overwritten with other data! The code in both PLCs had not changed and all other PLCs on the network were checked and confirmed that no other PLC was writing to the ControlLogix, the only other machine that was connected was an upper system (SCADA) but also was not writing to the area in question either.

Eventually ControlLogix was powered down and on powering back up the issue had gone away! Going to program mode and removing the DH+ cable had failed to clear the issue. This is the second time it has happened to us with a ControlLogix processor (not the same one!). I don't have the type of processor or firmware version to hand but can get them tomorrow if required.

Steve

you're certain in the logic there is nothing timer or condition based that will write to those locations?

That they are running on their latest versions of the programs? Power failure could restore an older version from memory that may not be correct and have the data placed in the wrong point. To write into the CLX you have to configure it to mimic the memory structure of the PLC5 so the write command will work successfully. What else would be writing as a PLC5?

Consider reconfiguring it to be a Read command based in the CLX if the data is not instant mission critical. You can have the CLX read every second so you get the data fast enough for any slow process. You won't need to have it mimic a PLC5 memory structure then, the CLX knows how to read a PLC5 and can dump the data directly into tags.

Nova5 said:

you're certain in the logic there is nothing timer or condition based that will write to those locations?

That they are running on their latest versions of the programs? Power failure could restore an older version from memory that may not be correct and have the data placed in the wrong point. To write into the CLX you have to configure it to mimic the memory structure of the PLC5 so the write command will work successfully. What else would be writing as a PLC5?

Consider reconfiguring it to be a Read command based in the CLX if the data is not instant mission critical. You can have the CLX read every second so you get the data fast enough for any slow process. You won't need to have it mimic a PLC5 memory structure then, the CLX knows how to read a PLC5 and can dump the data directly into tags.

Click to expand...

Hi,

Three engineers have looked into the code and there is definitely no code (conditioned or otherwise) that can write into the area of code.

The last modification was over a year ago and the machine ran fine up until recently. We also have the current version number hard coded into the code, so we know we are using the correct version.

The only other machine that is writing to the CLX is the SCADA system but this writes to a different set of tags and has been checked.

We could reconfigure to get the CLX to read but this goes against the standard code we use.

However, I cannot get my head around the fact that :-

- We know we are running the correct code and it has been checked.
- Every other PLC on the network has been checked and comms from them disabled
- Only powering off the CLX has worked

Steve

I have once experienced similar symptoms, with a Controllogix processor being written to by a server over Ethernet. Processor sets a value to indicate it has data to be read, server reads data and writes back a different value to indicate next set of data can be loaded. The write back was not the expected value, nor any value the PLC has logic to set. Nothing else writes to that tag.

Cycled power to the processor and have not seen since, so I never dug in further.

Processor was 1756-L71S running v30.

it sounds like this is a transient problem i.e. you cannot induce it to happen.


what was the exact symptom that you saw i.e. how did you know the data were being overwritten? what were the actual bits, expected vs. observerd?


If you could cause it, or detect it happening again, perhaps something similar to wireshark could at least identify whether the source is external or internal.

plvlce said:

I have once experienced similar symptoms, with a Controllogix processor being written to by a server over Ethernet. Processor sets a value to indicate it has data to be read, server reads data and writes back a different value to indicate next set of data can be loaded. The write back was not the expected value, nor any value the PLC has logic to set. Nothing else writes to that tag.

Cycled power to the processor and have not seen since, so I never dug in further.

Processor was 1756-L71S running v30.

Click to expand...

Yes, that is similar. At least it's not just me!

drbitboy said:

it sounds like this is a transient problem i.e. you cannot induce it to happen.


what was the exact symptom that you saw i.e. how did you know the data were being overwritten? what were the actual bits, expected vs. observerd?


If you could cause it, or detect it happening again, perhaps something similar to wireshark could at least identify whether the source is external or internal.

Click to expand...

The PLC-5 sends 2 integer words (N29:0 & N29:1) to the CLX. These are mapped in the CLX to 2 integer tags. The issue caused the machine to stop.

On checking the reason why the machine stopped the engineer on shift noticed the watchdog (N29:0 - this increments every second up to 32767 and then resets to 0) was a different value from that seen in the other PLC. He then disabled the MSG instruction in the PLC-5 expecting the watchdog to stop incrementing but it didn't. After disabling all the PLCs on the network including the PLC-5 in question the issue still occurred. The only machine left was SCADA so it was assumed it was that and the SCADA engineers started looking into it although they were 99.9% sure it was not them! Whilst they were looking into the issue the engineer in frustration turned off the CLX and on restarting the issue had gone!

Steve Cav said:

T]
On checking the reason why the machine stopped the engineer on shift noticed the watchdog (N29:0 - this increments every second up to 32767 and then resets to 0) was a different value from that seen in the other PLC.

Click to expand...

how different?

drbitboy said:

how different?

Click to expand...

The PLC-5 watchdog was incrementing every second but the CLX was showing the same incrementing values but also being overwritten with different values.

eg.

PLC-5 showing 123, 124, 125, 126, 127
CLX showing 123, 5678, 124, 7865. 125, 8787, 126, 4536, 127

Hope that makes sense!

Steve Cav said:

The PLC-5 watchdog was incrementing every second but the CLX was showing the same incrementing values but also being overwritten with different values.

eg.

PLC-5 showing 123, 124, 125, 126, 127
CLX showing 123, 5678, 124, 7865. 125, 8787, 126, 4536, 127

Hope that makes sense!

Click to expand...

you are making up those numbers (5678, 7865, 8787, 4536, etc.) here, and you don't remember the actual non-sequential values, is that correct?

drbitboy said:

you are making up those numbers (5678, 7865, 8787, 4536, etc.) here, and you don't remember the actual non-sequential values, is that correct?

Click to expand...

That is correct, I wasn't the engineer on site at the time.

Unless the memory is going bad, or the environment is in some high-energy cosmic ray/radiation field, something is changing at least one bit in that word.


From the OP where you said the PLC 5s were disconnected, the only possibilities seem to be something internal to the PLC and the SCADA. No matter how certain everyone is that they should be eliminated from suspicion, those are the only feasible sources of flipping bits. The power-cycle fix points to the PLC itself; is it possible an indirect address has been corrupted?



The only thing worse than a program doing something different from what I want it to do is when it does exactly what I told it to do.

drbitboy said:

Unless the memory is going bad, or the environment is in some high-energy cosmic ray/radiation field, something is changing at least one bit in that word.


From the OP where you said the PLC 5s were disconnected, the only possibilities seem to be something internal to the PLC and the SCADA. No matter how certain everyone is that they should be eliminated from suspicion, those are the only feasible sources of flipping bits. The power-cycle fix points to the PLC itself; is it possible an indirect address has been corrupted?



The only thing worse than a program doing something different from what I want it to do is when it does exactly what I told it to do.

Click to expand...

I forgot to mention that both of the 2 words being transferred from the PLC-5 to CLX were being overwritten. I agree it looks like SCADA but it seems to have been ruled out and there is no issues in the PLC logic including indirect addressing - I've checked it myself. I can arrange for the PLC to be swapped out for a replacement, which I will get done but this is the second CLX (unrelated area of Plant) to exhibit this, yet I've not been able to find any occurrences of this elsewhere apart from a post above which is similar.

Obviously, if it starts happening again, disconnecting all outside systems (PLC5 and SCADA, I think you said) and observing whether that stops the bits changing will point to either the CLX or SCADA system as the source. Unless that proves that it comes from the SCADA, the SCADA maintainers are unlikely to be of any use.

drbitboy said:

Obviously, if it starts happening again, disconnecting all outside systems (PLC5 and SCADA, I think you said) and observing whether that stops the bits changing will point to either the CLX or SCADA system as the source. Unless that proves that it comes from the SCADA, the SCADA maintainers are unlikely to be of any use.

Click to expand...

Done that already and the issue went away. However, then reconnecting the issue returned, which pointed to either CLX issue or SCADA but cycling the power (which cured it) seems to point to CLX as cycling the power to CLX makes it unlikely to be SCADA.