SLC505 Ethernet Port Lockup..

Is this possible?
Recently on a machine connected to an InTouch SCADA, we have been losing comms from the DASABTcp driver, & to a certain extent, does not recover very well.
I have tried disabling the IO Server to allow the port to timeout/disconnect etc, but when re-enabling the server does not recover, the same errors are still present.
The only fix that seems to work is cycling the PLC. For obvious reasons, this is not ideal
When I look at the Enet LED on the CPU, it is blinking intermittently, but not at the same rate when things have been functioning correctly.

So, a couple of questions I guess:
1. What could be causing this
2. Is there anything that I should be looking at in the PLC (Peripheral Servicing?)
3. is there any status words that if I connect via serial I can check if & when this happens again?
4. From 3, can I initiate an Ethernet port restart programatically?

It is possible that it is the CPU/Enet port is failing due to age, but surely this can be monitored?

lostcontrol said:

1. What could be causing this

Click to expand...

Too many to list.

lostcontrol said:

2. Is there anything that I should be looking at in the PLC (Peripheral Servicing?)

Click to expand...

Use a browser to navigate to the IP address to see the 5/05 ethernet built in web page. It can show you statistical counters that can be invaluable in narrowing down the possible answers to #1.

lostcontrol said:

3. is there any status words that if I connect via serial I can check if & when this happens again?

Click to expand...

You can use channel diagnostics provided that a diagnostic file has been assigned. You must do this offline. Once assigned, under the controller properties, channel configuration, a channel diagnostics object will become available. It is much like the web browser version, but you can access it from either channel.

lostcontrol said:

4. From 3, can I initiate an Ethernet port restart programatically?

Click to expand...

Maybe. I have had problems when moving things around on a network, this caused new connections to devices which now had different IP addresses. The problem was, it ran my CPU out of connections. I figured out later, that the old connections would have died on their own after 30 minutes which was the default conection timeout setting.

Not sure if you can change it at runtime with logic, but you can configure that timeout to a much shorter value during periods when you are making a lot of changes, then bump it back up when the network is stable so it doesn't have to rebuild connections for things that may connect intermittently.

lostcontrol said:

It is possible that it is the CPU/Enet port is failing due to age, but surely this can be monitored?

Click to expand...

Yes. Not sure why, if age related, ESD, etc., but here, we have about 30 5/05s and in the last 6 years we have had at least 3 5/05s die a slow death as far as the ethernet stability.

We will start seeing unexplained time outs and other errors, and it will get worse over a period of days and weeks.

OkiePC said:

Too many to list.

Click to expand...

I know, but thought someone might have crystal ball...

Use a browser to navigate to the IP address to see the 5/05 ethernet built in web page. It can show you statistical counters that can be invaluable in narrowing down the possible answers to #1.

Click to expand...

Ok, will look at this when there next early in the week. When pinging the PLC, the response was quite slow.

You can use channel diagnostics provided that a diagnostic file has been assigned. You must do this offline. Once assigned, under the controller properties, channel configuration, a channel diagnostics object will become available. It is much like the web browser version, but you can access it from either channel.

Click to expand...

Ok, will try & set this up as well

Maybe. I have had problems when moving things around on a network, this caused new connections to devices which now had different IP addresses. The problem was, it ran my CPU out of connections. I figured out later, that the old connections would have died on their own after 30 minutes which was the default conection timeout setting.

Click to expand...

Not sure if this my issue. There are 2x PC's with redundant IOServers, & they both lost connection to each other at the same time. Since then, the system did not recover.

Not sure if you can change it at runtime with logic, but you can configure that timeout to a much shorter value during periods when you are making a lot of changes, then bump it back up when the network is stable so it doesn't have to rebuild connections for things that may connect intermittently.

Click to expand...

If I could restart the Ethernet port, then that may go a long way to solving the issue.

Yes. Not sure why, if age related, ESD, etc., but here, we have about 30 5/05s and in the last 6 years we have had at least 3 5/05s die a slow death as far as the ethernet stability.

Click to expand...

This one is not too old from the 'in-service' perspective as far as I know. Another suspicion I have is power, I wonder if either the network equipment is dropping out momentarily, causing the connection to not complete, then coming back & trying to reconnect, causing the continued disconnect/reconnect cycle.
If I could monitor this, then that may also help...

If you are attempting to transfer large amounts of data between your SLC and DASABTCP I/O Server, then you might need to change the factory default settings for Channel 1 messaging. The factory default settings for these status bits will only allow one message to be serviced per sweep.


Comms Servicing Selection S:2/15

When set, only one communication request/command can be serviced per END, TND, REF, or SVC. When clear, all serviceable incoming or outgoing communication requests/commands can be serviced per END, TND, REF, or SVC. When clear, your communication throughput will increase. However, your scan time will increase if several communication requests/commands are received on the same scan. To prevent inadvertent changes to this setting, program an unconditional OTL instruction at address S:2/15 to ensure one request/command operation, or program an unconditional OTU instruction at address S:2/15 to ensure multiple request/command operations.

Msg Servicing Selection S:33/7

This bit is only valid when the Comms Servicing Selection bit (S:2/15) is clear (which selects service all commands). When S:33/7 is set and S:2/15 is clear, all outgoing channel 1 MSG instructions are serviced per END, TND, SVC, or REF instruction. Otherwise, only one outgoing channel 1 MSG command or reply is serviced per END, TND, SVC, or REF instruction.

BTW: It is not uncommon for a failing Ethernet switch to cause intermittent communication failures such as this.

I did have a tinker with those status bits, but none of them allowed the IOServers to recover very well. The only solution was to cycle power to the PLC.
I may look at replacing the switch, as well as checking the status bits mentioned.

Does anyone know if there is a way to trigger a Ethernet Port reset without cycling power?

lostcontrol said:

I did have a tinker with those status bits, but none of them allowed the IOServers to recover very well. The only solution was to cycle power to the PLC.
I may look at replacing the switch, as well as checking the status bits mentioned.

Does anyone know if there is a way to trigger a Ethernet Port reset without cycling power?

Click to expand...

The SLC has no hard reset capability for their Ethernet port, to the best of my knowledge. Some PLC Ethernet adapters have reset pushbuttons.

Have you looked at your log files for the DASABTCP I/O Server in the console? Are you getting queue full or poke failed messages when this occurs?

Bit_Bucket_07 said:

The SLC has no hard reset capability for their Ethernet port, to the best of my knowledge. Some PLC Ethernet adapters have reset pushbuttons.

Click to expand...

That's a shame. I'm sure if there is it won't be posted on a public forum.

Have you looked at your log files for the DASABTCP I/O Server in the console? Are you getting queue full or poke failed messages when this occurs?

Click to expand...

Yes, but think it was socket related. There was poke messages, but there was also a 1ms script trying to write which I have changed to 500ms since.
I can get the logs when onsite next to confirm the error msg

Bit_Bucket_07 said:

Have you looked at your log files for the DASABTCP I/O Server in the console? Are you getting queue full or poke failed messages when this occurs?

Click to expand...

The errors in the logs were more 'message revoked'.
This has been happening too often in the last week, so have enabled the Serial port for the WW comms.
I also noticed that the timeout in the PLC was set to 30min, made that smaller to see if there was any difference.
Now I have everything going via the Serial, it seems more stable. I have a topic setup for the Ethernet, only with 1x tag, but that is still flakey.

When the PLC stops responding to the DAS, it is very intermittent, if not impossible to ping, so getting the diagnostics from the web server are impossible.
I have setup the diagnostic table in the PLC, which I'm assuming is where the Web server gets its information from..?

Anyway, have sourced pricing on a replacement & given to customer. Hopefully that will solve the problem.

lostcontrol said:

I did have a tinker with those status bits, but none of them allowed the IOServers to recover very well.

Click to expand...

So, I'm curious. When you say that you "had a tinker", does that mean that you Cleared bit S:2/15 and Set bit S:33/7 in your PLC, and then stopped and re-started your I/O Server?

If you didn't do that, then I'd recommend that you at least give it a try.

Bit_Bucket_07 said:

So, I'm curious. When you say that you "had a tinker", does that mean that you Cleared bit S:2/15 and Set bit S:33/7 in your PLC, and then stopped and re-started your I/O Server?

If you didn't do that, then I'd recommend that you at least give it a try.

Click to expand...

Probably not in that actual order, or at all like that. But I can try that today when there.

The main disclaimer to all of this, is that the system has appeared to be running without me going near it for over a year. I know there was another engineer who did some changes late last year, but doubt that those changes caused this. These problems are only just happening last 2 weeks.

With a SLC, they need a spare around anyway. Not saying they aren't good PLCs but they are not exactly bullet proof. Just as a troubleshooting aid for cases like this, they need one. Swapping that spare in place should tell you PDQ which direction to go with further troubleshooting. If it doesn't fix it then you still got network issues...if it does fix it (likely) you are done.

I've found the SLC processors to be quite reliable. The chassis is however, rather flimsy, and I've seen a few power supplies fail over the years.

Bit_Bucket_07 said:

I've found the SLC processors to be quite reliable. The chassis is however, rather flimsy, and I've seen a few power supplies fail over the years.

Click to expand...

It depends on what you are comparing it with. I compare them all the the PLC-5 which simply isn't fair. I agree about the chassis being flimsy and we do see a lot of faults caused by chassis failures but I have not seen one damage a CPU.

By the way, I don't keep a spare for our sole PLC-5...it costs too much and we will never need it...(I know I just f**ked myself by saying that)...

I never had a problem with a PLC-5 chassis. I wish they had made the Controllogix Chassis more like the PLC-5 or Automax instead of the SLC.

Pinging is a low layer function so if you can't ping then this is most likely hardware related. Swapping out is the next step.