Contrologix "I/O Not Responding"

I have 5 independent but interactive machines in a room, each of which has a AB 1756-L72S plc. The main machine has this and a 1756-EN2T, IP address 192.168.101.1 and a WAP (Stratix 5100, 17830WAPxK9 192.168.101.20

On the four slave machines/plcs, I have discovered/added the 1756-EN2T and then under it is the backplane and then the main machine PLC, it's safety partner and then the 1756-EN2T module again (not sure that's needed or right). Then back at the same hierarchy as the original showing of the 1756-EN2T module is the 1783- WAPxK9.

Not convinced this is the right way to construct the I/O tree for the four slave machines/plcs.

Anyway, the I/O OK changes to I/O Not Responding so I'm needing help in understanding why and correcting this. Which is probably why my produced and consumed bit being sent from the main to the slaves is not responding in any kind of timely manner. Also, the bit basically stops the slave machines and when this does work, the slave machines will stagger stop, which may mean I don't have the RPI setting correct on the slaves. And of course, don't really understand how the RPI works or should be set regardless.

Any insight is appreciated. And no, I didn't set this up so I won't be insulted. But I think this is the best way (e.g. wireless) to approach a remote stop without adding cables/wires/festoons, etc.

Was this posted previously? I seem to recall Ken Roach telling you to slap the architect. Perhaps that was another post.

My initial thought is that you are overloading your WAP. You are probably requesting data faster than it can be delivered. When the controller doesn't get the data from the remote device in the time it expects, you get your I/O Not Responding messages. In most cases, wireless is half-duplex. You can send or you can receive, but you can't do both simultaneously like you can with a wired network.

Open your controllers one at a time and take a look at your Controller Tags. Change the "Show" option to show consumed tags. Edit the properties of those tags and view the Requested Packet Interval (RPI) setting. My guess is that this number is set too quickly for your network. Unfortunately, changing that setting can only be done offline and then downloaded to the controller which requires downtime.

Can you identify what the RPI is set for? Each consumed tag can have different settings.

** EDIT *** Just found the previous post added on to an eight year old thread. Good on ya for starting a new thread!

OG

Yes, this post was a reply to another, kinda related post and Ken asked me to start a new thread.

I looked at the pdf for the Stratix 5100. I believe it does have a setting for full duplex, and it might be the default, but I don't know how to get exec privilege to see, let alone, change this setting.

The only DINT I'm producing/consuming has a RPI of 20ms. The WAP itself has a RPI of 1000ms.

Oh, and should the unicast box be checked on all of the WAPs?

Less slapping, more spectrum.

The duplex setting on the Stratix 5100 is probably for its Ethernet side. Full-duplex Ethernet is very common.

Wireless is generally not full-duplex, because you'd need two separate frequencies. Yes, modern wireless equipment has all sorts of magical mutliplexing technologies but it's still fundamentally a shared spectrum.

20 ms is pretty fast for a wireless Produced/Consumed Tag, but it's not out of the question. Ordinary safety connections are generally 10 or 20 ms.

I don't have much GuardLogix experience, and none using GuardLogix on wireless. Hopefully there will be some more experienced folks along shortly.

I recommend investigating to find out if the wireless system is being impacted by the hardwired Safety I/O traffic on EtherNet/IP. Safety connections default to Unicast, but we don't know for sure if that's set up correctly.

Are any of the Safety I/O modules configured to use the wireless network ?

Have you configured the Produced tag to support multiple Consumers ?

Is the Produced tag a Safety tag or an ordinary tag ?

Producing a multi-Consumer Safety tag is probably fairly bandwidth intensive.

Edit: As of v19, Safety tags can be produced as Unicast.
Edit: As of v20, Safety I/O modules can be configured as Unicast.

But I'm still not sure about a Safety tag with multiple Consumers.

An excellent pointer to the user manual sections on the use of Produced/Consumed Tags in GuardLogix systems:

https://rockwellautomation.custhelp.com/app/answers/detail/a_id/40203

(Access Level: Everyone)

The stratix 5100 manual says that it supports multiple frequencies and when I checked it via the web gui, I see that Gigabit Ethernet and wireless channel 2.4GHz is used, but 5GHz is off (not sure why).

The tag being produced/consumed is an ordinary tag. As far as I know, there are no safety tags being shared. As a matter of fact, when I just checked, there is no safety program in the safety plcs. And of course, no safety modules either. Not sure why the OEM has this installed if it's not being used.

Just because it is a Safety Controller...

I was actually just replying in the other thread and about to hit the Submit button when I thought, hmm, maybe he's already gone and started that new thread Ken advised him to? And sure enough, here you all are!

So I'll just post it up as it was intended over there, including the response to Firejo's advice as I feel it's important to consider. Some of it might not make sense unless you've read the other thread...

...Whether I'd recommend it or not I'll reserve judgement on but I would like to make this point with regard to slapping anyone down...

CIP Safety is fully supported over Wireless using a Stratix 5100 Wireless Access Point. While its implementation may pose some difficulties, it is doable. There are lots of things to consider, but with careful planning and a bit of "luck" with the installation environment, it can work well.

But are we talking CIP Safety here or just CIP?...(EDIT: Ken has sinced asked the same...)

oxlacey said:

Have 5 1756-L72S GuardLogix 5570 Safety Controllers on independent machines in one room. Using Stratix wireless access points. Wanting to produce/consume bits back and forth...

Click to expand...

Just because it is a Logix Safety controller does not automatically mean we are looking to Produce/Consume CIP Safety I/O. But again, whether Safety or Non-Safety, it is supported over a Wireless Network when the Stratix 5100 WAP is used. There are one or two small considerations to be made depending on which is intended to be used here...

oxlacey,

A Safety Controller can have Safety Tasks and Non-Safety Tasks.

A Safety Task will have Safety Tags.

A Non-Safety Task will have Standard Tags.

Are you attempting to Produce/Consume Safety Tags or Standard Tags?

Firejo said:

... it might be possible to reduce the number of dropouts...by increasing the RPI’s for the consumed tag. I’d change it (the RPI) to as long as possible and as long as the application will tolerate...

Click to expand...

If using Produce/Consume with Safety Tags then the RPI of the Consumer must match the Safety Task Period of the Producing Safety controller's project. So, yes, you could adjust the Consumer's RPI, but you must adjust the Producer's Safety Task Period the same, and of course consider the impact it would have on the Safety Task (<<< this is the important consideration to be made if using Safety I/O).

If a Wireless Network's throughput is dropping packets then along side the Consumer's RPI there are other adjustments that can be made with or instead of the Consumer's RPI to try to alleviate the dropped packets issue. These include the Timeout Multiplier and the Network Delay Multiplier.

Under the Consumed Tag Connection properties go to the Safety Tab and there you should monitor the Connection Reaction Time Limit (CRTL) and Max Network Delay.

This is also where you can adjust the Consumer RPI, or, click on Advanced...

Here you'll see the Timeout Multiplier - This determines the number of RPIs to wait for a packet before deciding a Connection has timed out. You should set the maximum of 4x for Wireless.

Also you have the Network Delay Multiplier - This defines the message transport time that is enforced by the CIP Safety Protocol. The Multiplier specifies the round-trip delay from the Producer to the Consumer and back
to the Producer. You can use the Network Delay Multiplier to increase or decrease the Connection Reaction Time Limit. For Wireless, it is recommended to adjust the Network Delay Multiplier until the CRTL is at least 4x the RPI.

You should also make sure you are limiting the total Packets Per Second (PPS) over the Connection to well under the recommended 2,200pps (With one DINT I think you're OK).

As for the 20ms RPI?...

That is the default for the Tag Connection and is very unrealistic across a Wireless Network.

These are published expected norms for a Wireless Produce/Consume Connection for CIP Safety using the above settings and limiting the data...

Worst Case No Fault ≥ 200 ms
Worst Case Single Fault ≥ 360 ms

If you only have one DINT tag passing then you stand a good chance of getting the best out of this but do look a those timing and multiplier settings before messing around with the WAP.

Some of that is in the manual Ken linked and some of it is on the Knowledgebase and some of it is in my head...

EDIT...having only read your last reply, apologies for that, this installation is starting to look very strange?

Regards,
George

If I understand the last several posts.

You have a safety controller connected to your system via Wireless network.
If this is true, this is a definite no no. I cannot cite book chapter and verse, but my last job didn't allow this due to safety concerns.

Please contact your local tech specialist and discuss this with him.
You should then do a safety assessment in regards to the machine, specifically
what happens when the wireless network fails and the safety controller detects a fault or e-stop, stop condition and it cannot relay that information to the other plc's and thus causes machine damage, an injury, or worse?

If I am wrong, someone please correct me !!

regards,
james

James,

You may have overlooked this in their last post...

...As far as I know, there are no safety tags being shared. As a matter of fact, when I just checked, there is no safety program in the safety plcs. And of course, no safety modules either. Not sure why the OEM has this installed if it's not being used.

Click to expand...

This is why this application is starting to look so strange to me?

It "appears" as though they specified Safety controllers while only doing Standard Program Tasks. It's possible they were just future proofing it so as to allow CIP Safety to be added at a later date.

Risk Assessments of course play a big part in designing these Safety applications and for many cases the use of a Wireless Network (WLAN) may not be permitted. But for cases where it is. All manner of Wireless communications can take place for CIP Safety. I haven't time to go all into it but an application could have several Logix Safety controllers all accessing their Safety I/O over the WLAN . This WLAN could be a fixed island of information. A typical architecture would be an Autonomous WLAN where each controller has a local WAP set for Workgroup Bridge (WGB) mode and all communicating back to one master WAP set as an Access Point (AP). The controllers could Wirelessly communicate with each other directly but best practices dictate that you do not due to contention issues. You route traffic through the AP and use a Master controller to aggregate and propagate the information.

The key here is to answer your pertinent question...

Q. What happens when the WLAN fails on a CIP Safety application?

A. Connections time out and Safety Functions are executed.

Fail-safe.

Rockwell and Cisco have collaborated hugely, as we know, to bring these architectures to the marketplace.

Regards,
George

George, I very much appreciate your more-experienced posts on the matter.

When I stopped working on this sort of thing directly, CIP Safety was mostly over DeviceNet with the Omron-built I/O blocks. CIP Safety over Ethernet was in its infancy.

And Cisco's collaboration with RA was pretty weak at the time. Nobody had done the work to test and validate CIP Safety over wireless.

I think that Oxlacey should get the non-Safety connectivity working reliably, then move to Safety connections. If the simple stuff doesn't work, the complex stuff is unlikely to work either.

Geospark said:

The key here is to answer your pertinent question...

Q. What happens when the WLAN fails on a CIP Safety application?

A. Connections time out and Safety Functions are executed.

Fail-safe.

Click to expand...

Spot on George, and just one other interesting tidbit to add to that - when performing a safety validation, you actually have to consider the length of the timeout in your assessment. For example, if you have an RPI of 20ms, the maximum length of time before a remote safety output rack determines the lost connection and executes the safety function is 39.9999ms. Then you have the internal switching time of your output (say 5ms) and the time for your safety contactors to physically de-energise (say 100ms). Total 145ms, near enough. So you have to work out how far inside the light curtain Bubba can get at a full sprint in 145ms, and whether he'll be able to reach something hazardous in that time. If he can, it's back to the drawing board. Of course, there's also machine run-down time to consider, etc etc. "Safety" and "Simple" never did go hand in hand

Geospark said:

It "appears" as though they specified Safety controllers while only doing Standard Program Tasks. It's possible they were just future proofing it so as to allow CIP Safety to be added at a later date.

Click to expand...

I sure hope they're adding it later. GuardLogix is an expensive way to run a machine if you're not doing safety!

Let me chime in on the wireless side.

I’m currently testing some 802.11b/g/n radios (no 5.8GHz) and in the current setup I’ve got I’m able to get about 30Mbps of actual throughput (that’s using a TCP connection). While a lot of radios brag about 300Mbps the reality is that it’s “over the air” speeds and the real throughput is usually much slower. 30Mbps of consistent throughput using a TCP connection is screaming fast in an industrial automation real world environment (which is how I’m testing). One of the tests I’m currently running is using three L35E’s with two producing three tags each (one “connection status” tag and two “DINT” tags) and the other one consuming them. I’m using the status tags to monitor if I drop the connection or not to each PAC. With the RPI setting at 250mS I was still getting occasional dropouts that would last about 4 seconds which tells me that it is the UDP connection being reset. At the moment I’ve got the RPI’s at 500mS and the dropouts seemed to have stopped but I’m going to let it run overnight to see how it runs in the long term.

Having said all of that, I’m not saying you can’t make P/C work wirelessly but if you’re going to do it you better be prepared to have long RPI settings and to run tests in the environment the wireless will be working in. Even then you might run into problems down the road as technology advances and the 2.4GHz spectrum becomes even more crowded.

Now, let me explain why no 5.8GHz (because I know there is someone out there saying that I’m wrong not to use it). 5.8GHz technology has come a long way in the last 15 years (or so) however the basic laws of physics can’t be denied and the reality is you have to have pristine conditions for it to be more effective than 2.4GHz even with 2.4GHGz being as crowded as it is. In all of the applications that I’ve seen or talked with customers about when you dig down and look at what the MIMO radio that is using 5.8GHz as one (or more) of its transmitters is really doing the 5.8GHz side is doing very little unless the distance is very short and line of sight is as good as it can be (and a whole host of other things that need to be perfect). What gets people into trouble is that in the home environment 5.8GHz actually can work pretty well but in a typical industrial environment there is just too much noise and obstacles in the way to make it effective.

At the end of the day I have to agree with Ken’s comments in the related post (which I usually do). Anybody who creates an application with P/C being connected wirelessly without any testing of the application in the environment it will be running in needs a bit of an education in wireless technology (and a good slap upside the back of the head).

Just to clarify. There are five (5) OEM machines in a room. All can run independent, but there is a common goal for the 5 machines in taking spools of wire (main machine), loading them onto loaders (2) which themselves are sitting onto a car (2), and then the loaders/cars transporting/loading the spools onto a final station.

Only the main machine has a safety plc. And there is no safety program/tag/modules/etc. This plc is only being used with standard program/tags. I agree with an earlier post, evidently it was either specified for a future safety-related addition (not likely), or it is simply a part of the OEM's standard BOM for this type of machine (most likely).

The produced, non-safety, tag is to just inform the loader/car that the main machine is not running or has spools, etc. A second of delay is not awful. But right now I get random communication delay on the wireless for this produced tag anywhere from "instant" to "never".

I saw from an earlier post (I can't see the earlier posts while I'm writing this!) that to check some settings on the produced/consumed tag properties which I'll do. I think I also saw that someone has adjusted their RPI from the default of 20 to around 300? All things that I'll try and experiment with.

Hope that helps you folks zero in on what could be my original and still existing issue of I/O not responding.

Over a wireless connection, I would suggest using explicit messaging (MSG instructions) rather than Produced/Consumed. It gives you much more control over how you attempt to recover from a wireless dropout. Generally I'll recommend P/C over MSG instructions until I'm blue in the face, but at a certain point you have to use the right method for the right application.