Intermittent comm problem CompactLogix, PV+

[rstech]

Hi everyone,

I have a problem with a new machine I programmed about a month ago. It is up and running but every day or two the PanelView decides to stop communicating with the PLC. When I've gone on-site I've found that I also cannot communicate very well with my laptop. What I mean by that is, I can see the processor in RSLinx but I can't go online (it times out), and when I ping from a command prompt my replies range from 200ms to 800ms to time outs. So there is some form of communication there, just not a good connection. If I cycle power to the PLC everything is back to normal. I can go online without any problem, the PanelView communicates and when I ping the response is <1ms every time.

Here are the hardware details. CompactLogix L32E with firmware 19.11, PanelViewPlus700 with firmware 5.10.06 and another integrator has a GuardLogix system that we are exchanging 5 words with (producer/consumer). Everything is going to a non-managed switch and the other integrator also has a PanelViewPlus and a Point I/O assembly.

While the equipment is running there doesn't appear to be a hold up in communication between my CompactLogix and the other integrator's GuardLogix. The machine runs as normal but on my PanelView we get the error bar at the top and the objects get the error messages and question marks in them. The problem seems to get progressively worse as time passes. When I leave the site everything is fine, the next day they start to notice the buttons acting really slow on the PanelView (15 seconds sometimes to actuate a button), and then several hours later it will periodically lose communication for several minutes at a time sometimes not recovering. Although I am getting some conflicting reports from the customer it sounds like the problem is usually cleared after cycling power.

I tried replacing the PanelView logic module and it made no difference. A week ago I replaced the CompactLogix processor and it made no difference. When I've been on-site and the PanelView has not communicated I have isolated everything else off the network and had just my laptop and PLC on the network and I still can't communicate with the PLC; not until I cycle the power. I even connected directly to the PLC to rule out the switch and cables. I contacted Rockwell Tech Support and they said to upgrade the firmware in the PanelView to the latest which I did this morning. So it went from 5.10.01 to 5.10.06. I'm skeptical about that solution which is why I'm posting here and hoping someone has some ideas.

A few other notes. The PLC and PanelView programs are not very big. There is plenty of memory available in both the PLC and PanelView. The temperature inside the panel does not seem to be a factor, doesn't feel unusually warm (plant is air conditioned). I've never set up produced/consumed tags before so I don't know if this is a factor. Seemed pretty straight forward and as I mentioned this part always seems to be working. I've worked extensively with Logix and PanelViewPlus systems and I've never seen an issue like this. Does anyone have an ideas or things I might try? Thanks.

 

[OkiePC]

I had some similar symptoms that ended up being caused by a duplicate I/P on the network. The only other recommendation I can make is to change the switch to a manged one and ensure that the network has IGMP querier and IGMP snooping turned on in the switch(es) that are related to that system.

Can you power cycle only the PLC to get comms back to normal? Or are you power cycling the PLC, the PV+ and the Ethernet Switch?

 

[rstech]

The other integrator assigned the IP addresses but I could certainly check for duplicates. I see what you're saying about the Ethernet switch but there isn't alot on this network; do you really think a managed switch is necessary?

I can power cycle only the PLC and everything returns to normal. I did that this morning while I was on-site. Thanks.

 

[glenncovington]

Could you use a program like Wireshark to see what the traffic looks like? Wondering if some sort of buffer is getting filled with errors.

 

[rstech]

Wondering if some sort of buffer is getting filled with errors.

I was wondering the same thing but I don't know of a buffer in the CompactLogix that would cause this. Anyone know of anything like this?

 

[Ken Roach]

First I would go to the web pages of the CompactLogix and check the statistics and diagnostics for the Ethernet port. You can get much of the same information from inside RSLogix 5000.

In those, you're looking for any sort of incrementing error counter.

Running Wireshark (or even better, Frontline NetDecoder) would answer the question of "is the PLC not responding, or is the PanelView not requesting" ?

Your description of long PING times and difficulty connecting to the CompactLogix tend to suggest that the problem is either on the CompactLogix side or is a physical problem with the network.

I always recommend managed switches just for the diagnostics you can get from them. The ability to constrain multicast traffic is often just a bonus.

Do you have auto-negotiation enabled on the Ethernet port of the CompactLogix ? When you're using an unmanaged switch, you generally want to allow Auto-Negotiation to work. Verify in the Ethernet diagnostics that it did so and that it chose 100 Mb/s Full Duplex.

 

[cstca]

I had a similar issue with one of our system, communication lost randomly between the PLC and SCADA. After replacing all the hardware, in the end we found out it was one bad patch cable. I've replaced it and never heard about the system again. Did you check your cables?

 

[The Plc Kid]

If you have firmware 19 in your processor you should have unicast messaging but the other stuff that was already there from the other integrator may be a different firmware leveland have multicast messaging and will cause problems with a non maaged switch or Hub.

If it were me and it were possible I would want both controlers to be at the same major firmware rev.

A managed switch would be best.

Have you checked connections? A poorly terminated RJ45 connector can Wreak Havoc. Were the patch cables / Interconnect cables made and terminated there? You may consider trying some pre made cables just to lay on the floor or something for a quick test just to ture that out.

If you have access to someone witha cable quality tester it would help. Maybe the customers IT dept has something?

Could it be your laptop NIC port? Maybe try another laptop?

Do you have a wireless adapter built in to your laptop? If so you may want to disable it. Depending on how it is setup and the system you are connecting to is set up it could cause problems.

You may also want to disable IPV6 if it is enabled. Just disable the protocol. I have seen the IPX protocol casue issues also in some cases if you use IPX.

 

[rstech]

I had a similar issue with one of our system, communication lost randomly between the PLC and SCADA. After replacing all the hardware, in the end we found out it was one bad patch cable. I've replaced it and never heard about the system again. Did you check your cables?

Well you know what; I was about to quickly type in that all the patch cables have been replaced, it was the first thing we did. But...when I take a moment to think back, it was my customer (the machine builder) that said he changed them all. I didn't do it myself. I wonder.... Certainly something that can quickly be done again; lots of those hanging around.

 

[The Plc Kid]

You may want to document all the IP addresses and subnet masks for all equipment and compare them. Use angry IP scanner or similar tool to port scan the network.

Check to make sure none of the other equipment is using CIDR IP schemes. If you have CIDR without a managed switch or router it can cause issues with some softwarez.

 

[The Plc Kid]

Well you know what; I was about to quickly type in that all the patch cables have been replaced, it was the first thing we did. But...when I take a moment to think back, it was my customer (the machine builder) that said he changed them all. I didn't do it myself. I wonder.... Certainly something that can quickly be done again; lots of those hanging around.

Yea but did he buy premade or make them? If he has a p*&^ Poor crimp tool or it is just worn he may be making bad cables and not even know it. I have seen this happen on a couple of projects.

 

[rstech]

If you have firmware 19 in your processor you should have unicast messaging but the other stuff that was already there from the other integrator may be a different firmware leveland have multicast messaging and will cause problems with a non maaged switch or Hub.

If it were me and it were possible I would want both controlers to be at the same major firmware rev.

A managed switch would be best.

Have you checked connections? A poorly terminated RJ45 connector can Wreak Havoc. Were the patch cables / Interconnect cables made and terminated there? You may consider trying some pre made cables just to lay on the floor or something for a quick test just to ture that out.

If you have access to someone witha cable quality tester it would help. Maybe the customers IT dept has something?

Could it be your laptop NIC port? Maybe try another laptop?

Do you have a wireless adapter built in to your laptop? If so you may want to disable it. Depending on how it is setup and the system you are connecting to is set up it could cause problems.

You may also want to disable IPV6 if it is enabled. Just disable the protocol. I have seen the IPX protocol casue issues also in some cases if you use IPX.

I should clarify, the equipment from the other integrator is also new. They were hired for part of the job and I was hired for another part. We are both running the same firmware, version 19. As far as the patch cables go, there is one field terminated cable, that is the cable between the other integrator's panel and mine. We each have a non-managed switch so this cable runs between the two switches. The customer is pretty small and they don't have an IT department. I've been thinking of getting a tester myself because I've come across poorly terminated cables many times so maybe this is the excuse I need to go buy one.

As for my laptop it is possible there is a problem with the NIC card but I don't think so. I'm connecting to PLCs, PanelViews, networks every day and I've never had an issue. I'm connected to a Logix processor at the moment as I type this. I do have a wireless card in my laptop and I always have it disabled when using a wired connection.

Thanks for the suggestions guys, I really appreciate it.

 

[rstech]

Yea but did he buy premade or make them? If he has a p*&^ Poor crimp tool or it is just worn he may be making bad cables and not even know it. I have seen this happen on a couple of projects.

They were purchased, not terminated by him. Only the other integrator terminated one cable between our two panels. Maybe that's the issue. I expect to be back on-site in the next couple of days because they are starting production again tomorrow so I'll see about getting myself a tester and checking the cable.

 

[The Plc Kid]

Wireshark would be most helpful

Angry IP scaner port scan will give a report that can be exported to tst with all the IP addys,Subnet masks and Hostnames, for all equipment so you can check and compare.

Make sure you don't have any duplicate hostnames also.

Make sure no device has a default gateway set to a device that can not be a gateway such as a panelview,etc.

A down and dirty and cheap way to check the cable between the 2 panels is setup your laptop on one end and a ip device on the other or another laptop and send 50 or so packets accross and look at the results

Here is the command / ping -n 50 192.168.1.1

The -n sets a fixed number and the 50 can be changed to the ammount of packets you want and the ip addy you want.

Do a Trace Route to each device also and look at the response times

Here is the command / tracert 192.168.1.1

Change the IP addy each time and try each device.

 

[lprinsloo]

Hi rstech did you sort out the problem you had with loosing communications?