Intermittent Communications issue to two PLCs at the same time

[Drewster1123]

I've been dealin with some intermittent CIP explicit messaging issues. I tried reducing the amount of small MSG instructions and converted them to have one block message that fires every 25 ms. A read block of 25 DINTs and a write block of 25 DINTs. The other plcs are for 2 other machines, A draw furnace and hot setting machine.

The Draw PLC is a SLC 5/05 with an integrated ethernet port, and the hotset machine has a 1756-L55. the machine sending the messages is a 1756-L61s.

They are connected in a star configuration to a Netgear switch.

I did try to run a temporary ethernet cable from the switch to the draw but had the same issues.

it's strange to me that both MSG instructions hang up at the same time. what could be causing this?

a 4th machine is also reading and writing to the L61s and it doesn't seem to be having any issues with intermittent messaging which makes me believe that all the ethernet cables are good.

could this be an issue with the overhead?

Thanks in advance. comms issues are difficult to diagnose.

 

[Ken Roach]

I agree that communications problems can be hard to diagnose. Managed switches can sometimes help, especially with physical infrastructure issues. The questions I would ask if you were on the phone with me at Tech Support (15 years ago) include:

What model of 1756-ENxT module are you using with that 1756-L61 ? Have you evaluated its connection capacity, packets/second load, and hardware interface error counters ?

Does your logic execute the MSG instructions every 25 ms with any other conditions (like the completion of the prior message), or just with a repeating timer or similar trigger ?

How many total MSG instructions are there in your whole program ?

Have you captured any .ERR and .EXERR values if the MSG instructions end with a .ER bit ?

Do you manually time-out the MSG instructions by setting the .TO bit after a period of time, or do you have the timeout changed from the default of 30,000 ms to something smaller ?

 

[Drewster1123]

*Correction, all my messages are triggered on a 250 ms interval, not 25, typo.

1756-EN2T Rev 2.001

The new MSG instructions that I am creating are all essentially setup like this:



I have not looked into the connection capacity, etc I'll need to do a bit more reading to understand that. I did pull up the diagnostics page the other day too and nothing seemed out of the ordinary for out-of-order packets and collisions etc. although I didn't look around much.

There are about 30 MSG instructions that are running on the PLC.
Most of the MSG instructions are setup like so:

which I am not particularly a fan of hence why I am trying to group them all up into blocks that fire at an interval

I haven't seen any error messages yet. the messages just hang.

I did have a manual timeout setup as well but it didn't really correct the root cause so I took it out.

below are some screenshots from the diagnostics page.

 

[Drewster1123]

Would it be worthwhile to upgrade the firmware revision? it seems like it would increase the packet capacity?

 

[TheWaterboy]

I suggest setting the MSGs in a sequence.
Trigger the first one then use it's .DN or .ER bits to trigger the next MSG and so on in a cascade fashion. (Set the Timeout value for each MSG to a smaller value so they don't wait 30 seconds when they fail).
This way only one MSG is active at a time, they run as fast as they can, and the comm buffer doesn't fill up.

 

[robertmee]

I suggest setting the MSGs in a sequence.
Trigger the first one then use it's .DN or .ER bits to trigger the next MSG and so on in a cascade fashion. (Set the Timeout value for each MSG to a smaller value so they don't wait 30 seconds when they fail).
This way only one MSG is active at a time, they run as fast as they can, and the comm buffer doesn't fill up.

This....when doing multiple messaging, I have a sequencer that steps through based on .dn or .er of the current message. Using a single trigger or complement like you are can easily overrun the messaging buffers. A tell tale sign, switch from run to program mode and if you get the cannot switch mode popup, your message buffers are full.

 

[Drewster1123]

Ok, I'll keep cleaning up the MSG instructions and then try and sequence all of them together with a short timeout. Something like below. Need to add the .en, .dn, .er and .TO bits still but I already like the way it's looking.

 

[TheWaterboy]

Do like this - forgive the bad drawing. Drop the EQU and OR the result bits of the MSG.

Then find the .timeout parameter of the MSG tags and change that value to a much smaller number. default is 30 seconds to you can see its scale (in microseconds I think) You dont really need the ONS, MSG only fires once per true.

 

[JankyPLC]

I'll echo what TheWaterboy said. I've set up some very MSG heavy programs and setting them up in a cascading fashion like shown has resolved all kinds of "untroubleshootable glitches" while allowing comms to happen as fast as the system can handle it. Bonus points you can throw a timer on the loop to see the time interval for how long it takes to get through your message sequence to check performance/debug other issues.

 

[Drewster1123]

Ok i have it setup with the Dn. and .er bits all cascading down to the next MSG. seems like im getting about an average of 800 ms to complete every message and start back at the top.

also im assuming that the first message has the er and dn bits from the last msg. I still need to set the timeout time.

 

[TheWaterboy]

First msg should have an initiation bit that toggles every few seconds unless you really need that response time.

In my case I toggle the first one, let them all run through the sequence, then the last one starts a delay timer that then starts the sequence again after some time. Again in my case I am usually using radio telemetry so can't really run the radio nonstop, or at least that's not the best idea for longevity , so I pause about 10 seconds and then run them again over and over.

I have timing records, success/fail diagnostics and enable/disable bits in the stuff I do as well.

I also clear the DN and ER bits at the end of every pass to keep things clear.

Don't forget to change the .UnconnectedTimeout element value for these MSG's. Just remove a zero, that takes you from 30 seconds to 3 seconds.

 

[Drewster1123]

would it be worth it to split into msg that need to be triggered ASAP and have a cascade for those messages and then another cascade with messages that trigger every 30 seconds or something? just curious. not sure if I would need to do that for this application but there are many other machines here that I may need to do that with.

 

[TheWaterboy]

If you need something ASAP then use Produced/Consumed Tags and not messages. It's all about realistically evaluating how fast you REALLY need that information.
If you split these into 2 sequencers at different rates, then you could *potentially* have the same kind of buffer collision you started with, though with only 2 sequencers that is unlikely.
Based on the option you are suggesting, you are getting the idea. PLCs were meant to process bits and not meant to talk so the less often they do that, the happier they are. That's much less of a problem than years past but as you see by your original post, it's still worth keeping in mind.

 

[Drewster1123]

Yeah, I'm trying my best to take all the 1 and 2 bit messages that are firing as fast as they can and combine them into 1 large block read and write per machine. I read somewhere that it doesn't matter how many bits you send in a msg, it's always going to send 500-bit frames or something along those lines. I do agree that reducing the amount of reliance on messages is ideal but unfortunately its been ingrained over the years to these machines.

Cleaning them up will help trim the fat that isn't used anymore or isn't needed. then produced and consumed tags can be made and all that. It will just take time.

I really appreciate the help from everyone though. Forming some good habits.

 

[TheWaterboy]

True, with Ethernet the encapsulation often exceeds the payload. With telemetry the payload is maybe 500 Bytes at largest, usually much less, but even the MTU of Ethernet is 1500 bytes so not much is gained by keeping it small because it will need to pad to even send it, but the PLC still has to package the data up and that's where the issue comes into play.

Something else to consider. If you package all your alarm bits together, then on the far end you can run code against an array of elements that is already in a contiguous block. You can monitor that block of elements for changes to indicate something new has happened, or something needs to be MSG'd. After all if it hasn't changed, you don't really need to resend it.

When you start remote telemetry with a 300baud radio modem on a PLC-2 you conserve everywhere and its a hard habit to break.