Messaging stops periodically

[nukkin]

I have a CLX messaging to both CLX's and SLC's.

I have discovered that the Read messages originating in the CLX stop reading about every 60 seconds... pauses for about 10-13 seconds, then comes back and reads normally again... repeat... constantly. This may be happening with the writes as well, but I have not checked for that.

I checked for errors by moving out the .ERR codes to a register to view and I very occasionally get an extended error code of 204= Unconnected message timeout. However, if i had to guess, this error only occurs maybe once out of every 10 pause events... but it is always during the puase event and at no other time.

I have tried to take my time slice up from 30% to 50% with absolutely no change in this performance.

What could explain this? Where should I look next? I just dont know what else to look at.

I have included a trend of a heartbeat i set up to read 1 second pulses from two separate SLC's. These are reads that were already moving words, but had extra bits not being used. i slipped a pulsing bit into these extra bits to be able to trend the Read. Check it out...

that is a 10 minute trend window, you can see it drops out multiple times then comes right back.

 

[jkerekes]

We need more information: how many messages?, how frequently?, how many @ the same time? How are you triggering the MSGs? Can you use Producer/Consumer on CLX to CLX comms?

 

[OkiePC]

I would look at the SLC side of things. Perhaps you can change the MSG Servicing Select bit (S:33/6) and get better performance?

 

[nukkin]

I have about 37 active messages (both read and write) in this controller. I just went thru and captured their time intervals. They are not sequentially staged or anything, they just execute when the timer is done, which from what I gather is probably bad, right?
have a look at this spreadsheet picture to get a better idea...

 

[jkerekes]

Wow, that's a lot of messages! I think you need to have a better way of triggering them. A much more repeatable way. Do they all need to be that fast too?

 

[OkiePC]

They are not sequentially staged or anything, they just execute when the timer is done, which from what I gather is probably bad, right?

Yes, since there are so many, you probably need to stagger them at least, but reducing their numbers and frequency would be advisable as well.

 

[nukkin]

I bet alot of these messages do not need to fire at the speed rate they currently do. I do have one that needs to... and becuase of this pause time, i got a machine that pauses occasionally when its supposed to open a clamp. its waiting for a bit coming accross a Read MSG to show up. Even though the bit is set to 1 in the SLC, the CLX cant see it if its during a pause, so the entire line holds waiting for the read to re-establish itself. once that happens, the bit shows up, and the clamp releases, and all is good in the world. I dont know of any other problems becuase of this issue currently, their very well might be, but I will have a hard time justifing rewriting all that code to stage the messages over this one little clamp issue. See what I mean?

so, if there was a way to check and verify that i have a comm buffer that is filling up, or something that definitively says all these messages are certainly the cuase of my problem, well that would force me to schedule these read/writes and i would have no problem with that. However, if our current structure is frowned upon, but may or may not be the problem, i'd hate to spend all that time and still come up with the same problem. The only way to be sure to have one fix and be done, is to spend the extra time to troubleshoot and verify the problem, right?

On that note, what things can I look at that would prove were the problem lies? Any tips would be greatly appreciated.
Thanks.

 

[nukkin]

Another person has showed me the Rockwell tool called "Task Monitor". I pulled that up and one of the things it will show me is called "networking". It seems that everytime my Reads/Writes lockup, the Task Monitor shows me that "Backplane Transmit Buffers" are maxxed out at 100%.

Another funny thing is, I have found that 6 of my 37 messages do work perfectly fine. Only 31 of the messages have a problem. I started to order them on my spreadsheet in order of program solving, but I have already found that its not a case of the first 6 worked, then the 7th locked everyone else up behind him. According to my investigation, 3 fire off fine, then at least one (Possibly more, Im not done yet) then has trouble, then three more behind it work fine, then the rest bomb out.


Still Looking...

 

[Ken Roach]

Add two new columns to that spreadsheet: "Cached" and "CIP".

Cached messages establish a CIP connection once, then re-use it every time they fire. This uses up CIP Connection resources, but frees the crucial Backplane Buffer resource.

By "CIP" I mean DH+ and RIO versus pure EtherNet/IP or ControlNet. Any ControlLogix is going to have a limit on the per-channel, per-DHRIO, and per-controller messages you can send over DH+ (or block transfers over RIO).

 

[The Plc Kid]

Can some of the date from several messages be combined?

It may be a little better if you use reads only and no writes.

 

[asterof]

CLX and Messages

You can grind the PLC to a halt with to many messages
I would increase the UCB size and stager the messages
AB has sample code in the samples directory to show
you how to do that. Remember that everything goes across
the backplane and if you are filling up the que then
IO and every module on the Backplane stops when you fill
it up.

 

[OkiePC]

Also, you don't want to wait for the default timeout before aborting a message that is failing. I have gone as far as to take a node out of my sequence for a predetermined time period to ensure I don't keep clogging up the buffers when there really is a problem.

It is critical to properly handle errors with this much messaging. I agree with other's comments about trying to coagulate fewer and larger chunks of data with fewer messages (offload data collection to peers on the network, have them send it to you...)

 

[nukkin]

Add two new columns to that spreadsheet: "Cached" and "CIP".

Like this?

They are also in order of program solving, except for the scheduled routines.