Long-winded question about 5580 and EN2T

[danno4200]

I have a question. I work in a very large plant and this one (of many hundreds of control cabinets) contains one 5580 (1756-L83E), two 1756-L73 and seven 1756-EN2T in the 16 slot rack. The controller in question here is the 5580 (1756-L83E). It is having thousands of minor faults per second. I know what the cause of the repetitive (T:04 C:51) fault is. It is due to some data being pushed into a specific index of an array that is a few characters over sized. I will be fixing it, but I'm doing a little RCA here and trying to determine if it may be reasonable to assume those thousands of faults could be responsible for the primary EN2T module (acting as the host connection to a cloud service) randomly losing communications to said cloud service? This is a 24/7 operation and it has happened about 5 times (at least) over the last 5 months causing significant downtime each time (hour, to a few hours). Only recently did I notice all the minor faults, but I traced it back to have started happening about 6 months ago when someone made some logic changes that began overflowing the array index. I know how the logic functions would obviously play a role in this, but I'm just wondering if anyone might agree that these thousands of faults occurring steadily 24/7 could possibly, randomly cause the network issues we're experiencing. Like in combination with allll the other processing this controller is constantly doing. I will be fixing the logic, but someone else wants me to replace the EN2T module tomorrow morning, so I am going to wait a while to fix the logic to determine if it is the module or the faults. Just wondering if anyone might think my theory could be very likely. Also, the network cables and switches and all other devices from inside the panel and out appear to be just fine and IT ran traces on the lines with their fancy little network analyzer meter tool and determined they see no issues (between both the current line/port being used as well as the spare unused emergency line/port right next to it).

 

[Ken Roach]

Minor Fault T04:C51 - Program Fault: LEN value outside DATA limit.


I don't think that's the cause of your communication problems. At worst the controller is jumping to a FAULT routine and resetting, or the array-related instruction is failing to write the data you expect. Unless you see watchdog errors or really high execution times or it's running out of user memory, I don't think the two things are related.

While it's nice to know that the local wiring that IT was able to test are in good condition, their diagnostics probably ended at the cabinet door or at least at the Internet router.

If the machine were mine, I would probably plop in a Raspberry Pi (or OnLogic / Kunbus version) nearest to the switch and set up a Node-Red instance to pester the cloud service with a PING or a TCP keepalive or a login, and also poll the ControlLogix for a tag value. Count the successes, log the failures.

 

[Contr_Conn]

This fault usually happens when you are trying to manipulate with strings and exceeding available string size or pointing to an element that does not exist. MID, Insert and Delete instructions can cause this.
I recall seeing this error with older Modbus TCP AOIs but it does not happen in recent version.

 

[Ken Roach]

Ohh... it didn't occur to me that the connection between this big honking automation system and the "cloud provider" might be Modbus/TCP.

 

[danno4200]

Interesting replies. Thank you. There are 4 large profi networks, each connected to an individual EN2T via an SST profibus ETH module. I'm going to continuing digging here in a few minutes, but I really appreciate your replies and input so far. Thank you!

 

[Contr_Conn]

The technote for Modbus TCP AOI is BF28611 (Access Levels: TechConnect). It may not apply to you, but this minor fault most likely coming from the String handling.

 

[danno4200]

Yes sir. I'm aware of what's causing the issue. The logic used to be pushing a string of like 16-20 characters into a SINT array with a size of [26], but now about 50% of the time it's pushing 33 character strings into the same array. I just need to modify the array data type and size. There was a user defined data type that was supposed to be implemented and it was not. I'm just trying to root cause this random communications issue and wondering if these high quantity minor faults could be the issue, and not network switch, EN2T or cable issues. Thank you for your reply sir.