How to display a Major Fault on a HMI?

[Bullzi]

Hi everyone,
Got some big problems here and I could use some help.

I have a system that is using a 1769-L33ER along with 3 remote I/O Racks. 2 are 1734-AENTR Point I/O remote rack and one is a 1769-ANETR remote rack.

For HMI's I am using 2 Red Lion G308 and 1 Red Lion G09.

Here is my situation: Today the processor went into a major fault. It was a I/O fault that caused it. That was no big deal except that the HMI's all continued to show good tank levels. I assume that was the level when the controller went into a fault. So the tank levels showed good even though the levels were climbing, eventually overfilling.

I have a tag in the HMI's for IsDeviceOnline but that alarm never showed up so the Processor must have been responding to request.

I know that since the PLC was faulted it wasn't scanning the values for a change so that is why the HMI was stuck on a value that was "reasonable".

So how would you go about getting some kind of alarm on the HMI to let people know that the data they are looking at could be wrong?? Or how do I go about making forcing the tank levels to a really high value in the event of a Major Fault?

I need to find a way to try and keep this from happening again! I have other plans in place to also help but I cant have the HMI showing good data if the PLC isn't even working properly.

Thanks so much for any and all help!!

 

[OkiePC]

I don't think crimson drivers support that directly (reading the CPU status), so what you could do, is use a counter or the real time clock second in the Logix controller moved to an HMI tag which is monitored by the HMI for a failure to change states. In crimson, you could look at the bit (with the period as the delimiter) of the "From_PLC_Seconds" tag with a time delay of two seconds as the alarm config.

I think you will see the IsDeviceOnline tag work plenty fast with serial drivers but with ethernet it takes quite some time for them to drop out.

I have usually seen tags that appear on screen change to unknown status within a couple of seconds after they are not available, as long as the data is a direct reference to the tag.

Also, if you have anything in the "Simulate As" fields for tags during development, delete them, as they may appear when comms have been lost if I recall correctly.

 

[daba]

You could create a Program and Routine in the "Controller Fault Handler". In there you could retrieve the fault code and type with a GSV, and get the HMI to display these if not zero.

The processor will execute this handler code before shutting down on a Major Fault.

You can even reset the Type and Code values and SSV them back, IF, AND ONLY IF, you decide the fault is not severe enough to shut the controller down.

Also, the default for 1769 I/O is to major-fault the processor if a module connection is lost while in Run mode. You can change this on the Connection tab of the module properties pages. Then the controller won't shut down with a major fault, and you can write standard code to decide what actions to take.

 

[osmanmom]

are G308 Software HMI possible to program internaly (programming icon-structure text) to change another page by timer if keypad/touch not active few minute?

then the HMI memory can refresh.

 

[James Mcquade]

this is one possible solution.

Turn on an output that turns on a relay. as long as the minir / major fault bit is off, the output is on.
that relay powers indicator lights and an alarm horn.

when the processor has a fault, the output is turned off which
in turn turns off the relay. the indicator light and alarm horn sounds.

regards,
james

 

[nwboson]

Yes, I have run in to the same problem. So my solution was to write a little heartbeat function on the HMI that gets called every tick. That function checks to see if it sees a heartbeat bit toggling from the PLC (once every two seconds). Don't use once a second because the HMI and PLC can alias (remember Nyquist) if they are both toggling at same rate, so slow the PLC heartbeat half the speed of the Red Lion ticks.

If it does not see that bit toggle for awhile (ten ticks?), it sets a Red Lion tag that tells me all data from the HMI is 'frozen' garbage. So the idea is you count the number of times the tick function has been called since the last time it saw the toggling bit from the PLC: it should always be 1 or 2. Hiccups happen (not sure why), so I'd count to at least 5 before I was sure it was messed up. I use ten because I'm really just looking for a croaked PLC or connection: it does not have to be instantaneous.

Then I overlay the whole screen with a message that indicates communication has failed (if the 'frozen' garbage bit is set).

Oh yeah, okiepc is right on. Worth re-reading. IsDeviceOnline works, if I recall, but takes a long, long time (a minute?). Try some tests with IsDeviceonline(): force a PLC fault and check how fast it is. You may need this tick routine / PLC heartbeat approach to find the lockup quicker.

 

[Bullzi]

Thanks for the tips everyone. I know the IsDeviceOnline is not working during a Major Fault. I just seen the same problem happen again. I knew it was coming because they had to shut down power to some parts of the plant. So before I reset the major I looked at the HMI and there was no PLC Offline alarm. I also looked at the tank levels on the HMI and they were showing exactly what the SCP AOI output tag had in it. So it appears that the HMI was reading a register that had data in it, and the PLC was responding to the HMI's request. Since the logic wasn't scanning it was just froze on the last reading.

I am working on a Fault Handler for the Processor. Here is what I have in mind. Let me know what you think:
On Major Fault
1. Set a bit for the HMI to read PLC_MAJOR_FAULT. When active I will put a pop up on the screen warning that the PLC data is bad.
2. Write a 25 into level register for each tank. That will drive it over full scale 0-20ft. That way the HMI will show over full tanks and no one will try and put more into the tank.
3. If the fault is caused by no communication to a RIO adapter I might clear the alarm and see if it comes back. If it does I wont clear it a second time.

Since I haven't used a fault handler before I have a couple of questions:
1. If the Fault Handler only runs just before a shut down on fault, is there a way to "retain" the number of faults? For example: I have a fault again, my program looks for the code and determines that it is a I/O fault. I clear the fault and write a 1 to a tag to let me know that the fault happened once. Will that tag retain the 1 after the fault is cleared and the PLC is back in the run mode?
2. Does the Fault Handler Routine work with the Controller Scoped tags or does it only work with tags in the Fault Handler?

NWboson, Could you send me your Crimson code you did for that Heartbeat? I think I understand what you are talking about but if I could see it, that would help.
Thanks!!!

 

[osmanmom]

2. Write a 25 into level register for each tank. That will drive it over full scale 0-20ft. That way the HMI will show over full tanks and no one will try and put more into the tank.
e run mode?

this means you need 1 additional address 1 from PLC and 1 from internal HMI.
how you to move this additional address in RedLion HMI ?

2. That way the HMI will show over full tanks and no one will try and put more into the tank.
e run mode?

if this manual mode?why not make interlock due major/minor fault.

 

[osmanmom]

Others reference

http://www.plctalk.net/qanda/showthread.php?t=60950

https://rockwellautomation.custhelp.com/ci/fattach/get/155681/

 

[nwboson]

So here's some sample heartbeat Red Lion code. Two functions to write, one called fTick(), one called finit(). fTick called once per tick (from Display Pages 'On Tick'), fInit called once at startup (from Display Pages 'On Startup').

fTick:
// This function is called from the Display Pages 'On Tick'.
// So this function will be called once per second

// The variable plcHeartbeat is a PLC flag tag, read only, from the PLC
// that the PLC toggles once every two seconds.
// The variable prevPlcHeartbeat is an internal flag tag used to keep track
// of changes from the last reading of the plcHeartbeat
// The variable iTicksSinceHeartbeat is an internal numeric tag used to keep
// track of how many seconds since last heartbeat seen.
// The variable bPlcIsDead is an internal flag tag we set when we have
// determined our connection is dead or the PLC is dead.

if( plcHeartbeat != prevPlcHeartbeat ) {
// If we've seen the heartbeat change, all is good: there has been zero
// time since (zero ticks) since we've seen a heartbeat
iTicksSinceHeartbeat = 0;
prevPlcHeartbeat = plcHeartbeat;
} else {
// This function was called but we did not see a new heartbeat.
// So increment our once a second counter of the time since last heartbeat
iTicksSinceHeartbeat ++;
}

// If it's been 10 ticks (seconds) or more since we've seen a heartbeat, get mad
// Anything over 5 seconds probably OK here, but less than 5 might alias?
if( iTicksSinceHeartbeat >= 10 ) {
bPlcIsDead = 1;
} else {
bPlcIsDead = 0;
}


fInit:
// This function is called from the Display Pages 'On Startup'.
// So this function will be called once when the Red Lion gets going.
// We use it to initialize some variables.

iTicksSinceHeartbeat = 0;
prevPlcHeartbeat = plcHeartbeat;
bPlcIsDead = 0;

 

[JesperMP]

I think that you are describing a wrong problem and therefore also the wrong solution.
The problem of the HMI not indication a faulted system is by itself serious enough, but IMO not directly related to the overfilling problem.
If overfilling a tank may have serious consequences, and the filling is done completely manually, only with visual check of level sensors (*), then it may be necessary to implement an automatic safety to inhibit the filling.
IMO opinion it is better to inhibit the actual filling than just the indicator light.
If the filling is done via a local pump or valve, then inhibit the pump or valve.
If the filling is done via an external system with its own pumping system (f.ex. trucks from suppliers), then you may have to install a shut-off valve at the connection point where the external system hooks up.
I have done the last thing in the past, in addition to both green and red lights there was also an automatic fail-safe shut-off valve.

*: I am guessing that that is how the system works for Bullzi. Otherwise it would not be an overfilling issue.

 

[osmanmom]

Thanks for the info,

 

[osmanmom]

it looks issue are rise,

1.) HMI frozen,operator attend on the HMI visual check the level,then top-up a raw in tank if low without touch the screen,switch ON motor an another system.

2.) Plc or I/O fault,

3.) HMI request heartbeat from PLC if offline/etc

4.) Refresh the HMI memory/page.

 

[osmanmom]

Copy and paste

Fault Handling info

 

[Bullzi]

Jasper,
You are very close on your analysis of the system:
We are using the trucks pump to unload into the tanks so we cant stop the pump.
The Driver is to look at the local HMI and decide if there is enough room for him to unload. If not he is to go to the other location.
There is a system for pumping the tank out based on level of the tanks. When the tank reaches 5' it is suppose to start pumping until it reaches 3'. With the PLC in Fault that wasn't happening.

I fully agree with your assessment that we need to put a automatic valve in the line to keep the driver from overfilling a tank. Unfortunately my customer is against the idea since most of the truck pumps are Gear Pumps and they don't have PRV's. The hose would blow off the truck and make a mess everywhere. At least in the tank there is secondary containment if the tank overfills.

I really hate these types of systems. I like to install my own pump that way I have total control but this is the hand I was dealt.

Thanks everyone for the help!!