SLC processor faults when inserting a module in Run mode ...

Greetings to all,

I had originally posted this here:

recent thread about “stuck runtime errors”

but then I decided against hijacking someone else’s thread ... I’ve decided to leave in the references to the former thread for context ...

as always, Ken seems to be on top of things ... when I read Hedge’s post the first time, I got the impression that his “let’s-try-this-on-another-set-of-hardware-test” was made on a benchtop rig without connecting it to the actual field machinery ... now that I read it again, it looks like his reference to “ONLINE” probably refers to placing the whole system in the operating mode ... if that’s the case, then I think that Ken has probably hit the nail on the head ...

now for something sort of on a tangent ...

the reason for this post is to mention something that I’ve been aware of for several years but I never really felt quite “right” about it ... it’s not all that important but it’s just sort of puzzling ... anyway ...

suppose that we have an SLC system working fine and it’s currently in the Run mode ... we (for one reason or another) want to remove one of the I/O modules without shutting down the system first ... specifically, we want the rest of the system to keep chugging along in the Run mode while we yank out just one module ... normally we can’t do that because the SLC processor will fault just as soon as it misses the module ... but ... as most of us know, we can go to the Processor Status file – to the IO tab – and put a zero on the “I/O Slot Enables” table for the module that we’re interested in ...

and NOW we can pull the module out of the chassis and the SLC will keep right on running without a fault ...

so far so good ... now here’s the rub ...

if the processor is still in the Run mode when we push the module back in again, EVEN WITH THE SLOT DISABLED, we always get a “stuck runtime error” and the processor faults ... bummer ... I’d sure like a way to replace an SLC module while in the Run mode without having to shut down the whole system ... but so far, I’ve never been able to come up with a way to do that ... it’s just very annoying that you can pull the module OUT ... but you can’t put it back IN ... somehow that just seems “wrong” to me ...

so does anyone have a workable solution to this? ...

What's the error code?

Ron:

I've never had the courage (spare resources in case something went hideously wrong) to try hot-swapping SLC modules.

What is the exact error message you get when you insert the new module? If you've led a clear life, it might be one of those "recoverable" ones that can be fixed with a fault routine.

(My guess is that it won't be recoverable (and not because of any riotous living on your part). I can't say that I understand anything about backplanes, and just how the processor sends signals to each module on it, but I can see that inserting a module live (which would be getting power at the same time it is starting to communicate) could give the backplane cop a migrane, and so cause the process to stop (due to an unreliable backplane).

It would be on the same order of error as not having the correct rack size configured, even though the extra/missing slots are disabled

Consider yourself lucky you haven't blown a card or something else. The SLC system was not to designed to be "HOT Swapped" like the Control Logix system. If you haven't blown something yet, I would not suggest testing anymore. I may be wrong and if I am I know someone here could set me straight. Just as long as my foundation is OK.

Bitmore--Don't start none, won't be none.

Hee-hee.

I'm with Bitmore. This sounds like voodoo black magic to me. Unless you've found a spot in the SLC manuals which specifically states "THIS MODULE IS ADD/REMOVABLE WHILE THE RACK IS POWERED," you're probably going to get bit at some point.

I'm not saying there isn't a work-around to fool the processor. AB has got LOTS of work-arounds. But, I don't you should count on this to work at 6pm on the last day of your site visit.

AK

... do not attempt this at home, kids ...

Greetings to all,

first to Allen ... the Major Error code that I get is: “1F51h” and this is the description that pops up ...

A "stuck" runtime error is detected on I/O module. Exact slot cannot be determined.

Click to expand...

Allen said:

... it might be one of those "recoverable" ones that can be fixed with a fault routine. ... My guess is that it won't be recoverable ..

Click to expand...

you’re right, it doesn’t seem to be recoverable ... I’ve tried everything that I can think of to recover from this type of fault but nothing that I’ve tried yet has worked ...

Bitmore said:

Consider yourself lucky you haven't blown a card or something else. The SLC system was not to designed to be "HOT Swapped" like the Control Logix system. If you haven't blown something yet, I would not suggest testing anymore. I may be wrong and if I am I know someone here could set me straight.

Click to expand...

no, you’re completely correct, Bitmore ... the SLC is NOT designed for “hot swapping” ... but like someone once said: “sometimes a man’s gotta do what a man’s gotta do” ... the one or two times that I’ve heard about this being tried in the field were in plants where even a temporary system shutdown fell squarely into the financial “bad” column ... it didn’t work of course ... so the lesson to be learned is obviously: if you might need to hot swap a module someday, then the SLC platform is not the best choice ... and you’re right again, the ControlLogix platform IS designed for this type of treatment ... the official buzzword is “RIUP” (Removal and Insertion Under Power) ... I’ve hot swapped a bunch of those with no problems at all ...

as for my tests ... I’ve been tinkering around in the lab with fully functional hardware ... but this gear has been around the block a few times if you know what I mean ... I’d sort of hate to blow some of it up ... but then I wouldn’t lose any sleep over it if I did ...

but ... I’ve definitely decided not to try this myself out in the field until (unless?) I’ve gotten it to work in the lab ... and actually, unless you guys can come up with some more suggestions, I think I’ll just say that it can’t be done at all and get on with my life ...

it’s just (as I said in my first post) the whole thing just feels “wrong” somehow ... I mean, the slot is DISABLED ... and so the processor doesn’t mind when I pull the module out ... and it’s still DISABLED when I put the module back in ... so why does it fault going in - but not coming out? ... oh, well ... maybe this is just one of those little mysteries in life that we were never meant to understand ...

and you too are correct, akreel ... it's not really supposed to be done ... and so I'm beginning to doubt that I'll ever get it to work ... but IF (and that's a big IF) it could be done, then it would definitely go into my little notebook of "advanced guru tricks" ... heros and legends are made of stuff like this ...

thanks to everyone who responded ... I’ll let you know if anything new comes up ...

1F51h

From the RSLogix500 help files:

Code:

[b]Code     Description[/b]
xx51     A "stuck" run time error was detected on an I/O module. This error is 
         non-recoverable.

[i]©RSLogix 500 - Copyright Rockwell Software 2000, 2001, 2002[/i]

Click to expand...

So there you have it. If you get the fault, you die.
I suspect that the reason that you get it is that insertion of the module either interupts a good message on the backplane (and so the backplane cop knows that there's a module sending bad data out there somewhere, but doesn't know which one), or the new module goes through some kind of powerup CMOS routine, and hasn't gotten the word that it's not supposed to be talking (not having recieved the "inhibit module in slot" configuration) until it was too late.

It can't be done Ron. Let it be.

Re: ... do not attempt this at home, kids ...

Ron Beaufort said:

and you too are correct, akreel ... it's not really supposed to be done ... and so I'm beginning to doubt that I'll ever get it to work ... but IF (and that's a big IF) it could be done, then it would definitely go into my little notebook of "advanced guru tricks" ... heros and legends are made of stuff like this ...

Click to expand...

If it worked, which we've now determined it doesn't, I don't think I'd file it in my "advanced tricks" notebook. I'd file it in the "do not use without a SIGNED waiver" notebook. Imagine if you used a "cheat" like this one day and instead of saving the day, it let the smoke out in front of the customer. I don't think "this worked in the lab" is going to save much face.

I'm not saying I'd never do anything like it, and never have. But, just be careful! I told Tom last night about the time a program change I made (against my better judgement), to remove a permissive status because equipment wasn't ready yet, resulted in a robot crash. This happened in spite of the fact that I tried my best to prevent the "temporary bypass" from becoming an "everyday tool."

AK

You can hot swap modules in a SLC500 system by first going to your status file while online in RSLogix, be sure status file is in "structured view" if so your data table will have tabs accross the top of it, then go to IO menu, enter a "0" into the specific slot you want to hot swap, remove the desired card from that slot and re-insert new card into slot. After change is complete enter a "1" back into the slot to resume activation of that slot.

It can be done, would I recommend it? Not A Chance, have seen the smoke get let out many a times before by "Hot Swapping" and I can say it is expensive to put that smoke back into the card and even more embarrasing when it is your only card and everyone behind you is asking is it fixed now!!!!

Wait a minute, isn't disabling the slot specifically what Ron said DIDN'T work in the very first post? What's the REAL truth here?

... and so it was not 2-B ...

Greetings to all,

well ... I’ve decided to accept the fact that it CAN’T be done ...

specifically: you CAN’T hot swap a module in an Allen—Bradley SLC system ...

you can disable a module and REMOVE it from the chassis without causing a processor fault ... (yes, I’ll agree you should NOT do this – but you can) ...

on the other hand, even though the module is disabled, you can NOT REINSERT the module without causing a fault ...

I’ve tried it every single way that I can think of and I can’t get it to work – so I’m hereby officially giving up ...

but (for anyone interested) here’s my take on WHY I would ever attempt to do such a radical thing in the first place ... and that’s assuming that it could be done ... and remember that I’ve already admitted that it can’t ...

let’s just suppose that I’ve been called out to a job site to help with a serious situation ... the problem has already been tracked down to one defective I/O module located in an SLC chassis ... of course the recommended solution is to shut down the machinery ... power everything (including the SLC processor) down completely ... replace the defective module with a handy spare ... and then power everything back up again ...

so that’s what we SHOULD do ... we all know that ... we all agree on it ... but ... the trouble is that the plant manager REALLY wants to keep the plant in operation ... in fact, he absolutely DEMANDS it ... according to him (and we believe him), shutting down this particular process without losing a bunch of expensive product will take at least an hour or so ... and it will take at least two or three more hours to get it back up and producing again once the repairs have been completed ... he can’t keep his production schedule if he shuts down the plant ...

but suppose that I firmly stand my ground ... no, sir, Mr. Bossman, it doesn’t matter what you WANT to do ... this particular system can NOT be hot swapped ... it might blow out the new module ... it might damage the processor ... it might result in the end of civilization as we know it ... I can’t be held responsible ... I refuse to be a part of this madness ... it simply can NOT be done ...

meanwhile ... Doofus Dave, the plant grunt, reaches over and yanks the bad module out ... and then shoves the new module in ...

and somehow ... someway ... the whole durn thing keeps right on chugging along ...

so where would that leave me – the guy who insisted on doing everything strictly by the book? ... what are my chances of ever living this one down? ... and how about formerly “Doofus” Dave – who (being too dumb to know that it couldn’t be done) just “went-ahead-and-DONE-it” ... suddenly he’s become a legend in his own time ...

so that’s kind of what I had in mind when I said that IF (and that’s a big IF) this hot swap thing would work, then I’d certainly put it down in my notebook of guru tricks ... I can assure you that I would NOT go around making a common practice of hot swapping every SLC module that I came across ... and I’d DEFINITELY warn the boss that this is almost certainly GUARANTEED to fault his processor and shut down the plant ... and I’d be SURE to mention that the module, processor, chassis, nearby streetlights, etc. could all be DAMAGED - if not totally DESTROYED - by the foolish course of action that he’s insisting that I take ...

and if there were ANY (I repeat ANY) safety issues involved, then I’d warn him accordingly ... and then simply leave the building as quickly as possible ...

but ... once all of the histrionics were out of the way ... then I’d probably take a deep breath and go ahead and hot swap the module anyway ... (or better yet, get Doofus Dave to do it while we all watched) ... then if things went badly, I could always say “well, I told you so” ... but then again, maybe fate would smile upon me that day ... maybe it really would work just like it did all of those times back in the lab ... as I said before, heroes and legends are made of stuff like this ... what’s my alternative anyway? ... if it COULD be done, I’d sure hate to see Doofus Dave rush in and succeed where I had feared to tread ...

so that’s my position, folks ... and to fully understand this, you need to know that I’m the type of guy who (honestly) puts his seatbelt on just to move the car around in the driveway ... no joke, the seatbelt’s been a habit for years now – I hook it up without even thinking about it ... on the freeway, I’m the nut who goes five miles BELOW the posted limit ... sorry ... yeah, I know you hate it, but that’s just part of the magic that is me ... I also line up my shoes and my wife’s shoes right next to the door whenever we’re traveling and spending the night in a hotel ... the laces are left untied and loosened so that the shoes are ready to slip on in the dark - just in case of a fire alarm ... and I take the time to notice exactly in which direction the nearest fire extinguisher is located ... and I even look at the little dial to make sure that it's charged up ... and the route to all available fire escapes ... which one’s the closest? ... but what if it’s blocked? ... in short (no, sorry it’s too late for that isn’t it?) I’m a very careful kind of guy ...

so once again, you can NOT hot swap an SLC module without faulting the processor ... I accept that now ... and if you try it anyway, you certainly COULD damage something ... I fully agree with that ...

but ...

if I COULD somehow come up with a way to do it, do you think for one minute that I’d simply say “no, since you’re not really supposed to do this, then I’ll just force myself to forget about it” ... and NOT make a note of it? ... no way, Jose ... this one would certainly go into the book ...

because as I said before ... sometimes a man’s gotta do what a man’s gotta do ...

and to electrified3: I certainly wouldn’t recommend that you “go-back-and-try-it-again” ... but the method that you mentioned is EXACTLY what I’ve tried (many times) in the lab and NEVER gotten it to work ... not even once ... specifically, every time I plug a module IN, I always get a fault ... even with the slot disabled ... taking one OUT works OK as long as the slot is disabled ... but not putting one in ... if you can get it to work, then I’d sure like to know how ...

as always, thanks to Phil for a place where we can all come to share ... and to EVERYONE who responded ... my work here is done ...

Ron,
how about this:
When you want to insert the module, try to do the following in quick sequence (as fast as you can):
1. With the keyswitch put the CPU in PROG mode.
2. Insert the module.
3. With the keyswitch put the CPU in RUN mode.
This should clear any MAJOR ERRORs including the "stuck" runtime error.
The program should continue where it left including the last state of the outputs. Obviously you must check your program for the usage of the bit S:1/15 "first pass", and possibly change it to allow the "restart on the fly".
4. If OK, go online and set the module slot in question to be scanned.

I certainly wouldnt attempt the above myself. You should experiment with something non-critical first.

NB: Just to state the obvious:
Doing the above should be when the program is in a state where no harm can be done when the scanning is interrupted.

Man, Doofus Dave is everywhere!

You're right, Ron, there is a flip-side to my earlier comment.

Being a doofus myself I assumed that hot-swapping was permissible. I had an offline system and was just setting up the PLC for a project. I hot-swapped the PLC and bricked it. Not understanding the cause I bricked another PLC for good measure. I wish I had read this conversation before making this expensive mistake!

OLD THREAD ALERT

kb9okv said:

Being a doofus myself I assumed that hot-swapping was permissible. I had an offline system and was just setting up the PLC for a project. I hot-swapped the PLC and bricked it. Not understanding the cause I bricked another PLC for good measure. I wish I had read this conversation before making this expensive mistake!

Click to expand...

Maybe you did not.... have you tried a manual reset to factory default?