Devicenet Going on Strike!

Join Date
Apr 2002
Location
Just a bit northeast of nowhere
Posts
1,117
Well, not exactly, but not too far off the mark, either...

We've got two machines that run SteepleChase Visual Logic Controller...how I hate them... but that's not important right now...

On one of these machines, DeviceNet (implemented with an SST brand 5136-DN-ISA scanner card) will periodically crash. Boom, all communications lost. VLC does not realize this, and begins faulting out for other things related to I/O, like time-outs. If I had a nickel for every time I've been told, "It's got ten different faults, all at once! We have to check all of them!"...

Ahem.

So devicenet is crashing. Now, we've already changed the scanner card, but the problem remains. First surfaced about 8 weeks ago as a minor nuisance, than increasing in frequency to major-pain-in-the-tush status. Monitored the bus power supply, but couldn't find an intermittent on it (which is not to say it isn't there, I have doubts about my tester's reaction time to transients).

Question 1 : Has anybody sufferred mysterious complete network collapses on Devicenet, and what did you find? I don't care if it was a mouse (the furry sort) nesting in the wiring, I'm dying here. Need ideas on where to look.

Question 2 : I have a second DeviceNet card configured to monitor activity on the network. The thing is, I don't know what I'm looking at. I have a histogram of network usage (is 65% excessive for 8 nodes?) and a node-by-node breakdown of real-time usage, and some other stuff that might be useful (hey, I only got it running an hour ago :D). Any information would be helpful, and I do mean any.

This is probably not enough data for our Grand-Masters to go on, but I'm not sure what to look for here, so if you'd like me to check something specific, lemme know and I shall get me hance and probe the little... ahem...

Thanks!

TM
 
Everybody off the Bus !

Let's characterize the crash first. Most likely what you're seeing as a total network failure is a called in DeviceNet circles a "bus-off" condition. A-B scanners report this as an Error 92, and most slave devices chime in with a blinking red Network Status LED. Does your SST card driver give you a code like that ?

I'll run you through my usual litany of questions, though because you implied the system ran fine for some time they're going to be pretty basic:

1. Check voltage. 24 volts DC nominal between black and red wires would be nice, though many devices will run between 14 and 25. Less then 5VDC common-mode drop between extreme ends of the network is imperative.

2. Check termination resistance. Blue-White should measure 60.5 ohms or a little less (two 121 ohm resistors in parallel with some high-resistance loads at the transceivers).

3. Check shield grounding. Lift the shield at the single-point ground and make sure there isn't another path to ground. While the shield is off, measure voltage from shield to ground.


Now go around a do the "wiggle test". This found network-crashing wiring problems in a couple of cases; one in which the V+ shorted to shield because it had been nicked during wire stripping, and another in which the PLC scanner drop somehow had the blue wire not tightened into the plug. That one was characterized as "every time I open the cabinet the network crashes"... my suggestion that the PLC might be afraid of him didn't go over well.

Triple-check the termination resistors. If one has been pulled off, the network might keep running for a while but be subject to disturbances that cause Bus-Off.

Often Bus-Off is not a massive blank space in bit timing or a voltage spike, but rather an accumulation of CAN framing errors that finally passes the threshold of tolerance, usually for all the nodes at once. Unfortunately, most DeviceNet interface boards (like my 1784-PCD and your 5136-DN) are only interested in complete packets of traffic; they filter out bad CAN frames at the silicon level.

My favorite tool for CAN testing is SST's NetMeter. If you can rent one of these or a Synergetic DeviceNet Detective for a day you'll be able to measure continuous levels of CAN errors and maybe you'll discover a failing transciever inside one of your nodes.

If you could list the eight nodes on the network and tell us something about the overall length and installation that would help too. I've had plenty of "it's just forty-five feet on the drop, that's only twenty feet more than the spec." explanations of why wiring was intentionally done to violate the cable recommendations for DNet. The 20-foot drop rule is pretty hard and fast.
 
What are the status LEDs on the nodes telling you? I've seen a case where the DeviceNet scanner stopped talking to a PLC, yet continued to scan the network, updating the outputs with whatever values it happened to have. Believe me, that got the attention of the company that made the scanner! It's possible that Steeplechase has stopped communicating with the scanner card.

Having said that, I agree with Ken that the most likely culprit is the wiring. The Synergetic DeviceNet Detective that Ken mentioned has more capabilities than the SST software. One additional thing you can do is to use the SST software to log all of the message traffic on the network to a file. It only allows approximately 1000 messages to be logged, but you can filter out normal traffic and maybe catch what happens just prior to crashing.
 
Heading off to check stuff

Thank you, gentlemen, that is exactly what I was looking for!

Ken :

>>Let's characterize the crash first. Most likely what you're seeing as a total network failure is a called in DeviceNet circles a "bus-off" condition. A-B scanners report this as an Error 92, and most slave devices chime in with a blinking red Network Status LED. Does your SST card driver give you a code like that ? <<

Bus-Off sounds about right. Watching the DeviceNet monitor that came with the SST software, it seemed that everything was going down at once. The little blinking red network light does come on on every node, all at once, and the card as well. I hadn’t checked for the card fault number, but will next time it crashes, and I’ll let you know what I find.

>>I'll run you through my usual litany of questions, though because you implied the system ran fine for some time they're going to be pretty basic: <<

I goofed there – spoke to the engineer. It didn’t start 8 weeks ago, it started 8 weeks after it arrived. Little more than a year now, but very intermittent at first.

>>1. Check voltage. 24 volts DC nominal between black and red wires would be nice, though many devices will run between 14 and 25. Less then 5VDC common-mode drop between extreme ends of the network is imperative. <<

Power supply was my A#1 suspect. Just slapped a filter cap on it in hopes of eliminating transients from the welders. I know CAN is supposed to be impervious to noise, but I never saw a 24 Volt circuit that was. And I’ll check on the differential drop.

>>2. Check termination resistance. Blue-White should measure 60.5 ohms or a little less (two 121 ohm resistors in parallel with some high-resistance loads at the transceivers). <<

Will check.

>>3. Check shield grounding. Lift the shield at the single-point ground and make sure there isn't another path to ground. While the shield is off, measure voltage from shield to ground. <<

Will check also.

>>Now go around a do the "wiggle test". This found network-crashing wiring problems in a couple of cases; one in which the V+ shorted to shield because it had been nicked during wire stripping, and another in which the PLC scanner drop somehow had the blue wire not tightened into the plug. That one was characterized as "every time I open the cabinet the network crashes"... my suggestion that the PLC might be afraid of him didn't go over well. <<

Will check again. Never hurts to be sure…

>>Triple-check the termination resistors. If one has been pulled off, the network might keep running for a while but be subject to disturbances that cause Bus-Off. <<

Gotcha.

>>Often Bus-Off is not a massive blank space in bit timing or a voltage spike, but rather an accumulation of CAN framing errors that finally passes the threshold of tolerance, usually for all the nodes at once. Unfortunately, most DeviceNet interface boards (like my 1784-PCD and your 5136-DN) are only interested in complete packets of traffic; they filter out bad CAN frames at the silicon level. <<

>>My favorite tool for CAN testing is SST's NetMeter. If you can rent one of these or a Synergetic DeviceNet Detective for a day you'll be able to measure continuous levels of CAN errors and maybe you'll discover a failing transciever inside one of your nodes. <<

My next step was to buy one of these meters, and I was eye-balling the NetMeter. With the randomness of the fault, this may be a good avenue to follow.

>>If you could list the eight nodes on the network and tell us something about the overall length and installation that would help too. I've had plenty of "it's just forty-five feet on the drop, that's only twenty feet more than the spec." explanations of why wiring was intentionally done to violate the cable recommendations for DNet. The 20-foot drop rule is pretty hard and fast.<<

Will verify and report.

Steve :

>>What are the status LEDs on the nodes telling you?<<

They’re laughing at me, I tell ya, laughing… Seriously, there’s only one red flasher and one solid green when faulted. There should, of course, be three solid green…

>>I've seen a case where the DeviceNet scanner stopped talking to a PLC, yet continued to scan the network, updating the outputs with whatever values it happened to have. Believe me, that got the attention of the company that made the scanner! It's possible that Steeplechase has stopped communicating with the scanner card. <<

Interesting point! When we first dug into this, we thought the SST board was the culprit, particularly since the driver seemed to have been “dumped” from the computer’s registry. I’ve not seen any fault messages from SteepleChase to the effect of “DeviceNet Not Present”, but that may be because the VLC is still talking to the card. Or the fault may not exist.

>>Having said that, I agree with Ken that the most likely culprit is the wiring. The Synergetic DeviceNet Detective that Ken mentioned has more capabilities than the SST software. One additional thing you can do is to use the SST software to log all of the message traffic on the network to a file. It only allows approximately 1000 messages to be logged, but you can filter out normal traffic and maybe catch what happens just prior to crashing.<<

A particularly valuable tip. Will give that one a shot!

Thanks again!
 
Tim, I just want to add one thing as I'm not very DeviceNet literate yet, but learning rapidly. This sounds very familiar to an issue that myself & the controls electrician had sometime ago, posted as phantom inputs. This was a situation on a PLC5/40 using RIO where we had actual physical inputs in the program toggling at random. Talk about dangerous. We beat ourselves to death over this for the better part of a year, did all the monitoring & checking connections, found many potential problems, but none ever fixed the problem. Finally the electrician pulled every board out of the affected rack & closly examined them, ultimately it was discoverd that another board, not the board that had the phantom inputs, had apparently been wet at some point & there was corrosion on some of the components. Just thought I'd throw that in, hate to see anyone ever go through what we did.
 
Tim, Here's one that drove me crazy and the Hardware guru from Interlink. We had just installed a new network on a new line. the network would run for awhile and then fault out. I had the Interlink rep in to check the buss with what he called the troubleshooting wizard. Everything check fine on his end. I then had a guy in to check the panel grounds and also check for noise. We found some items and corrected, but it did not solve the problem. I also noticed that the proxs and photoeye's were acting funny so I decided to change them all from sinking NPN to sourcing PNP. Bingo, I havn't had a problem since I changed them. I have another line with Devicenet about fifteen feet away going the opposite direction. I have npn sensors on this one and never had a problem????? My biggest problem is with Devicenet software from Allen Bradley. Is it me or does it stink. I can't get the driver to load half the time. I have to close linx and reopen, and at times reboot the computer. Any idea's on this would help. I did just update my linx and RSnetworx and installed the hot fix for them. Any help would be appreciated.
 
Hi Tim. A few ideas.
Resistors at the extremity of the network on length, not just the main trunk line.
Resistors to be metal film, not carbon.
Power supply rise time to be under 250mS (I think this is the value). A slowly increasing supply can cause bus off.
The 24- of the supply earthed at only one point, and this point also has the shield earthed at it, with the shield not connected to earth anywhere else.
Check power supply or attached devices don't link the 24- to earth.

Thats all I can think of at present. Regards Alan Case
 
Last edited:

Similar Topics

Good day Forum Members I got a older Lincoln welder and hoping to make it work at our shop. Welder in question is the Lincoln Power Wave 455M...
Replies
4
Views
145
Hello Friends We have 10 Powerfocus 4000 with DeviceNet, We need to backup the configuration, the Powerfocus is detected but as unrecognized...
Replies
0
Views
67
Hello, I have a device with 68 words input. But one block on the Devicenet Scanner is only 61 words. I am trying to map this device to 2...
Replies
3
Views
462
I need to dust off RSNetworks DeviceNet next week for a customer, which means I need to get it loaded on my new laptop. I see my old laptop was...
Replies
3
Views
849
Something i ran into that has me stumped, hopefully someone can explain. I have powerflex 70's connected to a L7 cpu using a 1788-en2dn. The...
Replies
4
Views
1,238
Back
Top Bottom