Wonderware Woes

A customer of ours has a Wonderware system running on a Dell T1700 small form factor PC.

It is connected through a 4 port switch to an RO system with a Controllogix CPU.

It experiences occasional (and reportedly worsening) connectivity losses. When I first troubleshot it, I found the office grade hub between the SCADA PC and the Logix CPU could be power cycled and resolve the problem. I replaced the dusty old hub with a new Stride switch and replaced the cable to the PC with a new one (due to rodent damage to the outer jacket).

The problem seemed to clear up but I heard Friday it is still acting up and has been a problem. They are rebooting the PC "8 times per day". That may be an exaggeration but the problem is severe enough I need to take the next step. I am less than a novice with Wonderware. The last WW system I had to work on had a parallel port key, ran Windows 95, and I made the changes so quick and easily I barely remember it, just that it was a piece of cake.

This system was installed in 2013 and has a runtime license on the PC. I wrote down version 11.0.0400

I am pretty sure this was installed as a quick and dirty upgrade to an older WW package. I am also pretty sure that the Logix5k CPU replaced a PLC-5. They are still using RIO Flex I/O modules in remote racks and the addressing is all "conversion utility looking".

I am pretty sure that there is SLC/PLC mapping going on to maintain the old tags in the WW PC. I have not checked into the A/B drivers and I am not sure exactly how.

This customer bought a Development license, RSLogix5k and an intrinsically safe Toughbook about a year ago. I think the version on their laptop is slightly newer.

So, the Toughbook has the development key, the PC has the runtime key.

When I installed the little Stride 4 port unmanaged switch, I had some spare time and tried to network the laptop to the PC and share the license that way. I was unsure how to transfer the license from the laptop to the PC or what would be the best method to poke around in the WW app without risking some failure that I could get stuck on for hours or days.

I was able to set up the license network feature, set compatible IP addresses, but I still got an error when I tried to open Windowmaker. I did get to the point the PC could "see" the license on the Laptop. I even went as far as to "hide" the Runtime license on the PC briefly and that did not work at all for me either. The license feature appears similar to Rockwell concurrent licensing scheme with which I am intimate.

This site is a water treatment building for a fertilizer plant with tight restrictions on electric tools, phones, cameras, etc. Internet access is a bit of a pain from there so I wanted to get the advice of experts before I go up in the a.m. I may be able to post more details in this thread from the job site when I have the stuff in front of me to gather info.

I do not think that the WW app went south all by itself. I have a spare e-net cable between the WW PC and the Logix CPU I have verified does function. I do not have a media checker. I don't even have ethernet termination tools yet, but may have to get some in the morning just to show up and be able to do something for them.

When I witnessed the problem, I could still ping the PLC and I could see packets accumulate in the Windows IP properties monitor but they were not accumulating as quickly as normal. At one point in my checking, I too had to reboot the PC since cycling the new Stride switch did not clear up the problem. This was right after I put it in (the new switch) so I did not sweat that too much.

I suspect the motherboard e-net port on the cheap PC is going south or I have a cable issue, so to have real parts in hand I thought I would just grab a new PCIe low profile e-net card at Best Buy today. No go. They (nor Office Depot) stock any of those internal gadgets any more. Only had video cards and externals in stock. So I grabbed a USB to Ethernet adapter, which I can use someday somewhere anyhow. With that, though, I will need to move a license to be able to switch the WW driver to that new port, right?

This is probably too much info, but better than not enough, right?

So how would you proceed?

TIA,
Paul

What is the WW app using for an IO Server?

If it's using Wonderware's DASABCIP OPC server, you can try opening the System Management Console (Start->All Programs->Wonderware->System Management Console (SMC)). There should be an internal 'Log Viewer' that might reveal something.

Also, check if DASABCIP is configured to 'Run as a Service'. DASABCIP should appear in the SMC console under something like 'DAServer Manager->Default Group->Local->ArchestrA.DASABCIP.xx)

If it's configured to run as a service, you can check the windows event log to see if the service is crashing. This would break connectivity to the PLC. Windows will let you configure a service to automatically restart-upon-crash (up to 3 times, I believe). This wouldn't solve the problem but could perhaps delay it (or expose a crashing service as the problem).

2. Maybe consider installing Wireshark on the HMI machine. This might be difficult depending upon your client's security practices. It's sometimes possible to examine a packet capture to see what happens when communications are disconnected - i.e. you might see packets sent by the HMI but unacknowledged from the PLC, or perhaps incoming packets from the PLC unacknowledged by the HMI. You can leave a packet capture running in the background and have the client note the approximate time of failure.

-Trevor

Thanks, Trevor. I can put Wireshark on the PC. It stands alone and I have Wireshark install files on a thumb drive which is allowed in the facility without a special permit.

As for switching the ethernet device being used by the DASABICP driver (assuming I try out the USB to Ethernet device). I could try that as a method to rule out the onboard ethernet port on the PC. Can I do that with only a runtime license on the PC?

Also, any info you have about temporarily getting the customer's Development license moved from their laptop to the PC and back again would be appreciated.

Unfortunately I'm not very familiar with recent Wonderware licensing schemes.

In prior versions, the license was just a text file (*.lic) that you could move onto the target machine. If I remember correctly, the .lic file lived in a common folder (something like C:\Program Files\Common Files\ArchestrA\License, or maybe ...\InTouch\...).

I think that you can access the System Management Console without a Development license but I'm not positive. You can definitely check the list of windows services though as this is independent of Wondwerware (besides the need to initially configure the IO server to 'Run as Service'.

-Trevor

tvey said:

Unfortunately I'm not very familiar with recent Wonderware licensing schemes.

In prior versions, the license was just a text file (*.lic) that you could move onto the target machine. If I remember correctly, the .lic file lived in a common folder (something like C:\Program Files\Common Files\ArchestrA\License, or maybe ...\InTouch\...).

I think that you can access the System Management Console without a Development license but I'm not positive. You can definitely check the list of windows services though as this is independent of Wondwerware (besides the need to initially configure the IO server to 'Run as Service'.

-Trevor

Click to expand...

The licensing hasn't changed, yet... Recent release has a new license manager/viewer, but the core function is still the same.
So, you should be able to move the license back & forth very easily.
The SMC can be accessed without a license. You should also be able to re-configure the DASABCIP IO Server to use your USB-Ethernet adaptor from the SMC (it looks for what IP address to use...)

Another thing to look at for comms issues is the ControlLogix. Check the overhead Time slice. This has been a common cause on other sites, is not unusual for it to be set to 35% or more.

As others have said, the licensing is easy to transfer (for temporary testing purposes or rebuilding a new PC of course). You can find the location of the licence files by opening the Licence Manager from the Wonderware program folder and looking at the location of the licence file. Then just copy and import it into your machine. It is version specific though, so if you're running 10.5 vs 10.1 or whatever, it won't work and is not backward compatible as far as i remember, the licence needs to match the installed software.

the System Management Console (SMC) has a ton of useful information. But I would start by checking the Intouch Access Name definitions to make sure it's looking for IO data where you think it is. Once you get Intouch into developement mode, open up the tagname dictionary and look at a tag, click access names and then take a look at the configuration for that access name. This will tell you which application is providing the data for the tags.

If it is the DASABCIP driver, then that is configured through SMC as described by others. I have only used it once with a compactlogix PLC but i don't remember there being much to change.

Also while you're in that tag database, take a look at the item name for a tag. If it's symbolic (i.e. N9:0 or whatever), then your assumption about SLC mapping inside the PLC is probably correct. I would recommend to the client that some work be done to move this to using controller tags, makes things a lot easier to modify down the track!

If they are happy to go down that route, there are two handy utilities called DBLoad and DBDump which reside in the Wonderware program folder (you need to find them by browsing to C:\Program files\Wonderware... whatever). These allow you to export all the tags to a CSV file, modify in Excel and then import back in either all tags or just a subset, with varying modes of import (over-write, update, etc).

And of course before attempting any of this... backup the Intouch application by making a copy of the whole application folder!

Great info! Thanks very much.

No SLC/PLC Mapping in ACD file. There are many tags that are Logix native tag names, but most are the names that were created by the PLC-5 to Logix5k conversion utility.

Getting an error in the SMC log file:

The License File 'C:\Program Files (x86)\Common Files\ArchestrA\License\wwsuite.lic' was located, but the Feature Line 'Rockwell_DASpak' could not be found inside of it.

Click to expand...

EDIT: This is just one of a string of errors that pops up every 30 seconds. Screenshot of them to be added. I am not sure if this feature used. The system has not hiccupped since I have been onsite (2 hours).

I did find that the license manager still had a reference to the Laptop under network licenses from when I tried that about a month ago. I deleted that reference.

There are several backup files of the licenses. There is an Archestra license and an Intouch license. Some of the backed up files are simple file copies it appears (in New Folder and New Folder (2) under the main license path). Some of them have the .bak extension. Few of them are identical. I searched through a couple of them for a reference to the word "Rockwell" and "DASpak" and did not find either string.

If I want to "try out" one of the other licenses, do I simply copy it to the correct path or must I use the manager to manage it?

I have not examined the details of their laptop license yet.

I did take a wireshark trace and save it, but I don't think I should post it. I will try to take another trace and save it while the system is "acting up" if it ever does while I am here.

A watched pot...

EDIT: Also, I tried to open some of the oldest historical log files, but the file extension does not match what appears to be the only one accepted by SMC. It wants "*.aaLGX", but in the old saved logs there are two extensions "*.aaLDX" and "*.aaLOG". What to do here?

Look like it could be your license file. Open up both the Archestra.lic and wwsuite.lic files with notepad and see if you see something like "Intouch 2014 Runtime/Development xxx tag with I/O" at the top. The "with I/O" is needed if you are going to use Wonderware's IO server like the DASABCIP. If you have a development license, the "with I/O" may be omitted.

My partner who is here for mechanical work and was here all day yesterday said it froze twice. It seemed to happen when "no one touched the mouse for 30 minutes to an hour". I looked at the power options for the PC and found all set to good industrial values except PCI Express -> Link State Power management was in "Moderate Power Savings" mode. I doubt this is my issue since there is constant activity on the ethernet port. I am in the middle of "not touching it for a half hour" and once that proves or disproves to be relevant, I will try my darndest to make it freeze up so I can at least have something broken in front of me before I start fixin' it.

I also wonder whether wonderware is waiting for some idle period to "yank" the connection due to this apparent license issue. I'll keep an eye out here too for more advice and I really appreciate all the help so far. I need to take a WW class. We have at least a handful of customers using it. I also need to work 70 hours a week for the next six months to stay caught up at work...

Any controls guys strong in A/B and SCADA for hire in the Norman OK area?

Some time after 11 am it froze. By froze, I mean that values stopped changing. Some of the control buttons still appeared to work, but that could simply mean that the visualization part of the software is not "hand-shaked" with the PLC.

At exactly 11:00:00 I had some new errors appear in the SMC console log viewer. Component: WWSCRIPT, Message: C:\Ufdata\U10052015.csv -- No such file directory. That looks like some sort of logging attempt failed and my be unrelated.

So, I followed your advice about looking for the I/O reference in the lic file, and I noticed that the wwsuite license said something about Demo in the title. While I was viewing it, a window popped up warning me that the Demo license had expired. I had not seen that before, and no one had reported that to me. So, I found a wwsuite.bak file that looked juicy and renamed it after moving the previously active wwsuite.lic file to a subfolder.

After doing that, I shutdown the license server and restarted it. Opening the log file, now I see "Valid license is available". That was 10 minutes ago and there are no new log entries. I never stopped Windowviewer but my data "came back to life".

So, I think the license thing was the main problem. I am n ot 100% sure we don't have other less frequent issues, but we are about to leave it untouched again during lunch break and see what happens.

Also, a point of note. I ran a Wireshark trace while it was "frozen" and the traffic was minimal. maybe 2% as many items appeared as when data is flowing normally. I am not a wireshark nor EIP expert but if need be we can pick those traces apart later. When it was "seized up" the packets looked like they were still getting across successfully, there were just a tiny number of them compared to now with things working well.

EDIT: The missing csv file came back again at 12:00. I don't think it is a show stopper, but I do think I can fix it before we run off to eat.

Okay, time for a cheesburger...

Thanks again,
Paul

OkiePC said:

Demo license had expired.
Paul

Click to expand...

Yeah that will do it. Maybe they've changed the way they stopped an application from running on an invalid license but on older versions I thought they would let you run until you shutdown the Windowviewer or something of that nature. Maybe not. Well I think you found the problem. The SMC is a good troubleshooting tool. You deserve some fries with that cheeseburger!

DaveW said:

Yeah that will do it. Maybe they've changed the way they stopped an application from running on an invalid license but on older versions I thought they would let you run until you shutdown the Windowviewer or something of that nature. Maybe not. Well I think you found the problem. The SMC is a good troubleshooting tool. You deserve some fries with that cheeseburger!

Click to expand...

I owe you a beer too! I am not putting a bow on it until it runs for several days. Looking back through the available log files, that string of errors for the Rockwell I/O server has been occurring since May of 2014. That was a relief to find since it reassured me that I did not break something last month when I was trying to learn by trial. The error for the data logging traces back as far as early 2013. I think I fixed that too, simply by creating the missing folder. There were no new entries in the SMC log file from 1 p.m. until I left about 4:30 p.m.

There were about 4 different copies of each license file and three more .bak files. I picked one that seemed to have the right text for the errors they were having and turned it back into a .lic file (after making yet another spot for the original)...

Yes, the WW upgrade this customer paid for was quick and dirty. Earlier I said quick and simple trying to be polite, but now, I can officially call it dirty. They left several things not cleaned up and according to what I was told, the entries in the SMC log file date back to even before their latest upgrade.

Thanks again for all the help. Now I have at least a smidgen of confidence drilling into the next WW app I encounter which might be at a different customer tomorrow morning. We have a big project here and have proposed Ignition as the replacement SCADA package.

So, I get to learn another HMI/SCADA package but this one I get to build from scratch.

Bottom line for me now: WW Woes = Check the SMC log files first and go from there.

What probably happened is that someone borrowed a 30 days trial license to edit the program on that pc and but forgot to change the license file back to the original file. With all the freezes and reboots I'm surprise no one notice the warnings about demo license expired for over a year.

DaveW said:

What probably happened is that someone borrowed a 30 days trial license to edit the program on that pc and but forgot to change the license file back to the original file. With all the freezes and reboots I'm surprise no one notice the warnings about demo license expired for over a year.

Click to expand...

That is a good possibility about the license issues.

As for no one seeing the warning, if they did, it was not folks who talked to us. The PC is in a corner control room of the water treatment building that rarely gets attention. They visit the water building only periodically and only go in the control room to reset faults if something is wrong. I am not sure, but I suspect that the warning that I saw may vanish if nobody Ok's it within a few minutes.