Wonderware Intouch - Power outage and UploadTags Retriggered messages

[fourmutts86]

We had a power outage at a site and there are two machines that run their own instance of Wonderware Intouch 2017 SP3. With this power outage the AB PLC rebooted as well. Since then both machines in the SMC log are mentioning that UploadTags Retriggered every couple seconds due to the timer, however this never completes.

Is this something I need to do an offline update for as this application is not tied into a Galaxy server?

From doing some research some other forums mention to use the wwclient, but from the messages it seems like this is trying to upload tags not download. Anyways, please let me know if anyone is able to help, I can provide any logs needed.

 

[Bit_Bucket_07]

Sounds like the IO Server is attempting to upload the tag database from the PLC.

Are you sure that the HMIs and PLC are communicating with one another?

 

[fourmutts86]

That's what I'm wondering too, the other part of this is that we have an application in Galaxy that ties into the same PLC and doesn't have this issue after the power outage.

I can try a NMAP or Wireshark to see what is going on, though I can ping the device and everything else looks ok.

 

[5618]

Edit: you’re already past my first thought. Have you drilled down through the SMC, right-click, and reset ABCIP or further down to the controller?

Another edit: based on the logs, I expect ABCIP is running, but make sure it’s activated and green checkmarked. It could have previously been not set to auto start after boot.

One more edit added here: I don’t know what good it would do, but if you can make an online PLC edit (NOP or make a new tag) you could see if the program change shows up in the log to know that the server is actively monitoring the PLC. I’m not certain that would get logged if you don’t have auto synchronize selected.

 

[fourmutts86]

See attached, I reset the CIP connection and though the port is open the PLC it failed to initialize. I did also restart the machine a couple of times. The machine has been rebooted previously however the only difference is the PLC rebooted so I'm wondering if something is going on there.

 

[fourmutts86]

ABCIP - is all green too.

 

[5618]

Here’s one more thing to try: somewhere is a folder called CIPTagDB. Try deactivating the server, the moving the file(s) out of that folder to something temporary, and reactivating the server (you probably want auto start after boot). A new PLC tag DB cache should repopulate there. If that works, you can trash the previous cache you moved out.

In 2020 or 2023 it’s at \ProgramData\Wonderware\OI-Server\$OperationsIntegrationSupervisory$\OI.ABCIP\CIPTagDB

In 2017, it might be \ProgramFiles(x86)\Wonderware\OI-Server\OI-ABCIP\Bin\CIPTagDB

If it’s connected to the PLC, I expect a new file almost as soon as the server is activated.

Edit: don’t remove the CIPTagDB folder, just the files in it.

 

[fourmutts86]

Interesting just did that but I don’t think it will create a new file. I’ll give it a couple minutes.

 

[fourmutts86]

It’s definitely the PLC. I stood up a backup from 4 months ago and this was a known working copy with no changes to the PLC since.

 

[ynpmoose]

Not sure of your firmware version, but there is a known issue in firmware v32 where external HMI can no longer connect after a PLC power cycle. Quick fix is to re-download the PLC program into the PLC. The permanent fix is to upgrade to firmware v32.014 or higher.

Reference: https://softwaresupportsp.aveva.com/#/okmimarticle/docid/faq000021344

 

[ynpmoose]

Not sure of your firmware version, but there is a known issue in firmware v32 where external HMI can no longer connect after a PLC power cycle. Quick fix is to re-download the PLC program into the PLC. The permanent fix is to upgrade to firmware v32.014 or higher.

Reference: https://softwaresupportsp.aveva.com/#/okmimarticle/docid/faq000021344

Also an excerpt from the v32.014 release notes:

Editing of UDTs and AOIs Online Can Affect Online Behavior (00223654, 00230184, 880629, 1000207)

Controllers
First Known in Firmware Revision
Corrected in Firmware Revision
CompactLogix™ 5380
28.011
30.015, 31.013, 32.014, 33.011 and later
Compact GuardLogix® 5380 SIL 2
31.011
31.013, 32.016, 33.011 and later
Compact GuardLogix 5380 SIL 3
32.013
32.016, 33.011 and later
CompactLogix 5480
32.011
32.016, 33.011 and later
ControlLogix® 5580
28.011
30.015, 31.013, 32.014, 33.011 and later
GuardLogix 5580
31.011
31.013, 32.016, 33.011 and later


Workflows to potentially induce the anomaly

Create a user-defined data type (UDT), then, after the creation, modify the definition.
Take an existing UDT and modify it online.
Create a new UDT online. Take an existing UDT and add the new UDT as a member of the existing UDT.
Take an array of UDTs and change the array size.
Take the above workflows and handle them through a partial import online (PIO).
Delete UDTs online.
Effects on online behavior

Controller experiences a major non-recoverable fault after a power cycle.
Unable to upload the application into a new or existing ACD file.
Unable to go online with an existing application file.
After a controller power cycle, HMIs cannot reconnect to the controller and will require a download of the application.
Studio 5000 Logix Designer® application shuts down due to an exception.

 

[fourmutts86]

The firmware issue was the problem. They didn't upgrade it yet, but a redownload of the PLC program fixed the problem.

 

[ynpmoose]

The firmware issue was the problem. They didn't upgrade it yet, but a redownload of the PLC program fixed the problem.

Awesome. Glad I could help from the other side of Reading!