Synchronizing recipe data across machines

[kolyur]

I'd like to hear how others have handled this kind of situation. I have four identical machines, each with a Red Lion CR3000 15" HMI and Beckhoff CX5140 PLC. Communication between the HMI and PLC is via Modbus TCP. Each machine has a recipe interface on the HMI; the data is stored in retentive tags. The recipes are only handled on the HMI--the data is never transferred to the PLC--so the Beckhoff is basically irrelevant to this discussion. Each machine is on its own private LAN, but we have the hardware in place to allow Ethernet comms across machines.

The goal is to keep the recipe data synchronized across all four machines, with the following requirements:
- Recipes can be modified on any machine, and the others should be synced with the change in a reasonable amount of time.
- At any given time, one or more of the machines could be powered off/locked out. In this case, when the machine(s) are powered back on, they should receive whatever changes were made on the other machines.

This is further complicated by the format of the data. There are 50 available recipes, and each contains one 18 character string, one 10 character string, and four integer variables. The obvious choices for Red Lion communication are Modbus or OPC, neither of which have native string support. The strings will have to be broken into one or two characters per Modbus integer.

I've convinced myself that some type of external data storage device or server will be required... no single machine could be the master due to the downtime possibility. I have an old Red Lion Data Station lying around that I was thinking of using. Each machine could be set up to transmit a changed recipe to the "server" through a series of Modbus registers. But the tricky part would be making sure that each machine has the current data in the event of a shutdown. (We can assume that the server will have 100% uptime.) It's way too much data to transmit in one shot. Perhaps the machines would have to poll the server repeatedly on some schedule to ensure all the recipe entries are up-to-date.

This is getting much more complicated than I had anticipated. I'd appreciate any suggestions!

 

[Peter Nachtwey]

I would use a PC to keep a master database.
The Red Lion should be able to handle this with a little programming if it has the storage space.
If not a PC or even a Raspberry PI could do the job.



When the master is alerted to a change on any PLC then it is copied to the master data base and sent to the other PLCs.


What happens if 2 plcs change the same parameter to two different values at the same time?
There may need to be a read modify write mutual exclusion mechanism to keep two PLCs from accessing the same parameter at the same time.


The may be a need to keep track of some time stamps but the easy way would be to just download the whole database when the PLC come online. That isn't fast but it is KISS.

 

[CLanford]

I can see a scenario where you could use the PLC's as both the master recipe holder and the edited recipe holder that then updates the masters.

So you'd have two "sets" of recipes that would exist on each PLC that are retrieved from their respective HMI. One would be the "master" recipe and the other would be the "edited" or currently running recipe on each HMI.

On each PLC, the "master" recipe would be compared to the "edited" or active recipe (perhaps polled on an interval). If a change is detected between the edited and master recipe, then the edited recipe is copied over to the master.

This same process would happen between subsequent "master" recipes on the other PLC's. If a master recipe difference is detected, the master recipe from the unique PLC transfers to the others. When this happens, the individual PLC's then synchronize their "edited" and "master" recipes so the process doesn't over-write itself and get stuck in a loop. The trick to making this work would be that only 1 HMI could be editing at a time. So you would have to create an "edit" mode on the HMI's that would communicate this bit between all PLC's so that they would know to ignore this cross-check logic while one HMI is being updated. And you would probably use a falling edge trigger of this "edit" bit to begin the edit/master recipe synch and then the master/master synch to the other controllers. There would also have to be some handshaking between each PLC to ensure the correct new "master" is sent over to the other PLC's for synchronization and for master/master checks when one of the HMI's is powered off and back on.

 

[OkiePC]

I would extend the recipe structure to include the time and date edited and perhaps the username of the person who performed the edit.

Another possible means of exchanging data might be using the FTP transport mechanisms. (Make the DS+ a FTP server and enable Sync Manager on the machine level HMIs).

What about using one of the MQTT connectors?

I have not thought through any of those ideas. They're just things that first came to mind.

 

[GaryS]

I never like using one PLC as a server to hold data for others.
One failure will shut the whole system down.

If as you say all systems are identical with both the PLC and HMI running the exact same program.
If the systems are not on a common network then that’s the first thing I would do get them on a common network.
If the HMI I capable of storing the recipes then
Copy all of the HMI Programs into all 4 of the Red Lion HMI so any one can be used as a master.
Then configure them each to be able to select from the stores recipes and download to all 4 of the plc’s on the recipe download command.
When the recipe is downloaded to each plc I would set a flag in the each of the plc’s, indicating that the recipe has been changed that change would trigger each HMI to up load and update the recipe data file as needed
I would also create a download log file and update it with every download.
You may also log each batch as it is completed. This helps if produce problems are found later.
All you really need is one HMI and have one as a standby
If the Red Lion is not capable of handling that then you will need to replace it

 

[janner_10]

A cheap PC storing the recipes in SQL would be quick and easy to implement.