Synchronizing remote and local setpoint values

TLDR;
How to synchronize setpoint values between a local HMI and remote PLC connected by cellular modem, while preserving the most recently changed setpoint values as the active values, and prevent overwriting with old setpoints. Also, accounting for communication failures.


The Setup:
I am working with a system comprising the following parts:

1. A set of remote stations, each with a local PLC controlling various local processes. These remote stations have local HMI's which allow the local user to make changes to process setpoints.

2. A main office having a "master" PLC which is used to store and share data among the remote stations. Each remote station is assigned a block of registers in the Master for its setpoints and other process data.

3. A SCADA system at the main office which displays process data and setpoints retrieved from the master PLC, and allows changing of station setpoint values.

4. A cellular modem at each location providing communication between them.

The Goal:
I'm looking for algorithm suggestions to best ensure the setpoints at a remote station are synchronized with both the local HMI input and also the main office SCADA. We want both local and main office users to be able to change the setpoints, and whichever change was made *last* should be applied.

Limitations & Caveats:
- A synchronization takes place no more frequently than every 30-60 seconds between main office and a remote site.

- Synchronizations may fail due to cellular congestion, etc. When a sync fails, another 30-60 second delay is introduced until the next Comm cycle begins.

- The remote PLCs initiate all communications (both read and write).

- The Master PLC should act as a "data storage" location, and should need to do no processing or minimal processing on the data it receives.
It also updates a "communication successful" timer for each remote station, allowing us to track communication failures at the main office.

The Attempt To Solve:
I'm trying to design an algorithm that will ensure most recently changed setpoints are 1) applied to the remote station, and 2) visible to the SCADA users.

Currently, my logic is as follows (all steps performed on remote PLCs):

At each Communication Interval (30-60 seconds or longer if cellular delays or connection issues exist):

1. Read data from Master PLC and store in temporary location.

2. On successful read, compare data read from master with existing Active setpoints. If different, set flag indicating "master setpoints have changed".

3. Compare local HMI inputs with Active setpoints. If different, set flag that "local setpoints have changed".

4. If only remote setpoints have changed, copy remote setpoints to Active setpoints AND local HMI setpoints.

5. If only local setpoints have changed, copy Local HMI setpoints to Active setpoints.

6. If *both* local and remote setpoints have changed, copy Local setpoints to Active setpoints.

7. Send copy of Active setpoints back to Master, overwriting Master's copy of setpoints.


The Shortcomings:

1.
If step 7 failed (send copy of Active setpoints back to Master), then the next time we read from Master, we get the *old* setpoints, thus causing them to be different, and to be re-applied.

2.
Local setpoints only get applied on each successful Comm cycle (30-60 second delay or more) and are not applied instantaneously. It's not uncommon to have failed comm cycles, so this would impose long delays on the local user waiting for his setpoints to be applied.

However, if we applied local setpoints every time they changed, without waiting for a successful Comm cycle, then on the next successful Comm cycle, the master would be sending its previous copy of setpoints, which would then trigger a "master setpoints changed" flag, overwriting the local setpoints again with the master's old copy.

I thought about using timestamps, but that would prevent Master changes from taking effect at all, since the timestamp would always reflect what was set by the remote when the setpoints were sent to the master.


The Conclusion:
Sorry this post is so long - just trying to explain my thought process thus far.

I would greatly appreciate any suggestions for algorithms that would reconcile both copies of the setpoints while keeping the most recent changes active, and accounting for failures to read/write to the master.

Thanks for your time!

You might want to check into using the MQTT Protocol, this is a realtime data transfer between all of the "edge of Network" devices and the master(office). The MQTT protocol was developed so that remote "Edge of Network" devices that has limited bandwidth(cellular) could actually be connected together (as in one(1) large network) and still be able to provide realtime control of each device.

It sounds like you already have most of the hardware, but you may not be using the latest technology that is available for the process.

Just google MQTT, and Cirrus Link, Mr. Arlen Nipper the co-inventor of the MQTT Protocol has helped me with this technology and I'm sure he would help you also.

Thanks David. It appears my PLCs (AutomationDirect CLICK) don't offer support for this. They seem to only support Modbus/TCP, which is what we're using currently. I assume there would be some third-party hardware device installed to act as a translator between Modbus and MQTT?

Just by adding a small Maple System Hmi at each station, this will give you the ability to publish all of the tags with MQTT. A $299.00 per station investment to turn each into an "Edge of Network" device.

Please don't use Maple HMI. :'(

SP synchronization is one of those things that sound easy, but requires a tiny amount of thought to implement. Here are some pointers.

SP read address must be different to SP write address.
Your PLC should set SP write address to -999.9 or some other number which is never going to be used as an actual SP.
If it is ever not -999.9, update the actual SP.
In your case, take a timestamp, in the associated PLC.
When the 60sec sync happens, transmit the current SPs and timestamps.
Both PLCs then update their SP and timestamp value to the latest.

AustralIan said:

Please don't use Maple HMI. :'(

Click to expand...

You definitely don't have to use Maple System!

However, I still think using the MQTT Protocol would be the most efficient way to get your problem solved.

This is only my opinion!

Or you could just skip the steps 2 - 237 and just connect your SCADA via ModbusTCP driver directly to the Setpoint on the remote station..

But, that would be too simple I guess

AustralIan is right, it sounds simple but it's not. I am going to try and clarify a bit.

The key thing is to not even attempt to decide which SP change was last by checking current values. It causes way more problems than it solves.

Have the remote stations write back to the master always. This sychs the master as soon as the remote can.

If the master wants to update a remote, use a different tag call it "RemoteXUpdate" to write the new value to the remote. Once the remote accepts it, it the value will be written back to the master as usual.

An elegant solution for the remote to recognize a "new setpoint from master" is to have the remote change "RemoteXUpdate" to some standard value (Ian suggested -999.9) after it accepts the change.

The logic is simple and robust. The cost is an extra tag per setpoint.

mellis said:

AustralIan is right, it sounds simple but it's not. I am going to try and clarify a bit.

The key thing is to not even attempt to decide which SP change was last by checking current values. It causes way more problems than it solves.

Have the remote stations write back to the master always. This sychs the master as soon as the remote can.

If the master wants to update a remote, use a different tag call it "RemoteXUpdate" to write the new value to the remote. Once the remote accepts it, it the value will be written back to the master as usual.

An elegant solution for the remote to recognize a "new setpoint from master" is to have the remote change "RemoteXUpdate" to some standard value (Ian suggested -999.9) after it accepts the change.

The logic is simple and robust. The cost is an extra tag per setpoint.

Click to expand...

Ok, you explained some of the problems this aproach has, even solved a few..
I know all this and batteled with it when modifying elses systems.. But..

My question is - why not directly and one setpoint?
No complicated tricks and solutions when you don't have that problems in the first place..