Siemens S7-1500, OB35 and Profinet question

[doomsword]

We've got very fast machine whose position controlled by interrupt OB35 set at 2ms. For position reading we use Temposonic connected to TM Posinput1 card connected to IM 155-6 PN/2 HF and update time is left automatic and is also automatically selected to be at 2ms by TIA.We have two cylinders moving this machine so everything is x2.

What I see is that it seems every 3 minutes or so I start to read the exactly same number from SSI (resolution is 0.05mm and it simply can't be the same number as machine is constantly moving) for two OB35 cycles, mostly for one cylinder while rarely it also happens for other cylinder. When this happens it goes for some time (~30s or so) and seems like 10-20% of time I'm getting two consecutive same readings. Then it stops until 3 minutes are passed and it repeats.
This of course messes with regulators and machine is performing worse as regulators are trying to fix this.

I started to think this problem comes from inputs not being refreshed fast enough to PLC. Inputs are read with postfix which should mean SW is trying to read from periphery.
I/O addresses organization block and process image are also set to automatic which now I'm not sure is correct.

I see I could set update time for IM 155-6 to 1ms manually but I'm not sure if this will help, maybe it would overload the Profinet, all devices of this PLC are on same Profinet? Would changing organization block for these inputs to OB35, leaving process image at auto and not using improve situation (if I update faster to CPU memory I don't need to access periphery, right?). Another thing is how PLC measures 2ms for OBs and Profinet, are these synced or not, and what I see might be coming from this (every 3 minutes they get out of phase and I start seeing this problem)?
Or I should look for something else?

 

[mk42]

Realtime is relative. My best guess is that the IO is sending the data approx every 2ms, and the PLC is running the OB approx every 2ms, and 99.99% of the time, that's fine, but occasionally maybe the PLC updates in 1.9ms and the IO takes 2.1ms, and that's JUST enough to miss the scan.


It is also possible that every so often a network packet goes missing, gets garbled, etc, and the PLC just doesn't receive that data. By default, the PLC won't yell about it until it misses 3 in a row. Random blips happen.

One option is to do what you suggested, update 2x as often as you need the data. If it's just a timing blip, that'll take care of it. If it's a network blip, it'll give you a little extra buffer. It is highly unlikely to overload the system if one device is a little faster than the others. It's TECHNICALLY possible that it could cause a problem, but only if the machine is right on the edge, and it usually takes a massive system for that to be an issue. The other IO won't see the messages from this ET200SP drop to the PLC, unless they pass through in a daisy chain.


Another option would be to detect the symptom. Store the last value, and if it equals the current value, do something different than usual. Not sure on your process, that could mean "assume everything is on track, kick the can till next time". I would assume that assuming a sudden halt is real is unlikely, especially if there's a known issue. That said, you might need some sanity checks built in for hard stops, if it thinks it should be stopping, etc. Again, dunno your process.



The Best Practice answer would be to use Profinet IRT with isochronous updates, like is often used with drives. It uses a special sync protocol so that your IO send the data right when the PLC wants to process it, and it uses a special Synchronous OB (OB61 i think?) to run the code that matches your IO. It requires some special network setups, where all the devices the ethernet frames pass through need to also support IRT. Most Siemens IO can pass IRT frames across their two port switch, but very few network switches do.



Any of these 3 solutions could make sense. They could be complementary, you might want to use all 3, depending.

 

[Peter Nachtwey]

I have seen this problem many times. The problem is that the SSI feedback and the PLC are not synchronous. They never will be. You need a hydraulic servo controller.

Why?

The interrupt of the PLC will NEVER be exactly every 2 ms as pointed out above. This happens because during the end or start of scan housekeeping times of the PLC scan, the interrupts are turned off. This can delay the interrupts.

This was a HUGE problem back in the 1990s when SSI was first coming out. I had some heated arguments with Temposoncs and Balluff about their SSI MDT rods. They are returning their positions asynchronously to our motion controller, so we saw exactly the same problem that doomsword sees. I called these SSI rods asynchronous SSI rods.
Not only were there double or missed readings but the readings were never at a deterministic intervals so speeds can be calculated if you don't know the time between readings.

Our solution is that our SSI signal had to be generated by a FPGA that wasn't affected by interrupts or anything that the software was doing. This meant our SSI clock was repeatable to the nanosecond. Next we had to convince Tempsonics and Balluff that their position feedback had to by synchronous to our SSI clock. This cured the problem but it took Temposonics and Balluff a while to make the changes. Now there are no more missed or double feedbacks. On top of that the position feedbacks were gather exactly every millisecond or whatever our clock was set at. This means the time between two readings is constant so I can compute velocities or accelerations. There is no sample jitter.

https://deltamotion.com/support/webhelp/rmcwin/Transducer_Interface_Modules/SSI/SSI_Overview.htm

Our current motion controllers uses a FPGA to gather ALL the inputs at fixed intervals in parallel. This is much better than a field bus solution like Ethercat or Rockwell's buses where there are delays because the data is gathered serially instead of in parallel. Even if there is a time stamp, one must wait for the last axes positions if you are synchronizing two or more axes.

 

[JesperMP]

I am not an expert, but as mk42 desrcibes in hos option 3, I believe that this is what 'isocronous realtime' Profinet (IRT) is supposed to solve (https://us.profinet.com/profinet-rt-vs-irt/).
I do not know if it requires that the IO devices are also IRT rated for it to avert the problems described - I am guessing they must be.

 

[doomsword]

Thanks all for ideas, I changed update time to 1ms for start and got interesting results (first I left at 2ms and update to OB35 process image but it was missing readings for both frequently but overall would get cancelled and not affect regulators ).


So with update time of 1ms and inputs/outputs set to be updated to OB35 process image (no peripheral access to remote) I get both cylinders having same readings in same manner like with 2ms, meaning when it starts executing sometimes it has 'old' readings. This probably means real update time is 2ms in this case, since process image updating is synced to OB35 execution (my guess).


Then I did the same but with peripheral access to remote, what I got is similar behavior like with 2ms, but 'old' reading is now from 1ms ago and not 2ms, so is not exactly the same but halfway between old and what new reading should be. Also frequency increased to 90s, but since readings are slightly better it affects the regulators less.


It's clear this is syncro issue, since cylinder that is more often with old reading is one that is processed first in OB38 cycle, and one processed after rarely has old reading.
I checked execution time and with peripheral access is 270us +-30us, and without is 195us +-30us.
If I can't change anything related to hardware and basic idea of software (no isochronous processing but synchronous interrupt) maybe changing OB35 interrupt time to say 2.1 or 2.2 ms (speeding up to 1.8 or 1.9 ms seems worse as reading would be 0.8-0.9 ms old) might ensure real-time reading every time. I would need to test machine behavior after such modification.
In case I need to use isochronous (I have no experience with setting such system) I would need to use special OBs and read/write inputs/outputs via special functions? For me issue here could be how I ensure I control both cylinders together and not separately if I'm waiting for input update?



Regarding post by Peter, SSI is read by a PLC card and PLC reads from this card that is located in a remote through Profinet. So how I understood it card does SSI at 125kHz, cylinders SSI electronics sends data at 125kHz rate, card catches data every pulse and then card updates PLC CPU every 1/2ms, meaning CPU doesn't see each pulse. Or setting card to 1/2 ms update also means card reads from SSI every 1/2 ms?


Thanks all for help!

 

[mk42]

JesperMP, yes, that's exactly what it is supposed to do. The IO card must support it in addition to the interface module (his ET200sp HF already does) and network topology. Not sure if that specific IO card does, but seems likely.


Doomsword,
Unless you specifically set the IO to update with OB35 as a process image partition (PIP), then it will update with OB1, regardless of where you actually use it. Setting up a PIP will more or less have the same effect as reading it with a in your cyclic interrupt; I don't usually bother with the PIP for just a couple tags.


Changing the cyclic interrupt update to 2.1ms means that one scan your PLC may have the freshest value (on 1ms IO update), but then the next call it's .1ms old, then the next call its .2ms old, and so on, to .9ms old. Then maybe it is .99ms old or arrived just in time for the next one, then .1ms old again. Doesn't seem like that would be ideal for control.



IRT can be tricky to configure, I don't know where any good guides are. Once you start the process, there are several changes that need to be made before the PLC will compile; some of the error messages only make sense in retrospect. At a minimum, you need to make sure that the network topology has only IRT capable devices between the PLC and relevant IO nodes, create a synchronous OB (OB61 i think) in the PLC, use the properties of the sync OB to set how often you want the IO to update, and how many multiples of that you want the OB to be called (every update, every 2 updates, every 4 updates, etc), set up a sync domain with the PLC as master and device as slave, put the IO card into Isochronous mode, have it update with the Sync OB. I'm sure i'm missing something.



Your first IRT setup might be a contact Siemens/local distributor type question.



Regarding SSI cards, yes, the card catches every pulse faster than PLC can think, and then passes value to PLC at the defined update interval, regardless how much the value changes between updates.

 

[Peter Nachtwey]

It's clear this is syncro issue,

YES! It is difficult to sync to items if they have separate clocks. Eventually they drift a bit.

Regarding post by Peter, SSI is read by a PLC card and PLC reads from this card that is located in a remote through Profinet. So how I understood it card does SSI at 125kHz, cylinders SSI electronics sends data at 125kHz rate,

The 125kHz clock is the rate the bits are clocked in, not the rate the position feedback is updated.

card catches data every pulse and then card updates PLC CPU every 1/2ms, meaning CPU doesn't see each pulse.

That is right. The card must clock in the position one bit at a time and then it can send the position to the PLC.

Or setting card to 1/2 ms update also means card reads from SSI every 1/2 ms?

Reading from the card every 1/2 ms won't help. You will get the same data a lot of the time.

You need to know how a MDT rod works. Basically the MDT rod sends a pulse ( interrogation pulse ) down a wave guide and when the pulse reaches the magnet mounted on the end of the piston it causes a twist in the wave guide that is detected.

The time delay between when the pulse is send and when the twist is detected is proportional to the distance the pulse must travel. This is kind of like a radar or sonar. The speed or gradient of the rod is about 1800 m/s or 1.8 m/ms. This means it makes no sense to update any faster than every millisecond of the piston will extend 2 mm because it is necessary to wait for the previous pulse to be returned. Therefore it makes no sens to update every 0.5 ms if the piston will extend past 0.9 mm.
All this sending and timing pulses is hidden from the user but it is important to understand what is happening inside the MDT rod head.

So how long is your stroke?

Now here is the trick. The MDT rod needs to synchronize with the motion controller so the positions are returned at exactly the same intervals. I had to tell both Balluff and Temposonics why this must be done and how to do it otherwise the readings would be taken at uneven intervals so calculate velocities and accelerations would be impossible.

On the moton controller or SSI module side, the SSI clock must start at consistent intervals so the SSI MDT rod can synchronize to the motion controller.

 

[JesperMP]

Here are the specs for a Temposonic rod that supports IRT:
http://www.temposonics.com/portals/...ta_Sheet_R-SeriesV_RH5_Profinet_551962_EN.pdf

 

[doomsword]

@mk42, yeah, setting it to 2.1ms would not resolve the issue, it would make it worse actually when PLC is reading properly, what I wanted is 2ms interrupt that waits 0.1ms to execute logic.
@Peter_Nachtwey, this thing about Temposonic working as as sonar pinging magnetic ring I knew, but I never thought what it actually means, in sense how long it takes to get a position reading. Same for SSI, I presumed 125kHz means I get full position update every 1/125kHz time, not that is data transfer speed.
So regarding stroke, is variable and is normally 8-16 mm (could go above but normally is not necessary), machine can do about 30mm stroke, but those are just low and high mechanical limitations. This machine is following practically what is a sine trajectory, going up and down with different stroke and frequency. Highest speed of change should be ~150-200 mm/s (machine should be able to do this, but usually is slower),so about ~0.4mm/2ms max (normal high is about ~0.25mm/2ms). So should be within limits you wrote.
Temposonic has reading range of 50mm.
And after all this I checked what Scalance switches are used here, we have B Conformance Class, that don't support Profinet IRT (they are C Conformance Class), my remote is connected to CPU through two Scalance switches of B class.
So basically if I want this to work as how it should I just need to change everything.

 

[Peter Nachtwey]

So basically if I want this to work as how it should I just need to change everything.

If I was consulted before you got started, I could have steered you away from any design mistakes.

What is the application? What are you really trying to do?
The speeds don't seem unreasonable.
I would have used SSI rods with higher resolution. 0.001 mm is easy to obtain. This provides precise position, velocity and acceleration control.

I would use this for a motion controller.
https://deltamotion.com/products/motion/rmc70/
It will communicate with the PLC over Profinet. However, the RMC controller will handle all the closed loop control. The RMC has the hardware to interace with the SSI MDT rods synchronously. When Temposonics want to show of how good their MDT rods are they use our controllers.
https://youtu.be/8Djvq7cDv4U
Synchronizing two hydraulic actuators is easy.
Even if the opposing forces vary
https://deltamotion.com/peter/Videos/SyncPosForce.mp4

You hit the jack pot. I am a member of the International Fluid Power Society's Hall of Fame for my work in hydraulic servo control.

 

[mk42]

@mk42, yeah, setting it to 2.1ms would not resolve the issue, it would make it worse actually when PLC is reading properly, what I wanted is 2ms interrupt that waits 0.1ms to execute logic.
@Peter_Nachtwey, this thing about Temposonic working as as sonar pinging magnetic ring I knew, but I never thought what it actually means, in sense how long it takes to get a position reading. Same for SSI, I presumed 125kHz means I get full position update every 1/125kHz time, not that is data transfer speed.
So regarding stroke, is variable and is normally 8-16 mm (could go above but normally is not necessary), machine can do about 30mm stroke, but those are just low and high mechanical limitations. This machine is following practically what is a sine trajectory, going up and down with different stroke and frequency. Highest speed of change should be ~150-200 mm/s (machine should be able to do this, but usually is slower),so about ~0.4mm/2ms max (normal high is about ~0.25mm/2ms). So should be within limits you wrote.
Temposonic has reading range of 50mm.
And after all this I checked what Scalance switches are used here, we have B Conformance Class, that don't support Profinet IRT (they are C Conformance Class), my remote is connected to CPU through two Scalance switches of B class.
So basically if I want this to work as how it should I just need to change everything.

There are very few conformance C class switches. You have to buy special IRT capable ones that only go up to 4 ports; you only want them if you KNOW you're using IRT. Usually your IRT is run as a daisy chain from one port on the PLC (no IRT switch needed), and then all the rest of the IO goes out the other side with whatever normal class B switches.

 

[doomsword]

If I was consulted before you got started, I could have steered you away from any design mistakes.

What is the application? What are you really trying to do?
The speeds don't seem unreasonable.
I would have used SSI rods with higher resolution. 0.001 mm is easy to obtain. This provides precise position, velocity and acceleration control.

I would use this for a motion controller.
https://deltamotion.com/products/motion/rmc70/
It will communicate with the PLC over Profinet. However, the RMC controller will handle all the closed loop control. The RMC has the hardware to interace with the SSI MDT rods synchronously. When Temposonics want to show of how good their MDT rods are they use our controllers.
https://youtu.be/8Djvq7cDv4U
Synchronizing two hydraulic actuators is easy.
Even if the opposing forces vary
https://deltamotion.com/peter/Videos/SyncPosForce.mp4

You hit the jack pot. I am a member of the International Fluid Power Society's Hall of Fame for my work in hydraulic servo control.

It is basically a table with two cylinders moving it up-down that needs to follow sine reference with variable amplitude and frequency (this doesn't change crazily momentarily, but goes from 8-16mm stroke and 1.5-4.5 Hz to do a stroke) with additional requirement to control misalignment between two cylinders (should not tilt the table, machine is good when is <0.1mm). Beside that there can't be phase delay or error in amplitude in following reference so mostly is feedforward control from position reference with PID corrections.

Temposonic resolution is 0.005mm, I missed one zero, so is precise enough for requirements here.

I don't think this is crazy demanding application, just seems automation necessary to do it was underestimated, and now I'm seeing funny things.
This machine is always controlled like this, but normally it has a dedicated PLC just for it, in my case everything is on one PLC, so I guess that was hiding the problem I'm seeing.


Anyways I feel solution will be that I need to find a 'trick' to make it 'work', machine is operational but when its' performance is evaluated this reading problem makes it look worse than what probably is.
I'm thinking about making position estimation logic when it fails to read or force interrupt logic to execute some garbage logic to waste time until i get position update.



I think you were helping me when I first joined this forum, I was working as a student for a company and they gave me some proof-of-concept task that guy working on it before practically gave up on.

 

[Peter Nachtwey]

It is basically a table with two cylinders moving it up-down that needs to follow sine reference with variable amplitude and frequency (this doesn't change crazily momentarily, but goes from 8-16mm stroke and 1.5-4.5 Hz to do a stroke) with additional requirement to control misalignment between two cylinders (should not tilt the table, machine is good when is <0.1mm). Beside that there can't be phase delay or error in amplitude in following reference so mostly is feedforward control from position reference with PID corrections.

Are you using only a position reference? What about velocity and acceleration?

Temposonic resolution is 0.005mm, I missed one zero, so is precise enough for requirements here.

The resolution is for computing velocities and accelerations too. If your update rate is 1 ms, the minimum velocity you can detect is 5mm/s. I am retired now an don't go to trade shows anymore. That is where I would meet the Temposonics and Balluff guys and talk about tecnhical things. I told them they should provide all the resolution they can so the motion controller can compute accurate actual positions, velocities AND accelerations. These are necessary for the derivative and second derivative gains.

Also, whenever the actuators are traveling less than 5 mm/s you may not see a change in position from one millisecond to the next.

I don't think this is crazy demanding application, just seems automation necessary to do it was underestimated, and now I'm seeing funny things.

Yes! This should be easy.

This machine is always controlled like this, but normally it has a dedicated PLC just for it, in my case everything is on one PLC, so I guess that was hiding the problem I'm seeing.

Or they had he problem and couldn't see it.

I'm thinking about making position estimation logic when it fails to read or force interrupt logic to execute some garbage logic to waste time until i get position update.

The old cry "Fix it in software!"
That will work if you can model the feedback by using a Luenberger observer and then model the system.

I think you were helping me when I first joined this forum, I was working as a student for a company and they gave me some proof-of-concept task that guy working on it before practically gave up on.

There are all sorts of things that can screw up a project. When I was working I would usually get asked questions about an application before they got started to see if it is feasible. I could ask a few questions, feed that info into my Mathcad work sheets and python programs and say yay or nay or what must change to make it work.

For instance on your project I would like to know what is the natural frequency of your machine. It should be at least 4 times the 4.5 Hz ( the frequency of acceleration ). I would also ask about the valves to make sure they are fast enough and linear. Also, I would like to know about the accumulator.

Thee are a lot of magazine articles I wrote over 20 years that cover many topics in hydraulic servo control design. Most are on the magazine "Power and Motion" now. We have a forum for motion control at
https://forum.deltamotion.com
or subforum
https://forum.deltamotion.com/c/slug/motion-system-design/14
It is pretty dead because most people just call before they start now and they don't want to give away what they are doing to the competition.

Here is a good thread about calculating the speed of a cylinder where forum member Norm Dziedzic and I do some calculations and examples.
https://forum.deltamotion.com/t/the-vccm-equation/378

 

[BryanG]

Probably talking out of my backside here, best be honest upfront Fortunately I have had a life time of looking foolish, so one more won't hurt. I feel like a kid talking to grown-ups

You think the problem is that there are two 2ms timers and they go out of sync. Can't you do it with one 2ms timer. Call OB35 on receipt of the message from the Temposonic.

 

[doomsword]

...

We are using velocity, not just position for regulation.


Regarding 'Fix it in software!', I had a talk with some (also including automation) people from my company and they're not convinced about what I'm saying is a problem, this machine was run at slower intervals (slower interrupts) so they're asking why it wasn't an issue for slower control. I said is synchronization issue but they don't think it really is (or don't understand what i really mean) since 'disturbance' I'm seeing lasts for 30s or so (for me it's appearing like this because cyclical interrupt and input update times are almost constant value numbers that are very close in value (time) to each other, and when you juggle them thousands of times they will mostly sync but you will have a period they desync then resync again, which is not random, what I saw when I doubled update and disturbance got 'faster'), so they think it's mostly regulators (which is at least partially true, when we resync i see regulators unwinding, their gains are then causing disturbance). They expect it should be more random behavior, wrong readings can't go for 30s when we are reading every 2ms, something in this line of thought.

So, they proposed few tests for me (mostly playing with regulators (such as increasing feedforward and decreasing proportional to mitigate effect, 'freezing' some of regulators outputs during bad reading etc.) to try to solve or improve this situation.


EDIT: I got recordings from another machine in other place from colleague, it has the same problem, I think they didn't notice it because update time was faster than execution time, so instead seeing position frozen, it was from one update before. Also it was more frequent, probably because of S7-400 PLC and Profibus, but I can't put my hand in fire I'm totally right for this machine as I wasn't there.

Probably talking out of my backside here, best be honest upfront Fortunately I have had a life time of looking foolish, so one more won't hurt. I feel like a kid talking to grown-ups

You think the problem is that there are two 2ms timers and they go out of sync. Can't you do it with one 2ms timer. Call OB35 on receipt of the message from the Temposonic.

I wish I could do this exactly like you wrote as it is what I'm trying to achieve.