Data log time is reset in HMI

[gsanghvi]

We are using RED LION HMI Since las 15 years. To day we have found that My log file has data up to 22 Mar 2024 1:16. Then After new Log File is created with 1 Jan 1997. and data is logged in to that with Time different time stamp

 

[Elcan]

Somebody forwarded this to me today:

Good morning,

Today we are experiencing an issue affecting all Red Lion HMIs (screens) worldwide.
This problem impacts all of the brand's models. This is a timestamp bug, which explains the simultaneity of the failures.
The problem manifests itself as a black screen, which may indicate "SOFTWARE EXEPTION" or possibly "INVALID DATABASE".

In the majority of cases, a restart allows the screen to be put back into service. Sometimes multiple reboots may be necessary. Feel free to wait a few minutes between turning the power off and on again.
Once restarted, the internal timestamp will be 01/01/1997. Any “LOG” recordings and other events will therefore no longer be up to date. Without impacting the operation of the machine. It is recommended not to attempt to set the time and date.

Red Lion is currently working on fixing this issue. As soon as a solution is communicated to us by Red Lion, we will update it.

We regret this event over which we have no control, hoping that it will not impact you for too long.

We obviously remain reachable in case the screen does not restart.

Regards

The TCM Solutions technical team

 

[OkiePC]

Here's an email I just got back from Red Lion Tech support followed by my support request

Hello Paul -

We are aware of this issue and our development team is urgently working on this for resolution. We will update you on this matter as information becomes available.

We apologize for any inconvenience on this matter and thank you for your patience.

Thank you.


Lindsay Wenke, Technical Support Generalist I
Red Lion Controls | www.redlion.net
(877) 432-9908
The information contained herein, including attached files, is provided for informational purposes only. It is general information and may not reflect upon or address your situation. Red Lion makes no representation about specific knowledge of the customer's system or the specific performance of the system. Red Lion is not responsible for any damage to equipment or connected systems and disclaims any liability for actions you take or fail to take based on the information provided.

From me:
Mar 22, 2024, 9:45 AM EDT
I have a customer with 5 Graphite 15" HMIs that all suffered a "Software Exception 0F-2000-57AC-3707". They all recovered fine after touching the screen to continue. This happened on all five units within an hour or so of midnight last night. It was noted that one of them had an incorrect time on its alarm history this morning. The others appear to have the correct time. We think the night shift operators may have corrected the time using the controls we built into the application on those. I suspect this may be related to DST, since these units are still running an older version of Crimson 3.0 that probably has a different Daylight Savings Time calendar.
Last week, I had a similar occurrence from a customer with a CR3000 running on the latest version of Crimson 3.1 where at the moment of (present day) daylight savings time change, his HMI clock reverted to 1-1-1997 and he didn't catch it until 3 days later trying to review a flow trend. I don't think that unit triggered a software exception. It made for some funny looking trends with dashed lines running off the edge of the page.
I just wanted to report this and find out if there are any recommendations to prevent this going forward aside from, obviously, upgrading them to 3.2.

 

[OkiePC]

My customer that had this issue last week, and we thought was okay after we corrected the clock now has a new issue. The clock is fine...the logs, not so much...

 

[padees]

Yep.
See attached. I just got this in an email.

 

[Rube]

Delayed Y2K?...lol.

 

[Ken Roach]

Whoosh. From version 2.0 to version 3.2, so this must be something deep in the common codebase or the operating system. While it's easy to LOL about the non-events of Y2K, those of us who debugged and prevented it remember the nagging feeling that assumptions kill.

In the USA, daylight saving switched at 2 AM on March 10, 2024. We've got evidence ourselves from Paul that this could have happened as early as last week, not today.

It will be interesting to see how Red Lion handles this. I don't expect a deep dive into their OS and a public lessons learned discussion; they're notoriously tight-lipped about what's under the hood. Maybe the answer's going to be a set of one-time-only operating system upgrades for the whole product line, maybe they draw the line at Crimson 3.2 only. Or maybe they say "well, that's the only time that's ever going to happen. Don't go ... back in time" [Huey Lewis starts playing on the overhead].

I'm ordering a FlexEdge this week. No time functions in my application !

 

[atc222]

My experience has been that this happened today. Have one customer where 10 graphite displays went tits up.

Starting to hear from other customers who are just noticing the problem... Not a fun day for the Red Lion folks..

 

[Ken Roach]

Yeah, this weekend is gonna be a donuts-and-all-the-coffee affair there in York, PA.

 

[Ken Roach]

It looks like Red Lion has been sending out various worldwide e-mails, with varying degrees of detail.

One that was quoted to me suggests that this is basically Y2K but with an arbitrarily chosen epoch and increments.

>Crimson tracks time internally using a two methods: The first is to count the number of seconds since 01/01/1997, and the second is the count the number of 200ms ticks since that same epoch. The latter clock is used for data logging and other similar purposes where better than one second resolution is required. Both values are stored in 32-bit numbers, which works fine for the 1 second clock, but for the 200ms clock creates a wrap-around after slightly more than 27 years. To be precise, the 200ms clock will wrap-around at 01:17:39 on 03/22/24.

 

[drbitboy]

>>> start=datetime.datetime(1997,1,1)
>>> end=datetime.datetime(2024,3,22,1,16)
>>> d=end-start
>>> d.total_seconds()
858993360.0
>>> d.total_seconds()/(math.pi*1e7)
27.342607865423194
>>> math.log(d.total_seconds()*10)/math.log(2)
32.9999998333918
>>> math.log(d.total_seconds()*5)/math.log(2)
31.999999833391797

>>> datetime.datetime(1997,1,1)+datetime.timedelta(seconds=(1<<32)/5)
datetime.datetime(2024, 3, 22, 1, 17, 39, 200000)

 

[drbitboy]

Llis

 

[Ken Roach]

The function computes an integer representation of spacecraft time in milliseconds by taking the spacecraft clock represented in seconds as a 64-bit floating point number, multiplying it by 10, and casting it back into a 32-bit unsigned (i.e., non-negative) integer. (“Floating point-to-integer conversion” is necessary in order to convert a decimal number issued by the clock into a binary value recognized by the computer.) The computation does not guard against an overflow condition where spacecraft time in milliseconds can no longer be fully represented within 32 bits. This occurred when the spacecraft clock reached 429,496,729.6 seconds, which corresponded to the date “DOY 223” (2013). At this point in time or beyond, the computation results in a floating point error that ultimately triggers a processor reset.

 

[Ken Roach]

That article is sobering. Even literal rocket scientists can make this kind of mistake. And the writeup of the PATRIOT failure at Dhahran back in '91 is even more sobering.

 

[TheWaterboy]

Whats the odds this will be running in 27 years... send it