FactoryTalk View: Any workaround for 6-digit precision limit?

Referencing KB document QA9877:

"...FactoryTalk View SE Graphics supports single precision floating point values, and by default only provides for 6 significant digit accuracy. Note: digits displayed before the decimal point count as part of the total significant digits. Any digits displayed after the 6 significant digit are approximations only and should not be treated as significant..."

I am working on an ME V11 project and require 8 digit accuracy for several display and input fields. These are all PLC REALs with four digits before and four digits after the decimal point.

I can create two DINTs and HMI fields for each of these and then merge them in the PLC, but this seems unnecessarily cumbersome for both me and the machine operators.

In typical Rockwell fashion, the KB article lists single-precision/6-digit accuracy as the "default", but gives no indication of how to change the default or if this is even possible. I assume it is not. Has anyone found any other workaround for this limitation?

There is no way to achieve 8 digit accuracy with IEEE754 32 bit encoding. At best you can achieve 7 digits of precision (but not necessarily accuracy).

Can you switch to LREALs (64-bit floating point)? LREALS have 15+ digits of precision.
Those two numbers, 2321.066650390625 and 2321.06640625, represent two adjacent numbers that can be exactly expressed in REAL (32-bit) floating-point format, and no number between them can be exactly represented. I.e. this is analogous to how many integers there are between 1023 and 1024: the answer is none.


When you have a REAL 32-bit floating value, the next values that can be represented above and below it will be between one 8,388,608th and 16,777,215th fraction of the value away.

REALs only provide a little better than 6 digits precision. Some say 6.5. drbitboy is correct about using LREAL. Also, it is important to do any math operations in the correct order. As a General rule, do the math on similar sized numbers first to avoid round off errors.

If LREALs are not available then use DWORDs. Another trick is to split the DWORD into a 16 bit integer and a 16 bit fraction. Both the integer and fractional part will provide 5 digits of accuracy.

drbitboy said:

Can you switch to LREALs (64-bit floating point)?

Click to expand...

FTView ME does not (apparently) support LREALs. The tag browser does not see the my test variable. I get a line of asterisks when I enter it manually. I did a quick search, but found no documentation of this limitation.

Peter Nachtwey said:

Another trick is to split the DWORD into a 16 bit integer and a 16 bit fraction. Both the integer and fractional part will provide 5 digits of accuracy.

Click to expand...

This was my original "plan B" and it looks like what I'll have to resort to.

Thank you to everyone.