Floating point has several different word and byte order formats, sometimes known as Big Endian or Little Endian formats.
Floating point is a mantissa and an exponent, so if you read a floating point value (in Modbus a 32 bit floating point value is 2 Modbus registers of 16 bits each) and the word or byte order is not correct, the Modbus master's interpretation of the value is really bizarre, frequently a value with a very high exponent, like 22nd power. Sometimes the value is so bizarre that master might even consider it nan - not a number.
An extreme value might very well trigger an out-of-range error.
It would be worth exploring which format the slave is and which format the master is using to interpret the value.
Or just changing formats to see if the interpretation suddenly makes sense, which is why I like the generic PC Modbus masters like Modscan32 - it's really easy to change formats with the click of a button.