Set your drive display to show output frequency. Then, run your system and start or stop in a typical scenario that will produce reverse rotation. Watch the drive frequency display and the forward/reverse indicator to see if the drive is telling the motor to roll backward.
If the drive is not telling the motor to roll back, then you can be sure the load on the motor is pushing it backward.
If the drive is telling the motor to roll back, you have a control issue with the drive. There are lots of possibilities but I would first try to determine if any forward/reverse inputs are switching unexpectedly. If not, I would look at speed loop tuning (only if the drive is operating sensorless or flux vector). The P and I gains can be way too low or way too high allowing this kind of behavior.
If none of this gives you any useful data, come back and we will help further.