The VFD's I've dealt with require a 3 wire set-up for the external pot, the analog input has to be "voltage". 0-10v is most common unless it is a forward/reverse application. The VFD has an on-board 10v power supply.
If you don't have the owner's manual, download it from the mfg's website.
Check the wiring against the owners manual.
See if you can change the "reference source" to "local" (onboard keypad for example) temporarily and observe your ability to control the frequency with the alternate reference source.
Check the "option" setting for the "type" of analog signal associated with the pot. Should most likely be 0-10v.
Try running it WITHOUT the pot connected.
Check the "actual" voltage of the on-board power supply.
Find out if this is a "new" problem, or has it "always been thus".
Some VFD's DO have an "offset" associated with the analog input signal.
Stationmaster