I've finally had some time to fiddle around more with my motor a little more.
I think I have a slightly better understanding of what's going on and I'm back to the original question I proposed:
How does the Ultra 3000 know how far one step pulse is when it step/direction follower mode?
I tried adjusting the aux. encoder counts but that doesn't change anything. The aux. encoder settings must only be used if you put the ultra in aux. encoder follower mode (makes sense). So when i put it in step/dir follower mode, is it moving one step per built-in encoder count? The encoder is 260k counts per rev, so i feel like that's pretty high.
As far as I can tell, the only settings that effect the RPM are the counts out of Mach 3 and the gearing in U-W. When I max out the pulses that Mach 3 can achieve on my system (something like 35khz), I have to gear it up 1:540 in U-W to achieve the desired RPM. This is causing my vibrations due to the course stepping of the motor.
I feel like if there was a way to tell U-W that one step pulse = 1.8deg (for example) that I could then lower my electronic gearing WAY down and achieve a smoother rotation.
As an alternative test, I set the motor mode to preset velocity mode, added a few switches, and can run the motor at 8 different preset RPMS using the digital inputs of the switches. Everything runs smooth like that so my motor tuning seems fine.
I think at this point i need to ditch the step/dir input.
It would take some programming, but I could run my spindle pulse output (from the mach 3 PC) into my A-B PLC, let it count the pulses to determine what my desired RPM is, then convert that to an analog output and feed that into the the Ultra. That should allow finite control of the RPM but not with a pulsed input into the ultra.
I wouldn't be able to rigid tap, but i probably wouldn't have done that anyway.