OkiePC, I'm especially concerned that the DC and AC motor are of the same base speed. That way I know that the motors will develop the same torque levels. I am going to assume that this is true to make this go faster.
A 3hp 1750rpm motor develops 9 ft-lbs torque continuously and, with a 25amp drive will output at least 9 x 25/15 = 15 ft-lbs peak (that's assuming the 25amp drive rating is peak). The AC motor FLA seems too low for a 5hp but, assuming its ok, and using the 10,5amp AC drive rating as peak, the continuous output torque would be 15 ft-lbs and the peak would be 15 x 10.5/6.1 = 25.8 ft-lbs. Clearly, the motor is not the limiting factor in the new system.
Next, we look at the drive. Due to the V/Hz design and especially AB's tendency to use incredibly slow processors, the drive is not able to rapidly detect end of cycle and get the motor braking torque developed quickly. As a result, when the processor finally gets around to realizing that braking is needed, the braking level must be very high. That is where the High DC Bus faults are coming from even with the snubber brake package in place. I would expect that the decel distance is a little random as well for the same reason.
The 5% duty cycle rating on the snubber brake is simply saying that the resistor will reach its max temp limit if the brake is used at full capacity 5% of the time. Since your brake is not getting hot, I would not blame the duty cycle limit. It is important that the brake resistor be the lowest resistance permitted for the drive. Probably should check that.
My judgement here is that the drive is the culprit. I would change to a good fast sensorless vector drive and keep the same motor. I don't like saying this but, for your sake, AB is NOT one of those drives. Their sensorless vector products are at the low end of the scale performance-wise. If you must use AB, you will need to chose their flux vector drive and change the motor to a model with encoder provisions on the back side. Then add a 1024ppr encoder, an encoder option card in the drive (do you hear the cash register ringing?!!!), and add wiring from the drive to the motor for the encoder leads. You will have an excellent system with this package.
Or, pick a premium, fast, sensorless vector drive and that will do the job with the existing motor and no encoder. It's your choice.
Since I have been accused of brand favoritism here before, I will decline to offer any brand suggestions. If you want to talk about it privately, just contact me by email and I will oblige.
Having said all this, if my assumption about your motor choices is wrong, all bets are off and we will need to do this again!