You need to increase the punch control % to reduce the punch
I would inspect the front endbell to make sure there is no slop with a new bearing fitted, and if its okay try to find a better quality bearing; castle only uses abec 1 rated items and they are fine for 90% of people; just us 10% of nutters that manage to kill them.
Castle really should use better bearings though- if they buy in bulk then the cost increase would be minimal...