A typical potentiometer doesn't have the power rating needed - they are usually rated for only about 1/8 watt or so. Let's say a fan is rated 12v and draws 100mA. If you set the pot so the fan sees half the voltage, that means the pot is dropping the other 6v, and therefore has to dissipate 0.6w (100mA x 6v). That of course assumes that the fan will still draw 100mA at the reduced voltage, which it probably won't, but even so, the pot isn't strong enough. You'd have to use a power rheostat, and those can get more pricey, are larger (to dissipate the heat), and you'd have to order them from someplace like Digikey, Mouser, etc.
Yes, there are switches with 4 or more "throws"; those are typically rotary switches. RadioShack has a
6 position rotary switch, but is only rated for 0.3A. This could work if you are positive each fan will pull less than that.
If this is done right, you could mount your switches to a blank 5.25" expansion bay plate so they are all within easy reach.