Like JBC said, just taking the baffles out will definately make the exhaust louder unless V&H (or Big City Thunder, etc.)makes a set of quiet baffles to replace the existing baffles and that is your plan. I could not find any quiet baffles listed on the website but may not have been looking up the right slips ons either?
If removal is neccessary you could try letting the bike warm up with the bolts removed from the baffles and see if they will move then. Or you could use a slide hammer set-up (carefully of course) to convince the baffles to come out. There should be a plate inside the baffle where the set screw is that you can get a slide hammer attachment on to use as in th pics below.