Absolutely a matter of whatever makes you feel best and who cares what everyone else thinks. I wear either my leather jacket or an older HD mesh armored jacket 95% of the time. Kevlar (Draggin Jeans) pants or textile, mesh, armored pants a good many times, too.
I just returned from a couple days at Bike Week and got some pretty scornful looks from all the bad, Harley folks every time I put on my full face modular helmet. I could care less. I only wanted to bring one helmet and it came in real handy on the chilly, wet ride down on Saturday and the cold ride home this morning. Besides, I kind of like the way my chin and jawbone look now and wouldn't want either of them rearranged if I went down.