I am most comfortable in my sweatpants. Most likely because they keep me warm and that helps a lot when I do my morning workout. I tried do it without the t-shirt and the sensation of skin sticking to the mat was unwelcome to say the least.
I just don't see the point in working out naked. I actually think it was made up just to cause a stir, or scandal, among "regular", "normal" people. Just to poke us in the eye.
If you are really good in what you are doing and like the idea go ahead.