Mac --
Pagan and LeLana BOTH raise excellent points.
Got an SO? Then, as Pagan implies, you don't want to look "out of the ordinary" as you depart directly from or return directly to an SO. Same is true if you're leaving/returning to a work environment full of nosey gossips. If you have an SO, then Woody Cialis's Coed thread, which is titled, "A Hobby Primer for the Married Man...", is MANDATORY reading:
http://www.eccie.net/showthread.php?t=48854
I have no SO, and live by LeLana's "presentation, presentation, presentation": Dressed as if for a date, and wearing designer boxers with some sex appeal.
On choice of attire, it's also wise to take "setting" into account: for ex., if you show up at some young hottie's residence incall late on a weekday morning, "black-tie" might draw unwanted curiosity from her nosey neighbors; not so much on a Saturday night, especially if she's a more mature lady living in an upscale condo or apt.
As LeLena indicates, the most important thing is to be clean and well-groomed -- whether or not you're at liberty to "dress up."
What to wear?
I'm pretty darn simple, and definally always looking cute
) 
Mmmmmmm-I'm hungry!!
