Make sure you clean your helmet pads regularly and your glasses every ride. Everything you wash off your face after a ride is still on your hat and specs, so you don't want to put that back against your face the next day. You probably don't need to wash your helmet pads every time, but once a week will help.
I also wear thin coolmax skullcaps whenever I ride. I find they soak up/evaporate a lot of sweat before it ever gets to my helmet pads or glasses, and don't feel hot at all. They're not much help with the helmet straps--those I wash from time to time.
Note that I keep my hair VERY short, and they may not work as well for longer hair. One benefit I get that anyone would get (no matter the length of hair) is that the skullies can be positioned so they're between the helmet pads and my skin--since they're so easy to wash, I have fresh, clean material against my head every day.