I think it is the wind over the ears. I've ridden bicycles for 50 years now, 30 of which without a helmet. I've always heard the roar. Try cory's experiment, then add this amazing twist: take your hands off the bar and cup them. Place them against your cheekbones, palms facing toward the rear of the bike. With your ears out of the wind, dead silence! You can hear the birds chirp 1/4 mile behind you. I do think it's possible that helmet design could slightly increase or diminish the air flow across the ears.
In the early 80s, someone marketed a set of clip-on ear flaps that went in front of your ears similar to those cupped hands above. I'm sure the contraption worked, but it made you look like a complete idiot. It didn't sell well. Perhaps someone here remembers the name of that thing.