various factors
I've climbed a lot of the Alpine TdF passes while cycling as part of organized tour groups as well as many of the Giro climbs when I lived in northeast Italy while stationed there with the U.S. Air Force. A lot of Alpine/Dolomiti climbs are just plain hard due to several reasons:
1) Length. The Madeliene and the Telegraphe/Galibier are very long. Nearly 20 miles of climbing. While they have some stretches of tough grade, its the length and overall elevation gain that gets to you. At normal human pace, you are looking at nearly 2 hours to climb each of those. The Telegraphe/Galibeir is higher, but the Madeliene makes me suffer the most between those two.
2) Grade. The south side of the Marmalada/Passo Fedaia in the Dolomites, after fooling you with an easy opening segment, has a 14% grade that goes on for miles with no let up. When it does let up with a few switchbacks, it still throws extended 14-15% ramps at you until you reach the top. Ouch. The Alpe d'Huez is also steep...especially at the bottom. It kicks up right from thestart with a 11+% grade that goes on for quite a while before you get to the first switchback. The Alpe does give you some break in the switchbacks because the turns are relatively flat...of course the pros use that as launching pad while I use it catch a mini-break for a few strokes. While the Alpe d'Huez is not excessively steep, it's steep enough and persistent enough that it makes you work very hard.
3) Heat. The Alpe d'Huez and the Marmalada are very exposed to the sun while not being high enough in altitude to be cooler than the valleys they come out of. There is not one inch of shade on the Marmalada once you get on the steep upper 2/3 of the climb.
4) Where the steep parts are. Some climbs hit you right off the bat with no warmup (Alpe d'Huez) which puts you in the red with a long way to climb. Some climbs throw their steep parts at you near the top when you've already spent a lot of energy to get there (Madeliene). I haven't climbed the Ventoux, but I hear its a tough combination of length, grade, and heat all packed into one.
5) Pace. Pretty much under your control unless you are racing. I learned to pace myself carefully over long climbs. On the lower 2/3 of a climb, you should feel like you could climb faster if you want to...but beware, you will need that energy for the top 1/3. Go into the red too early and you will suffer very badly to reach the top. There is a reason you see the top TdF contenders waiting until the last few km to attack on summit finishes. Extremely difficult to recover on a climb and if you stop its very difficult to regain your rhythm.
My favorite climbs of all are the Dolomites. With some exceptions, they aren't too long (2200-2500' elevation change), aren't too steep (7-9% grades), start at higher altitude resulting in cooler summer temps (1500' at the base), are packed in close together so you can climb multiple passes in an afternoon ride, and are extraordinarily beautiful.
