My last bike was an XL TCR Composite2, and I found the front end a bit flexible as well, though I'm pretty sure it had nothing to do with the fork, stem, or handlebar. In my estimation, it was an issue of torsional frame flex, esp due to the top tube. I briefly pondered having the toptube reinforced by a carbon specialist, but never did it.
I think that stiff bars would make the most difference, a stiffer stem would help a bit, and the fork not so much (Giants forks are fine by me). That having been said, none of it will change the fact that the front half of the frame is wiggly.
For frame stiffness, sit with your legs gripping the saddle, grip the bars above the brake hoods, lock your arms and rock your shoulders side-to-side... you'll be able to look down and see the frame moving side to side. to check the bar/stem, keep stabilizing the bike with your legs, but grab the ends of the drop and try to twist the opposite ends of the bars up and down. Look and see how much the ends of the bars are moving relative to one another. Now just think how much its all moving around when you sprint out-of-the-saddle.