I think that with any bike worth owning, it's worth making it fit well.
That may mean some swapping around of components. Try not to use that awful word "upgrade," because you're not going to be increasing the objective value or awesomeness of your bike. You're making it a better tool for you.
That can mean a different stem, different handlebars, different saddle. If you're using toe clip or flat pedals, cycling shoes and clipless pedals make a pretty big difference. If you don't know what you're doing, pay someone to do it for you. Ask some of the roadies in your area who they go to for a fit. Keep good records, so you can restore it if you tinker with it and make it worse and for future bikes.
Tires are a good purchase too, although I have to say that I'm noticing that difference less and less on road bikes, at least when it's dry out.
Drivetrain, wheel and fork upgrades are very expensive. IME, they don't do much. Certainly not as much as making the bike fit, or buying new tires. My experience is that a bike that goes, stops and shifts is already most of the way "there" in terms of maximum efficiency. So replacing a component that works with another component that also works is not a very good value.