I have two wheelsets for my gravel bike - 28s on one, 40s on the other. I have a few other road bikes, but do occasionally ride the gravel bike in street form for certain rides, e.g., slower, more casual rides, in-town "doughnut ride", etc.
My gravel bike is all carbon, light(ish) at 18lbs., etc., It's a nice bike. But, it's still not a road bike. The longer wheelbase, more upright position, wider bar... I didn't expect to feel the difference, but I do. It's not a deal-breaker. If I had one bike, it'd be a full-on gravel bike - not a road bike that can handle wider tires or an endurance kinda road bike - and two wheelsets.