Can you hang, and take good pulls, in fast, drop "Worlds" group rides? If so, then you're probably in good enough shape for Cat. 4.
Can you drop other racer-types? If so, you're probably in good enough shape for Cat. 4.
If not, start riding with faster, but more importantly, patient cyclists, who will wait for you at the top of a hill or after a sprint. You'll get stronger and learn how to ride in a group.
Also, depending on where you live, sometimes the Cat. 4 races are faster than Cat. 1/2 races. As an example, the Glencoe Grand Prix Cat. 4 race saw lap times exceed the upper category races due to the presence of semi-pro Cat. 1 CX racers who are Cat. 4 road riders due to the idiosyncracies of the USAC licensing and upgrade system.
Regions of the country with large pools of talent have lots of folks with big engines who can't get upgrade points for whatever reason -- Chicagoland is one. However, in Ohio and Indiana, with their smaller scenes, the races are sometimes a little slower and forgiving.
Your experience may vary, basically.