Let's take it one part at a time.
Frame material is NOT a major factor. True in that all the commonly used materials are capable of being built into a good bike.
A carbon fibre bike is not inherently faster than an aluminum or steel one. Comparing four bikes, aluminum, steel, titanium and CF leaning against a tree, this is true. Bikes have no inherent or innate abilities, riders do.
It may be a little lighter (but may not), True, a CF bike can be lighter or heavier.
and that may make it a bit faster is some situations In general, a lighter bike will make you faster. If you do the calculations, and if all variables are controlled at the same value (drag, weight, frame properties, wind, elevation, temperature, air density, rider, rider's nutrition, rider's condition on the day of the tests, components, others), and weight is the only variable, then yes, it will take less energy to get you from point A to point B which might mean that if you have more energy to give then that extra energy will translate into more speed. Now whether this increase in speed is actually detectable, and whether it makes any difference in your day to day life, is up for debate.
but it isn't necessarily faster in most settings True. See previous answer.