If it's a very very slight wobble, you should be fine. Anything more, it's most likely your BB which should be replaced. Based on your description, no way it's the crank arms or spider, but just maybe your chainring is slightly bent (which can usually be fixed/bent back if you've got a good touch).