Towel or facial sponge?

I would recommend you wash your face with your hands instead. If you want to exfoliate, get an exfoliating acne wash. Towels tend to leave little strings and whatnot on the skin. This could also cause irritations; and yes, you're right about the facial sponges. Sometimes they are too rough on the skin. If you absolutely want to use a towel to clean the face, try dampening it first.