I've had the same problem. Sometimes shooting a little compressed air in the port or taking a swipe in there with a clean cloth (not a q-tip or kleenex) can get rid of gunk that might build up from normal use (like the phone being in a bag or pocket that has dust, dirt, lint and such.) My old 3G (with the wide charger) was doing the exact same thing until I cleaned it out - had a TON of dog hair, lint, grime, tobacco and such from like four years of riding in my pocket. Pushing in the charger just pushes down and compacts whatever is in there.
Some people suggest a toothpick or using a little rubbing alcohol (like 90% so minimal watercontent) but whatever you use I'd recommend being gentle. Here's a video where they gently use a pin. Whatever you choose to try, I'd first shine a bright light in there and take a good look (even better if you have a magnifying glass). Here are some more suggestions if mine don't help.
My folks have big cover cases for their mobiles which have flaps that close over the charging port, I just looked in their charging ports and there didn't look to be any gunk, so maybe a good idea for moving forward with a lint free mobile lifestyle?