The Royal National Lifeboat Institution (RNLI) crew from Tower lifeboat station were called to the scene when the alarm was raised just after 8pm.
Passengers on a Thames water bus spotted the man holding on to a barge used to collect driftwood and alerted the authorities.
The man, who is Italian, told his rescuers that he jumped in because he thought he saw someone else struggling in the water.
He is thought to have been in the river, around 8C at the time, for up to 10 minutes.
Lifeboat crew Stuart Morrison, Chris Walker and Stephen Wheatley pulled the man from the water and on to their vessel before taking him to the London Ambulance Service who were waiting at St Katherine Docks.
Mr Morrison, 54, told the London Evening Standard: “It was dramatic.
“The tide was on the ebb which means he was in the path of the rubbish collector and so he was in the jaws of death as we rescued him.”
A Met police spokesman described the man as ‘suffering the effects of being cold water for a period of time’ but was ‘conscious and talking’ to police officers.
Water Safety Map applaud those try to save people in danger in water. However, please make sure you take care yourself when coming to the rescue.
Full details: www.watersafetymap.co.uk/map/details/199
“London Bridge” by www.flickr.com/photos/damo1977