It happened around 5:30 Saturday night near State Road in Tacony.
Police say the man was walking his Yorkshire terrier along the river when the dog got loose and went into the river.
The man went into the river and rescued his dog, but because the water temperature was in the 40s, he was unable to swim to shore.
The man grabbed hold of a pier piling some 60 feet from shore when a passerby saw and called for help.
Police marine units were able to rescue the man and his dog and neither required hospitalization.
