No matter how many times you wash your hands (and one should always do this before handling philatelic material), there is always a tiny amount of oily residue that adheres to one's skin---and which can migrate to any kind of paper item. When this oily residue gets onto a stamp or cover, it will stay there forever and, in the long run, promote the deterioration of the item.
STAMPS should be handled with the readily-available "stamp tongs" that come in many sizes and styles. By "styles," we mean the various kinds of tips on the end of the tongs. Some tips are shaped like tiny spades, while other tongs have tips that are shaped to a tiny point at the end. Pick a pair of tongs with which you will be comfortable. Tongs take a little while to get used to but after a while, every collector using them begins handling stamps like a pro! Remember, though, that the tongs with small points at the end can sometimes puncture the paper of a stamp. Be careful.