We are constantly exposed to the cold virus, primarily through touching things and then rubbing our eyes or noses. Our bodies ward these off all the time as they live in our nasal cavities. Under certain conditions (stress, changes in temperature, whatever) our immune systems fail and we succumb to symptoms.
Unlike, say, chicken pox, it's pretty rare to know exactly who exposed us to any given cold or flu. They're just all around us all the time.