There a many reasons for this but one of the most fascinating is that we actually constantly alter our memories every time we access them. We don't do this on purpose but instead it's an involuntary side effect of the process of activating memories.
How Our Memories Work
Memories are essentially stored as connections between neurons in the brain. These connections link together various experiences, stimuli and emotions -- which is why hearing a song or smelling cookies can suddenly bring fragments of memories hurtling back. It's unusual for us to forget memories completely but if they aren't accessed often enough or if they aren't linked to enough cues, there won't be enough strong 'in-roads' for us to find them.
This all takes place as a result of the brain's inherent plasticity. The brain is plastic just like muscle and it changes shape and size in response to the way we use it. Every experience you have has this effect of altering the layout of your brain and each time you repeat a movement or experience, you strengthen the pathways that represent it. Sing a song over and over again and the synaptic connections between neurons that encode that song will strengthen (specifically, this occurs through the myelination of axons -- the thickening of a sheath that protects cells and enhances their conductivity).
What Accessing a Memory Does to It
Every time you access one of your memories, the same thing happens. The process of retrieving those memories strengthens the connections that lead to it and those that create the complete picture. Meanwhile though, you also create new connections with whatever is going on at the time. In a hot room? Then you may well end up creating a connection that leads to you remembering the event as being hot -- rightly or wrongly.
Then there's the role of our 'schemas'. These are the blueprints we hold in our brain for various things -- our prototypical examples of things we see in the world. For instance, we have schemas for what people look like and when we picture a scene, that's what the people in the background will look like. Similarly, we have schemas for what lamps look like and what table cloths look like. We can't pay attention to everything going on around us at once and so when we remember an environment we often find ourselves unconsciously 'filling in the gaps'.
At the same time, you actually form a memory of yourself remembering that memory. Often it's this memory that you then retrieve the next time that you try and think back -- including all the inaccuracies and mistakes from that fabrication.
In short then, the simple act of remembering something is enough to distort our memory of that event -- eventually to the point of it becoming entirely false.
As researcher Donna Bridge put it:
"A memory is not simply an image produced by time traveling back to the original event--it can be an image that is somewhat distorted because of the prior times you remembered it. Your memory of an event can grow less precise even to the point of being totally false with each retrieval."
You can't step in the same river twice!