Eh, it's one of those things like how Clara was a timelord once but remembers none of it, and if she was the one who recommended the Doctor steal that TARDIS, why the heck did he listen to her unless he knew who she was? Furthermore, it seems to counteract the description of events in "The Doctor's Wife" which is my favourite episode, and you DO NOT mess with my favourite episode-
(Stops, takes deep breath, repeats "Wibbly wobbly, wibbly wobbly, timey wimey..." until calm)
There are things that are there for an episode's plot, and there are things that are there for an arc's plot. Pretty sure this was the former, and will be largely forgotten. Besides, the Doctor has crossed his own time stream plenty of times, with no ill effects other than amnesia. This time, the Doctor never even left the TARDIS, which is a dimension unto itself and doesn't seem to affect the external timestream (see: Journey to the Centre of the Tardis).
CHTrilogy- Just enjoy what you can, and if it helps, look at some episode recaps and reviews online- that's certainly helped my understanding a few times, and increased my enjoyment. Don't worry about how much you know- fandom is meant to be fun, and if the "Gotta watch 'em all" mentality starts to feel like a chore, it's probably not worth doing. If people you meet are still criticising you for not knowing what colour the Doctor's toothbrush is, then, well, I'm not going to say that they're "bad fans" or "not true Whovians" or anything like that, but I will say that they are probably horrible people, and that your time could be spent in more welcoming company. Fandom should never be a superiority contest.