First, I had left apparently... only to return with no memories of this place. Sora, remembered me being here before. I had been telling of him of Castle Oblivion at the time it seems.
Then Roxas... he disappeared too. Now he has no memories of this place, but why? It makes no sense.
Why would this place take us here, then send us back out with no memories of it, before bringing us back? Did we find out something we shouldn't of?
Something... we needed to forget? I can't think of another reason.