No último episódio da 9ª temporada, por que o Doctor precisou apagar a memória dessa pessoa?

Porque não era seguro que os dois ficassem juntos.

O próprio médico diz (ao relatar suas aventuras a Clara no restaurante, quando ele não sabe quem ela é):

DOCTOR: There was only one way to keep Clara safe. I had to wipe some of her memory.
CLARA: Of what?
DOCTOR: Of me.

E mais tarde, quando fala com Ashildr no final do universo, ele explica porque (ênfase minha):

DOCTOR: Clara's my friend.
ASHILDR: I know. And you're willing to risk all of Time and Space because you miss her. One wonders what the pair of you will get up to next.
DOCTOR: Nothing. Nothing at all. I know I went too far. I get it. That's why I'm doing what I'm doing.
ASHILDR: And what would that be?
DOCTOR: I'm taking her back to Earth. Somewhere safe, somewhere out of the way. I'm going to wipe her memory of every last detail of me.

Quando Clara "inverteu a polaridade" do bloqueio neural, podemos supor que ele fez exatamente o mesmo para o Doutor, como ia fazer com ela: limpou suas memórias dela . Foi muito seletivo, deixando suas memórias de eventos (para que ele ainda fosse a mesma pessoa mesmo após a limpeza), mas certificando-se de que ele nunca seria capaz de encontrar ou reconhecer Clara novamente. Mais tarde, na lanchonete, vemos que ele lembra muito do que eles passaram juntos, mas nada sobre ela pessoalmente, como ela se parecia ou nada. As únicas lembranças que ele perdeu são as de Clara como pessoa - ele ainda se lembra de tudo o que eles fizeram juntos e, claro, tudo da sua vida antes de conhecê-la.

DOCTOR: When something goes missing, you can always recreate it by the hole it left. I know her name was Clara. I know we travelled together. I know that there was an Ice Warrior on a submarine and a mummy on the Orient Express. I know we sat together in the Cloisters and she told me something very important, but I have no idea what she said. Or what she looked like. Or how she talked. Or laughed. There's nothing there. Just nothing.

Ele explica isso no próprio episódio:

Doctor: When you wake up, you will have forgotten me. You'll have forgotten we ever even met.

Clara: And why would I want that?

Doctor: Because it's the only way. That stuff in your head, the image of me, they could use it to find you.

^{Doctor Who Series 9 Episode 12: "Hell Bent"}

No entanto, o fato de que ele está disposto a fazer o compromisso (potencialmente apagar sua própria memória em vez da dela) sugere a verdadeira razão: ele e Clara são basicamente perfeitos um para o outro, e são incrivelmente > perigoso para isso. Ele dá essa resposta um pouco mais tarde, também:

Doctor: You and me together. Look how far I went, for fear of losing you. This has to stop. One of us has to go.

[...]

Doctor: It's okay. It's okay. I went too far. I broke all my own rules. I became the Hybrid. This is right. I accept it.

^{Doctor Who Series 9 Episode 12: "Hell Bent"}

Que combina muito bem com um comentário feito anteriormente por Ashildir / Me:

Me: What if the Hybrid wasn't one person, but two?

Doctor: Two?

Me: A dangerous combination of a passionate and powerful Time Lord and a young woman so very similar to him. Companions who are willing to push each other to extremes.

Doctor: She's my friend. She's just my friend.

[...]

Me: I know. And you're willing to risk all of time and space because you miss her. One wonders what the pair of you will get up to next.

^{Doctor Who Series 9 Episode 12: "Hell Bent"}