Teoricamente, sim, você deve usar um passaporte japonês.
De acordo com o Lei de Controle de Imigração e Reconhecimento de Refugiados , Artigo 60 e 61 no Capítulo 7 ( Tradução em inglês ):
Article 60 (1)
Any Japanese national (except for crew members) who departs from Japan with the intention of proceeding to an area outside of Japan shall possess a valid passport and shall receive confirmation of departure from an immigration inspector in accordance with the procedures provided by Ordinance of the Ministry of Justice, at the port of entry or departure from which such person departs.
(2)
The Japanese national set forth in the preceding paragraph shall not depart from Japan unless he/she has received confirmation of departure from Japan.
Article 61
Any Japanese national (except for crew members) who returns to Japan from an area outside of Japan shall possess a valid passport (a document that certifies Japanese nationality if he/she is unable to possess a valid passport) and shall receive confirmation of his/her return to Japan from an immigration inspector in accordance with the procedures provided by Ordinance of the Ministry of Justice, at the port of entry or departure at which such person lands.
No entanto, na prática, você não precisa. Há muitos japoneses menores de idade nascidos de casais japoneses-estrangeiros nascidos fora do Japão, mas eu nunca ouvi ninguém foi negado uma entrada / retorno para o Japão.