Gandalf era de fato um administrador da Terra-média
A missão para a qual os Istari foram enviados foi para guiar as pessoas livres da Terra-média contra Sauron. Como tal, os cinco membros da ordem podem ser considerados mordomos da Terra-média, no entanto, onde 4 falharam Gandalf conseguiu. Gandalf afirma isso com bastante clareza antes de dizer a linha que você cita em sua pergunta.
The rule of no realm is mine, neither of Gondor nor any other, great or small. But all worthy things that are in peril as the world now stands, these are my care. And for my part, I shall not wholly fail of my task, though Gondor should perish, if anything passes through this night that can still grow fair or bear fruit and flower again in days to come. For I also am a steward. Did you not know?
Return of the King - Book V, Chapter 1: Minas Tirith
É claro, não só desta passagem, mas também da missão do Istari que ele realmente protege e cuida da Terra-média.
Em seu ensaio intitulado O Istari , nos Contos Inacabados, Tolkien nos dá uma pequena visão de sua missão e porque Gandalf foi o único a permanecer fiel a ela:
for he dwelt in no place, and gathered to himself neither wealth nor followers, but ever went to and fro in the Westlands from Gondor to Angmar, and from Lindon to Lorien, befriending all folk in times of need.
Esta ideia é mais reiterada em O Silmarillion:
But afterwards it was said among the Elves that they were messengers sent by the Lords of the West to contest the power of Sauron, if he should arise again, and to move Elves and Men and all living things of good will to valiant deeds.
Of the Ring of Power and the Third Age.
E novamente no Apêndice B ao Senhor dos Anéis:
It was afterwards said that they came out of the Far West and were messengers sent to contest the power of Sauron, and to unite all those who had the will to resist him; but they were forbidden to match his power with power, or to seek to dominate Elves or Men by force and fear.
Appendix B: The Tale of Years