Isso é tratado no ensaio de Palantiri em Contos Inacabados:
These Stones were an inalienable gift to Elendil and his heirs, to whom alone they belonged by right; but this does not mean that they could only be used rightfully by one of these "heirs." They could be used lawfully by anyone authorized by either the "Heir of Anárion" or the "Heir of Isildur," that is, a lawful King of Gondor or Arnor. Actually they must normally have been used by such deputies. Each Stone had its own warden, one of whose duties was to "survey the Stone" at regular intervals, or when commanded, or in times of need. Other persons also were appointed to visit the Stones, and ministers of the Crown concerned with "intelligence" made regular and special inspections of them, reporting the information so gained to the King and Council, or to the King privately, as the matter demanded. In Gondor latterly, as the office of Steward rose in importance and became hereditary, providing as it were a permanent "understudy" to the King, and an immediate viceroy at need, the command and use of the Stones seems mainly to have been in the hands of the Stewards...
(minha ênfase em todo)
Então, para responder à sua pergunta: É apenas a linha de Elendil e aqueles a quem eles dão permissão?
Sim, mas deve-se notar que os comissários por padrão normalmente tinham essa permissão.