O próprio Tolkien aborda isso em um ensaio chamado "Leis e costumes entre os eldar". Existem basicamente dois problemas:

1. A libido élfica diminui com o tempo. Ao contrário da crença popular (e da aparência externa, de uma perspectiva humana), os elfos fazem idade; e à medida que envelhecem, ficam menos interessados na procriação.
2. Ter filhos é exaustivo . Os elfos acreditam que o processo de gravidez deles causa um impacto físico e espiritual mais pesado, comparado com o que acontece com os humanos. Eles são compreensíveis relutantes em ter muitos filhos, porque qual é o sentido de ter uma dúzia de crianças se isso literalmente matar você?

De "Leis e costumes":

It might be thought that, since the Eldar do not (as Men deem) grow old in body, they may bring forth children at any time in the ages of their lives. But this is not so. For the Eldar do indeed grow older, even if slowly: the limit of their lives is the life of Arda, which though long beyond the reckoning of Men is not endless, and ages also. Moreover their body and spirit are not separated but coherent. As the weight of the years, with all their changes of desire and thought, gathers upon the spirit of the Eldar, so do the impulses and moods of their bodies change. [...]

Also the Eldar say that in the begetting [conceiving], and still more in the bearing of children, greater share and strength of their being, in mind and in body, goes forth than in the making of mortal children. For these reasons it came to pass that the Eldar brought forth few children; and also that their time of generation was in their youth or earlier life, unless strange and hard fates befell them.

^{_{History of Middle-earth X Morgoth's Ring Part 3: "The Later Quenta Silmarillion" Chapter 2: "The Second Phase" Laws and Customs Among the Eldar}}

[A]ll the Eldar, being aware of it in themselves, spoke of the passing of much strength, both of mind and of body, into their children, in bearing and begetting. Therefore they hold that the fëa [soul], though unbegotten, draws nourishment directly from the fëa of the mother while she bears and nourishes the hrondo [body; later changed to the word hroä], and mediately but equally from the father, whose fëa is bound in union with the mother's and supports it.

^{_{History of Middle-earth X Morgoth's Ring Part 3: "The Later Quenta Silmarillion" Chapter 2: "The Second Phase" Laws and Customs Among the Eldar}}

Os elfos têm apenas um pequeno número de crianças em um pequeno período de tempo.

Os elfos só procriam por um certo período de suas vidas, como Tolkien disse (entre aspas):

The bad news is that elves tend to lose interest in sex after they've had kids. "With the exercise of the power (of generation), the desire soon ceases, and the mind turns to other things…they have many other urges of body and of mind which their nature urges them to fulfil." They do look back happily on the sexually-active time in their lives, though, a period of one to several hundred years.
_{What Tolkien Officially Said About Elf Sex; quote inside from "Laws and Customs of the Eldar", one of Tolkien's essays.}

Quando eles fazem têm filhos, não há muitos:

Seven children was the usual upper limit in ancient times, but in future ages, partly due to varying levels of corruption in various elven societies, these norms were often exceeded rarely, even when corrupted do they ever succumb to deeds of Lust.
_{Elf children}

Então é uma vez e feito, enquanto Elfos são jovens, e então as coisas ficam mais lentas. Durante a maior parte de suas vidas, os Elfos não têm filhos jovens (nem mesmo relativamente jovens).

Para complementar a excelente resposta de Jason Baker, eu queria acrescentar algo sobre o número de crianças também do mesmo ensaio no anel de Morgoth. Gostaria de salientar que parece haver algo na resposta da HDE que deve ter sido originada de um wiki; no entanto, o comentarista Harry Johnston, naquela resposta, perguntou se alguém poderia traduzir uma citação nessa resposta. Eu fiz isso, mas ele e eu pensamos que não se parecia com Tolkien. Já que fala sobre o número de crianças que os Elfos tendem a ter, eu pensei que acrescentaria algo do mesmo ensaio que o Jason citou:

The Eldar wedded for the most part in their youth and soon after their fiftieth year. They had few children but these were very dear to them. Their families, or houses, were held together by love and a deep feeling for kinship in mind and body; and the children needed little governing or teaching. There were seldom more than four children in any house and the number grew less as the ages past but even in the days of old, while the Eldar were still few and eager to increase their kind Fëanor was renowned as the father of seven sons, and the histories record none that surpassed him(4).

Notas de rodapé:

(4): Na verdade, havia dois rascunhos deste ensaio; Christopher mostra o principal e depois mostra divergências. Neste ensaio, na nota de rodapé 4, ele tem isto a dizer sobre a versão A:

'.. while the Eldar were still few, and eager to increase their kind, before the weight of years lay on them, there is no record of any number more than seven', with 'seldom' written later above 'no'.

Resumo

Nas eras anteriores, eles estavam mais ansiosos para aumentar sua espécie, quando tinham menos. Com o passar das idades, esse desejo diminuiu - como a resposta de Jason aponta; a outra resposta sugere que sete era o limite superior, mas geralmente não era mais do que quatro e, à medida que as idades passavam, o número crescia menos. Fëanor foi um dos que realmente teve sete e nenhum o superou nesse sentido.