Qual foi exatamente o motivo da antipatia de Denethor por Faramir?

Isso é discutido no Apêndice A (ênfase minha):

Faramir the younger was like [Denethor] in looks but otherwise in mind. He read the hearts of men as shrewdly as his father, but what he read moved him sooner to pity than to scorn. He was gentle in bearing, and a lover of lore and of music, and therefore by many in those days his courage was judged less than his brother's. But it was not so, except that he did not seek glory in danger without a purpose. He welcomed Gandalf at such times as he came to the City, and he learned what he could from his wisdom; and in this as in many other matters he displeased his father.

^{Return of the King Appendix A: "Annals of the Kings and Rulers" I "The Númenórean Kings" (iv) Gondor and the Heirs of Anárion The Stewards}

Não é verdade que Denethor não gostava de Faramir, é que ele não conseguiu viver de acordo com os padrões de masculinidade de Denethor, especialmente quando comparado a Boromir, então ele sempre foi uma espécie de desapontamento. A morte de Boromir ofendeu Denethor muito, e ele respondeu atacando o filho decepcionante que ele havia deixado.

Para ser justo, parte disso é abordada na edição ampliada do filme O Retorno do Rei :

Denethor: Ever you [Faramir] desire to appear lordly and gracious as a King of old. Boromir would have remembered his father's need.

[...]

Denethor: Boromir was loyal to me! Not some wizard's pupil.

^{Return of the King (2003)}