I’m making a fighter who was tortured in the past. He has a badly healed broken nose, one of his ears was cut off, his face was branded, etc.
I was thinking it would be neat for him to have advantage on Intimidation checks because he’s horrifying-looking and disadvantage on Charisma checks because of his looks - but I was wondering if that would be balanced, or if maybe he should have other disadvantages to even it out.
There is actually precedence for this in the variant rule for Lingering Injuries on page 272 in the Dungeon Master's Guide:
Horrible Scar. You are disfigured to the extent that the wound can’t be easily concealed. You have disadvantage on Charisma (Persuasion) checks and advantage on Charisma (Intimidation) checks. Magical healing of 6th level or higher, such as heal and regenerate, removes the scar.
These injuries are intended to provide either a neutral or detrimental effect, so the game designers, at least, see it at as balanced.
In my experience, ability checks that involve Intimidation are prompted by the GM much less frequently than other Charisma checks, but that will vary from table to table and from game to game. Overall, I don't see a balance issue with your proposition.
Tudo o que foi dito, it is ultimately up to your GM whether to allow it in the game, as there are no rules to cover starting with injuries in the rulebooks (and the Lingering Injuries rules are optional).
The character is clearly better at intimidation than normal, but worse at other tasks requiring charisma. The weakness here is easily circumvented in usual circumstances by having a dedicated "party face" --- that is, another character who handles the more diplomatic tasks dedicated to charisma while the scary Fighter sits on the side. The weakness in other charisma tasks doesn't really kick in unless the character is forced to make charisma checks as a part of their adventuring, or if their player is eager to push their character towards more challenging rolls despite them being quite weak at them.
How big is the impact depends almost solely on the GM, though. Ability checks have few rules and their role varies a lot between tables: some GMs allow bypassing entire encounters with well-executed Persuasion, Deception or Intimidation checks, while in other tables all those checks will earn you is a cheaper night in a tavern or five gold off the price of a fancy broadsword. There's no single right answer here --- you will have to judge where your game lies on this spectrum yourself.