Veja o que o consulado geral francês tem a dizer :
The foreign spouse of a EU national enter France without visa if holding:
- A valid travel document ;
- A valid UK residence permit with the endorsement "family member of EEA national" (this endorsement is compulsory to be visa exempted);
- and if they are joining or travelling with the EU national.
Attention: To avoid unnecessary problems whilst travelling it is advisable to carry along evidence of family relationship.
Parece ser irrelevante que o documento de viagem não seja um passaporte, desde que seja válido.