I believe that it is impossible to have a schedule with these properties. The schedule below was found by a computer search and may be the best balance that it possible.
[(M2 F2):(M1 F4)] [(M4 F3):(M6 F5)] [(M3 F6):(M5 F1)]
[(M2 F5):(M6 F1)] [(M3 F2):(M4 F6)] [(M5 F4):(M1 F3)]
[(M3 F4):(M4 F5)] [(M1 F6):(M6 F3)] [(M2 F1):(M5 F2)]
[(M2 F6):(M3 F3)] [(M6 F2):(M5 F5)] [(M1 F1):(M4 F4)]
[(M5 F3):(M4 F1)] [(M6 F6):(M2 F4)] [(M3 F5):(M1 F2)]
[(M1 F5):(M5 F6)] [(M4 F2):(M2 F3)] [(M6 F4):(M3 F1)]
Each man plays with each woman once, same sex opponent pairs occur at least once and at most twice. The problem is the opposite sex opponent pairs, everyone has a player whom they don't oppose and another player whom they oppose twice. For example M1 does not oppose F1, but opposes F4 twice.
For other number of players other than 12 and 20, it is possible to construct a similar schedule with exactly the same properties, simply take one of the spouse avoiding schedules and add an extra round (for example, follow the link to 16 players in
this thread.