There is a way of solving this, and you could have all 20 players compete in each round if you wanted. Start by looking at
this ancient thread, where the message I wrote contains a downloadable file accessible from the paper-clip link (you need to be logged in to the forum to see the paper-clip). Look at the 1st worksheet of this file - you can assign "Foursome 5" to be the byes, then make the following changes to the player names:
A1 to A5 -> M1 to M5
A6 to A10 -> F1 to F5
B1 to B5 -> G1 to G5
B6 to B10 -> L1 to L5
The transformed schedule should meet all your requirements - it was constructed using 4 mutually orthogonal Latin squares where each square represent one of the groups of players above. To see an example look for
a set of MOLS(5) on this Wikipedia page. The other tabs in the spreadsheet gives similar schedule for multiples of 4 players from 20 to 72. Hope that helps.