A good rule of thumb is to pick a couple main colors and an accent color, then mix up the patterns and textures. In the first board, I've used a combination of blacks and grays with teal as an accent color. A nice floral or plaid print in the same color family would look great, too!
In the board below, khaki and brown are the main colors, with green as the accent.
All of the outfits are from Old Navy or GAP (a few years ago) so they are affordable. Take items that you already have, then go buy a couple fun accessories or patterned pieces to tie everything together.
A variety of pattern and texture, while in the same color scheme, adds a sense of casualness that I think many people are hoping for in their family or group photos. If nothing else, choose pieces that you and your family love and feel comfortable in. Happy Fall!