I don't see any problem there, nope. I discourage people from getting a small rack if they're beginning with this because it's a good idea to have a broad-based starter rig when you first get into modular. But when someone has a specific musical concept in mind and they know it'll work best in a smaller form factor, then by all means go for it. And given that you're running a DFAM already, snagging two more 60 hp Moog cabs (watch your module depths!) for a total of 120 on which to expand the DFAM makes a lot of sense. Moog's designers already thought of this by offering their matching cabs and double and triple racks. Slap a couple of uZeuses in those and get busy!
One thing, tho...given that you're building this up around a DFAM and thinking of adding a Pam's to that, also consider a couple of things to deal with clock modulation and logic. That way, you can patch up all sorts of timing strangenesses that key off of counts (dividers and multipliers), CV (comparators, derivators, etc), pulse manipulation (width controls, clock delays) and randomfactors (probabilistic skippers, etc). Ladik makes a number of these that fit into 4 hp each, and Doepfer has their divider and ratcheting multiplier, and there's a few companies that offer nice, space-thrifty Boolean operator modules. May as well make that sequencer turn some cartwheels if you've got the space for a few of those.