I think if you want 2 voices you're better off going for the 6u rackbrute, or better yet a mantis (more space, better power and no rack wart wasting precious hp) - this is because the support modules that are needed for 2 voices are really squeezed into 3u and it will leave you some space to expand into without the extra expense of buying a second case...
& maths takes up a lot of space out of 83hp...
good starting advice: buy a bigger case, buy fewer modules to start!
I'd get a full size plaits clone - not the beehive - trimmers are notoriously imprecise - so if you want to use it for tonal music, it's difficult to tune!!!
I want this rig to provide lead melodies/arpeggios and atmosphere.
how are intending to sequence? no sequencer (except the random ones in Pams) & no aropeggiator... the keystep?
where's the atmosphere? no reverb, no delay???
What are some other modules that I should be focusing on for modulation? Or after I get a Maths, hypothetically, should I turn my focus to VCAs?
you've probably got enough modulation between Maths and Pam's... take a look at my signature - it's a rough guide to building a modular synth that's as versatile as possible, for the least outlay... you're missing sound modifiers and utiltiies... I'd want at least one filter for each voice - and as plaits has 2 outputs (not stereo as such) I'd want 3 and possibly more... plus at least one module that can do delat and another for reverb... unless you have pedals or something similar (which you don't state)
you'll also want to mix your sound sources...
sequential switches, sample and hold, attenuverters, attenuators and offsets etc etc are also highly recommended and incredibly useful - utilities are the dull (& inexpensive) polish that are needed to keep the shiny expensive modules shining
a separate clock divider is also very useful... great for creating bass lines from other audio... /2 = -1ve, /4 = -2ve
& yes maths has attenuators/offsets - but you'll be using them for patch programming maths - so will need more (download the 'maths illustrated supplement' for more information) - BTW patch programming sounds complicated, but it's just patching - in this case self patching a module to itself to get extended funtctionality...
& yes Pam's works as a clock divider - but I'm not sure it can divide external sources...
How many VCAs do you think I'd need for this type of setup?
more than you think you do... vcas are extremely useful and powerful - they are for modulation as well as for shaping notes - 2-3 per voice is a good rule of thumb - although as they're also useful for controlling volume - more is always better!!! there's a reason behind the meme - you can never have too many vcas!!!
a good starting point would be a veils clone (or similar) - which is a quad cascading vca - cascading in that it has mixing capabilities built in... you may find you want more than one of these (or at least an extra dual or triple vca)
"some of the best base-level info to remember can be found in Jim's sigfile" @Lugia
Utility modules are the dull polish that makes the shiny modules actually shine!!!
sound sources < sound modifiers < modulation sources < utilities