I agree with Ronin here. Doing a true polyphonic modular build is going to result in something very large and very, VERY spendy. Let's say you're talking a four-voice system. This means you need 12 VCOs if you want three per voice, four mixers to sum each set to their respective VCFs (also four of those), four exponential VCAs for just the audio level processing, EIGHT envelope generators (one each for the VCFs, one each for the VCAs), a one-to-four buffered mult per voice to allow you to send buffered CV to each VCO plus the VCF cutoff, 12 LFOs so that you can have discrete modulation sources for the VCO sets, VCFs, and VCAs to get pitch, timbral, and dynamic modulation, a four-voice MIDI converter unless you have a four-voice analog keyboard laying around, and some sort of stereo mixer to give you a stereo spread of your voices. Plus, if you want CV panning per voice, then you'll need four MORE LFOs, or perhaps looping AD envelope gens to make that happen, plus a CVable pan per channel on the mixer. And that doesn't even cover such things as clocking for sequencing/arpeggiation (the latter being quite useful with a polysynth) and the necessary module(s) for those two functions, FX processing, a stereo output module, and ultimately, a case with the space and amperage to deal with all of that.
OTOH, an 8-voice Moog One costs $6k. Same architecture (3 VCO analog) with a lot more going on, program storage, built-in controllers, MIDI, all the usual fun stuff. And while $6k sounds like a pile, consider what the four-voice iteration above would wind up costing, with which you wouldn't get half the programming functionality. And far less clunky to tote around, given that the Moog fits in a case, and the modular would be in a cab about the size of a medium-sized HDTV.
Moral of the story: polysynths should be polysynths unless you have the space and budget of Junkie XL, and modulars don't make good polysynths for the typical user.