There's a bunch of issues here. First up, if you're looking to lose something, lose the drum modules. These days, the drum machines we're getting don't have many disadvantages over modular drums, so my suggestion would be to get a standalone. DO make sure it has clock in and clock out, though, as there'll be times you'll want to drive the Pam's with it, and at other times you'll need to drive the drum machine's clock with something in the cab. Also...noise module. Very necessary if you're trying to design your own percussives.
Next up...of the Joranalogues, I'd lose the Contour. As you'd suspected, Maths fits MUCH better here. The Dual comparator and the CV processor are super-useful for a number of things...the latter, modulation tampering and the former, if you add some Boolean logic, is a fantastic way to screw with clocking behavior by using freerunning slow LFOs to shut down and/or turn on different voicepaths.
BORG filter, not the Boogie. Rationale here is that the Borg gives you two VCF/gates with variable response. And this means you can have screwy things like a bandpass gate or a highpass gate in addition to the typical LPG thing, or you can couple both up together for weird filter curves.
I would also consider adding some dedicated ADSR envelopes for your main VCF, main VCA, and the like. The Zadar is awesome (esp. with its expander; the Batumi has one as well) but it seems more like the proper implement for all sorts of other modulation sources that you can implement all over the place in this build. Have a look at Doepfer's A-140-2...it also gives you some CV over envelope parameters, and it's only 8 hp.
The VCOs need a submixer. Since it's a set-n-forget module as a rule, a 2hp one will work here. BTW, since you really only have the three VCOs here, you can toss the Buffered Mult. These come in handy when you get above a certain amount of splits in a single CV and you want to avoid detuning from voltage sag...but three isn't quite in that area yet. More like five (or more) works with these. And you don't need buffering for anything else, so I'd suggest keeping some inline mults handy so that you can regain space you'd lose to mults.
Last thing I'd consider here would be to reorder the rack so that the signal flow makes more sense. As an example, right now you've got the Diode Chaos in the upper right...and the Select2, which has your sample and hold capability, is bottom-middle. These really need to be co-located, and there's plenty of other spots that need this, too. Try and envision what you're doing here as building up functional "blocks"...random sources here, VCOs there, filters in...well, you get the idea. Grouping by function really snaps a modular's capabilities into focus.