Doesn't seem workable to me. Those utilities that you don't think are necessary are, in fact, VERY necessary. Same goes for the modulation sources and the modules needed to properly control them.
I've done plenty of work over the years with "deterministic" sounds as well...but the difference is that while I might have a certain module or two that I know I want to feed a sound through, I also know that a sound is far more interesting to a listener if it's doing something more than just sitting in the mix like a bump on a log. So, the choice of operating modality for that would be either...
1) Grow a couple of extra hands, then you can work all of that module or two's controls simultaneously. Or...
2) Put the right modules in so that you don't have to have your DNA mutated for those extra hands.
Basically, you've definitely got the idea of what modules to use to alter audio...but the lack of modulation sources, any way of mixing/controlling those, and no way to arrive at any routings other than straight-thru as a result really puts a helluva spike in this build's heart. And that's even if you ARE using it as a processor; I have ZERO idea of how you're getting it to do proper FM without the attendant crossconnection modules between two VCOs or the absence of a proper complex VCO.