How do you define hi-fi? You refer to it often. Do you mean hi-fi consumer audio? Record players, amps, speakers, a collection of vinyl, etc... or hi-fidelity musical instruments? What experience do you have with creating music... especially synthesizers? Depending on where you are starting from, you'll have different challenges. My concern is that you're coming from the position of enjoying the playback of great music and now you'd like to create music. Modular is an expensive format to be your entry point into making music. While not impossible, you're stacking the odds against you if you're learning to make music at the component level. It's not a terrific analogy... but it's like buying Formula-1 race car as the vehicle you're learning to drive with.
The worst thing would you be spending a small fortune, getting frustrated and burned out, then walking away from your endeavor. Modular synths are awesome... but then again I have some bias. :) I just want you to have an enjoyable experience. Have a think about where you're at and if you're entering modular a little soon... if you're coming from hifi audio.
There are no wrong decisions and I've assumed a bit in my response. But if you've never programmed a synthesizer before, you're taking on a huge challenge.