I just tried a minimal live rock farm and overall it's not doable. You lose a ton of bacteria buffer as far as breaking down food, so you might have mini ammonia spikes every time you feed. If you're planning on not feeding then your water will get way too clean as all of the nitrates and phosphates are removed by your skimmer and eventually your corals will starve to death.
Basically the deal is that you need to feed something to avoid dangerously low nitrate levels, but you also need a bacteria buffer to break down that food which you'll find in the live rock. Stability is the most important part of a saltwater system and the majority of that comes from your live rock, so with less of it you have less healthy corals and as a result slower growth.