I don't think that 20mpw is going to put in in the league of a long distance runner. You'd need to be running one run of 20 miles plus running virtually every day. You have nothing to fear.
I run, swim, cycle and have just started playing hockey. I run 45 mpw with a LSR of at least 15 miles. I am not skinny. Having said that, I'm not the most muscular of people you'll ever meet. 
If you enjoy what you do and are happy then you've got the balance right.