Have you tried handheld bottles? I use them on short runs. If you're new to running don't compromise on hydration. Runaid "doughnut" shaped ones are cheap and come in several sizes. Or you can get bottles that fit in a holder with a strap for running with in your hand. Ultra runners use these a lot.
For longer runs I use a belt with small bottles - Nathan do good ones. Fuelbelt ones aren't any good now - they used to be excellent.
And on the longest runs I take a small running rucksack with a 2-litre bladder. You can stop the water jostling about by turning it upside down once you've filled it and using the tube suck all the air out.
I've never got on with waist pouches our belts with large bottles. I can't stand things bouncing about when I run.