I just got started too, so here's what I've got going so far:
We built a 3.5 sided goat shed built sorta skid style. It has a dirt floor with a foundation of treated posts - nothing concreted or dug in, we wanted to be able to move it if we should ever want to. Walls are framed with 2x4s and we used T1-11 siding to sheet it all. The roof is sheeted and then we used metal roofing we got for free from Freecycle. So all that to say we had to buy about 30 2x4s and 6 or 7 sheets of siding. I could have made it for less but wanted to keep it from being an eye sore since it's very visible from our front porch as well as to a couple neighbors. Price will vary in a huge way depending on lumbar costs for you but we built it for about $200. We got half our fencing for $75.00 on craigslist (chainlink) and bought t-posts and horse fencing for another $250ish.
My 2 nubian goats were $350 each, bred. I bought them from an organic dairy farm that was down sizing. However, my husband and I both did trade work for the purchase price. I brooded chicks to laying age and my husband did some construction work for her.
Hubby made my milking stand from the same plans that Ohiofarmgirl used and it works well. I think he spent less than $50 for the lumber and hardware for it. Saved LOTS of $$$ there - the price of milk stands is crazy! I bought a 4qt ss milk pail from my local feed store for $10 and a mini strainer from Hoegger goat supply for around $15ish. It fits my canning jars that I use to store my milk. I do have to buy fiters to fit - the box of filters was about $7 and it has 300 filters in it. Twice a day milking means I'll go through 2 boxes annually.
Alfalfa/grass mix hay was $120/ton. We bought and stored 2 tons - so $240. We have a 20ft delivery truck so we were able to haul it ourselves.
My girls are dry so I'm not using grain currently, but when I do I get 45 pounds organic oats for about $11.00 and 45 pounds organic barley for about $12.00. Not sure how long a bag will last yet since my girls are dry. I've spent about another $50.00 on other feed supplements to get started.
I recently paid another $175 on some meds, kidding supplies and some other necessities (hoof trimmers and such) but much of this will likely last me a very long time. My does were raised *naturally* and I hope to continue, so I have not purchased vaccines but did get some antibiotics to keep on hand just in case. Some of what I bought I hope I'll never use but I figured it was best to have it and not need it then need it and not have it. If I had a bigger feed supply store around me I may not have needed to buy some of it, but we're limited and I didn't want to be caught with an emergency and not have access to what I need.
I bought a bucket for watering - about $15 bucks for a 4 gallon flat back rubber bucket that's weather proof. I spent $5 on a 2 compartment mineral feeder and another $8 on the feeder that attaches to my milk stand. My hay feeder is modeled after a version that Freemotion made and posted on backyardherds (Thanks Freemotion!) and aside from a couple hinges and ball-bungee cords, it was made with material we already had lying around.
Ok, so there it is. Now that it's all listed I should go back and tally it up, though I'm not really sure I want to know.
FWIW, for our family size, when we're NOT buying organic milk, we spend about $50 per month on milk. Double that cost when we have room in our budget because we prefer to use organic milk. That's JUST milk, and we don't "drink" our milk anymore, can't afford to. That's just what we use on our breakfast grains, for baking, and for coffee. We'd go through twice that if we all were drinking it - which hopefully we will be come milking time. And this doesn't include other dairy products that we use (yogurt, cheese, sour cream, cream cheese, ice cream, cottage cheese). That's another EASY $100 per month. So what I'm getting at is my goats should easily pay for themselves in milk. I have a few friends that are interested in buying milk if I should ever have extra (strictly for pet use, of course

because you can't sell raw milk to drink in WA). It may take a little bit to pay for the shed and the fencing, but not long if we don't have to buy anymore dairy products - except butter.
I know there will be surprise expenses along the way (vets, etc), but to have good quality, healthy, clean, raw milk - PRICELESS!!!!