suggestion #1, if the goat you are getting is male get him wethered, you do NOT want an intact male goat when he goes into heat...belive me, most breeders dont even want intact males during rutt.
he will grow just as nicely as a meat goat if hes wethered.
#2, get him a friend a second wether if your just raising for meat/scrub, goats are highly social and a single goat is a VERY miserable goat, its my experience that loan goats tend to cause the most trouble with fence busting too.
goats will eat brush and scrub but some breeds can be picky, quite honestly if your looking for true scrub goats (goats that will eat anything) id look into the TN fainting goats, or the spanish goat, both of which are great for raising for meat but typically NOT picky about what they eat.
boer goats are meat goats, bred to grow fairly fast and very large and can be picky eaters, also depending on the quality of scrub you have available youd probbaly have to suplimental feed.
in terms of fencing, yes goats can be very hard on fences, youll want cattle pannels or sheep/goat fence with well set corner posts (t-posts work well for inbetween corner spans) but put the post on the OUTSIDE of the fence, it will provide more support when the goats stand on and rub against the fence. if the fence is on the goat side and the posts on the outside of the fence youll get more support.
goats are wonderfull if you get the right ones for the job you want done, but i wouldnt suggest an intact male and id never suggest a single goat.
if all you want the goat for is to remove scrub id go with spanish goats, if your looking for meat id go with fainters(mytonic) or boers, but the fainters arnt typically as picky about browes lol.