Mine is extremely low in nitrogen too, the only place I can grow things has taken me years of work and mixing chicken manure in. I’m about to expand my garden spot for next year and I plan on getting as much horse manure as I can to start turning into the soil. At least inside my fence is better. I’ve had animals fertilizing and clover growing in it for years. Next person to get this property will be far better off.
*nods* i think years ahead for my gardens when it comes down to composting in place (aka bury and let the worms figure it out). if i bury the longer term items down deep enough they're usually somewhat decomposed the next time i get to that space down that deep. often i find places where i've buried things and it has turned into material that looks just like peat moss. it's a treasure hunt.
if you are doing rotations of different crops you should be able to get by without a huge amount of fertilizer being required or added. i only add my worm compost (and worms because i don't sort them out) to crops that are the heaviest feeders (tomatoes, onions and some peppers) the rest of the crops do well on 2nd year or even 3rd year rotation through that space.
another way to get some nitrogen is to have some alfalfa, large red clover, trefoil and clovers, you can chop this and use it as part of a mulch and eventually that's prime fertilzer thanks to the worms.
keeping the worm buckets has been such a great learning experience for me and not using any synthetic chemical fertilizers has also been good. it makes me think of the longer term and how to keep nutrients cycling in place and also aware of how much i harvest from an area so i know what i have to return to that same space.
harvesting sod (letting an area go fallow and then taking chunks of what grows and burying it) after a few years is also a way to take advantage of what nature, weeds, worms, ants, etc. can do for you in bringing up nutrients from your subsoil (and below that your bedrock). this ultimately your base carrying capacity for the bulk minerals and some trace elements for a location. so above that for minerals you only have to account for what you remove and what you might have to supplement for plant growth, but once you have an area properly set up there shouldn't be too much need to bring in trace elements. the harder parts are keeping your erosion rate within bounds and also keeping certain issues from getting worse (salinity if you irrigate, hard pan layers if you till, etc.)
and of course, road kill will be a highly rich source of nitrogen and other nutrients not normally used in a garden. while it may not be pleasant at times at least it is usually available.
. Wait is the hard part. Been looking at seed this morning for only the 2 that I feel I "need"....s/h made them unacceptably expensive. So, by adding something else, a spread the s/h cost over more. 
. Any seed addict can see that makes sense!!
