What you're looking for is a construction loan which is much more difficult to get than a regular home loan. Consider that building a home from the ground up has many more factors than labor and what I would call cosmetic fixtures - there are building codes, building permits, inspections for this, inspections for that - it's very involved and at the end more money than many people thought they would need.
There are programs for down payment assistance and great programs for first time buyers - USDA is one, but most of them would, in my experience put you through a process of paying off debts and cleaning up credit before they give you a loan. There are many things they look at and where low to moderate income might not be a problem, pending debts or job stability may be.
" Responsible lending" is the new words in the mortgage industry and because of everything in the economy and foreclosure issues . . . they are looking at everything to the letter
I'm the first to want everyone be a homeowner and advocate housing for all, but all things considered, you may want to get on a path of preparation towards that goal by observing your finances and clearing off debts - ask USDA, they will be more than happy to help you with what you need to qualify again - it may take time, but it will be worth it to be debt free and know your financially stable to purchase a home ( refer to an account who can advice on credit and finances). Also the Housing Finance Agency in your state, may have something for you - ask what you need to qualify or look them up
Best to you and hope this helps! Mary
CA. Real Estate Broker