I'm going to give you the rope, but it's up to you not to hang yourself with it.
If you can't borrow the money from the builder or the bank then borrow it from yourself. do a google search on how to borrow from your retirement savings (IRA, or 401).
can you make mortgage payments with a credit card? if you can then you can always pay your mortgage with your credit card and pay for your new building with the cash you would have sent your mortgage company. this will only work until your credit cards are maxed out.
you could also contact your utility companies and see if you can sign up for a payment plan then use the extra cash flow to help finance your new building, you can ask them how many months you can go without payment before they shut you off. You could probably get at least 3 months of your utility bills to go toward your new barn.
skipping or short paying some of your current loans is another way you can raise cash.
doing all this will ruin your credit rating, but it won't matter because you'll have your new building, and you'll eventually pay everyone back. you'll look like a deadbeat on paper but it's one way to get the cash you need to build your building.
good luck
Whatever you do, do not destroy your credit rating, it's not worth it in the long term of things.
Banks are very careful when lending money today. I've found that they make decisions, and rightly so, to protect their investment. As a single guy back in the 70's I purchased my first house. I had a job as a teacher, so that was considered a low paying but "solid" job and was also an officer in the Army National Guard with more "solid" income. The loan qualifications were very conservative. My income wasn't enough to qualify. So I had to exaggerate (lie ?) about additional part time unreported income. I got the loan, and because I didn't have to worry about "family maintenance funding", the payments were easy to make. Might be the same, or even easier today. So, be creative, but make sure you're not jumping over the financial cliff.
As for the building, I use as a gage that a dried in pole barn building with concrete should be no more than $10-$15 per sq. ft. Price depends upon what options you have. No frills is what you need right now. Nice feature about pole barns is if placed on the lot correctly, adding on is easily done when funds are available. The building price quote you received seems to be TOO HIGH. How much was allowed for site preparation? A 40' x 40' @ $15 per sq. ft. is $24K. Somebody is getting rich at your expense. Better shop some more. Maybe the cost for what you're trying to do is why you aren't getting a loan approved?
After price shopping some more, I'd still build it, even less concrete, to just get started. I'm not one of those financially conservative, wait until I saved ALL the money saved up. I tried that back in the early 80's. A person couldn't save fast enough to keep up with inflation. When this low interest bubble bursts, you'll be happy to have a low interest rate loan. There's a good reason why there's banks, and that's why I use them. Only problem with some folks is they forget that after you borrow, you are obligated to pay the $money$ back. Like I always tell my buddies, "I'm not going to let a little thing like money get in my way of doing something", especially when I see something I really want that's a bargain.
I'm doing a 3,600 sq. ft. pole barn build right now (see gallery section). I borrowed the $dough$. I have "good stuff" to sell that'll cover the building cost, but I'm not going to short sell "good stuff" or pull money outa' my 401k when interest rates are at an all time low. Doesn't make financial sense. AND, I must be doing something right 'cause my credit rating, verified for this loan, was in the top 1%. Makes me wonder what kinda' financial decisions the other 99% are making

. It scares me to even think about it. Again, if you owe, pay it back, and make sure you're doing it "ON TIME".