Hi Zeppe807,
I have been doing bladesmithing for the past ~8 years. Trying to remain frugal and space conscious I have went through multiple different forge designs.
I have settled on two different types of forges for general forging work. Each based on what your intended goal is.
My high temp forge is based upon a Don Fogg vertical style forge. The vertical forge is great for welding as the flux and junk falls to the bottom of the forge. This is powered by an old residential oil burner, converted to burn WVO and kero to save on propane costs. This has kao wool insulation with a top coat of hard refractory to prevent the flux from eating away at the insulation.
My second forge is more of a traditional horizontal forge such as yours. However I much prefer a blown style of propane burner. It is an old oxygen cylinder lined with Kao - wool insulation.
With your venturi style burner, you will need a "high pressure" propane regulator that will allow adjustments form 0-~25psi. In your picture it looks like you have a BBQ style regulator, most of which are regulated in W.C. and translate to ~1psi max.
Yeah the regulator is off a BBQ, I had it lying around, I thought I had a regulator off an old cabover camper, but I couldn't find it. What is the easiest way to plumb the lp tank to a hose, is there an adapter from the tank threads to a npt?
I prefer the blown style burner as it is much more efficient, especially when using smaller 20lb propane bottles.
I will have to look it up. The smaller BBQ bottles can actually freeze up with a venturi style burner when turned up to welding temps.
You also may want to consider a insulation blanket such as Kao wool or the like on the outside of your firebrick. Firebrick by itself is not to good of an insulator.
Here is a link to a really great bladesmithing site. VERY knowledgeable / friendly / talented people over there.
http://forums.dfoggknives.com/
As well as a good tutorial on vertical forge building with a blown burner.
http://www.dfoggknives.com/forge.htm
I feel like I am registered to too many addicting sites already, please don't tempt me 
As for why your burner does not seem to be very "torch like". You really need to verify the pressure going into the burner. Also what diameter is your orifice?
I drilled it out with a jet drill set to .04". I made it to the specs set out in the link above. I am sure that I am not getting the pressure needed. I suspect you have a lack of pressure going in, or too large of an orifice.
Welcome to the great hobby of knife making. Warning... your friends / neighbors may begin to think you are crazy.
They already do!