First off, your tool is hanging out too far. (Lord I'd like to be told that on a daily basis) Shorten up the tool so you have just enough to reach your depth needed on your part. Second...straighten up the tool and toolholder. Clock it around counter-clockwise so you are just a tick more than your bar being parallel to your part. That doesn't sound right.
Set your boring bar parallel to the outside of your part, then move it counter-clockwise, maybe one or two degrees. The way the boring bar is made, even setting parallel with the boring bar, neither cutting surface of the insert will be parallel. You will have a slight angle each way.
Once you have the toolholder and boring bar set, then raise the boring bar up above centerline somewhat. Then set the boring bar so that it just reaches the depth that you have to go, plus a few thousandths clearance so the toolholder doesn't hit your part.
It's hard to tell by the pic, but if you JUST HAPPEN to have a diamond hone, dress the point of the tool so you have a couple thousandths radius if you are using a sharp, zero radius insert.
Add a little oil, and by all right, you should get a good finish. From what I can see of the pic, it just appears the tool is out too far. Plus the more of an angle you have, the further the tool has to stick out. The loser it is to parallel to the part, the shorter it will be.