Attach a length of 1 x 4 or etc. to two studs, screw a couple of hooks into the board to hang the ladder. Or, couple of nails, whatever. The ends of the board can extend past the studs if you want the ladder placed in a spot away from where the studs are located. With a 4 ft. Ladder, not much weight involved.
That's a great simple idea...can't believe i didn't think about that. i was thinking more along the lines of a great hook that somehow anchored into the drywall strong enough to hold it but your idea would work fine