I suspect that 18 gauge (or even smaller) wire would be fine. There is not much current that needs to flow for a switched signal to drive an alarm. For reference, most doorbells are on 18 gauge wire. A more key spec for the alarm is the current draw (and whether that all flows through the high limit switch or not). It's probably in the well under an Amp range, which would also push me towards 18 gauge being "plenty" electrically.
You probably need to be more concerned with physical issues, such as "is the insulation and overall physical strength suitable for local conditions?" [whether direct buried, in conduit, run across the lawn, etc].