Good idea. The top that Gary linked rolls up to tennis ball size and will fit under the seat. A cheap cycling version sounds a like a plan. Probably rarely use it but knowing it's there if conditions change could make a big difference to your day's riding.
You can get them with and without sleeves... perfect to have always in the pocket for when you rairly if ever need...but are then super happy for that one or two times a season when the cold cuts in.
GOREWEAR are closing down so their stuff is hugely discounted. Picked up really thin wind stopper sleeveless top to wear under my Knox mesh jacket on cold early summer mornings.