Men have it rough with no real choice of switching from winter wear to summer wear other than with lighter wools.
Women however, have the opposite problem. There are too many choices and so women can appear too casual or even sloppy. Thankfully, we have skirts that allow some airiness. Furthermore, we have the option of wearing short sleeve or three-quarter length sleeve blazers that allows for further breeziness.
Air conditioning is usually quite high in most office buildings so the only time that a person could be affected is to and from work and lunch time. It is perfectly find to take off the blazer to reveal a summer friendly blouse or tank top during these designated off times.
It can be a slippery slope to casual and inappropriate in the summer time. Just remember that the dress code doesn’t change because its summer and so you shouldn’t either.