I think (I'm not at a games-capable computer ATM) that it actually does both in the battlescape. if an operation that CAN be canceled is active, it cancels, but if there's no such operation, it opens the options (and even closes them again), so no problem there.
I'm just asking for the same functionality in the GEOSCAPE.