Also seems palettes use 3 bytes instead of 4. How do I calculate what color am I dealing with from 3 bytes to 4?
Alpha channel is ignored by OpenXcom. First byte is red part divided by 4, second byte is green part divided by 4, third byte is blue part divided by 4 (i hope i didn't mess color order).
Also I noticed some things are hardcoded for ex menus in XCOM1/XCOM2 use certain colors. Where is this menu color area and is it constant between the two games?
Menus are available in rulesets:
UFO interface and
TFTD interface.
Palette has maximum 256 colors. I have a strong feeling it's impossible to merge UFO and TFTD palettes into single one and expect it to display consistent image, because various element from first game will use the same color index as elements from second game, but expecting different color of this exact element. For example look at gazer aliens from both games. They use the same index, but they look very different. Overwriting colors in TFTD palette will result in some unit or texture looking weird.