Tried Open Broadcaster Software's "Game Capture" mode and it gave me a blank screen.
If you want to use game capture mode in OBS, you need to use OpenGL. In OpenXcom, go to Options > Video > Display Filter ... and change it from "Disabled" to "raw*".
PS: just to clarify, OpenXcom doesn't have any special requirements, you can use any recording software you want (I am using Bandicam for example)... just need to configure and use it properly.
PS2: don't use Fraps though, that's the worst piece of software ever