I just tested Dxtory with Openxcom latest, while using the 'Openxcom' OpenGL filter. It seems to work really well, besides the frame counter being garbled, just a minor thing. I haven't used other capture software, but Dxtory has worked extremely well for me with plenty of games. The demo version is generous, has tons of options available to play with and the registered version is way more than worth the price!
Check it out here : Dxtory :
https://dxtory.com/v2-download-en.htmlI use Lagarith lossless codec, :
https://lags.leetcode.net/codec.htmlit uses much less CPU than xvid (important for other 3d games), much more disk space, but less than Dxtory's own codec which is also lossless and has several settings. I use Lagarith YV12 mode which is practically lossless but uses less disk space than RGB options. Then I convert and do basic editing in windows movie maker. Windows movie maker's WMV format (lossy) reduces the movie size to a fraction of lagarith's. There are probably newer and better formats but I don't have the software for that. WMV looks very good with a limited amount of colo(u)r banding in situations where there is a very wide range of colo(u)rs used in a single frame. It doesn't have 'blockyness' problems , which is an allowance for its filesizes since they are much bigger than xvid's maximum quality. So the codecs that take more CPU might be better suited for openxcom which doesn't need a lot of CPU power. Your experience may vary, I haven't tried any other codecs yet.
All of this keeps in mind that I am by no means an expert on this stuff.
Get the legit Dxtory, for the sake of the programmer who put so much sweat and tears into it. They have updates pretty often too.