I been working on an yet unreleased "big mod" that has all non UFO graphics that i'm doing custom a lot of stuff including creatures UFO and UFO blocks terrain weapons armors etc...so here is what I learned.
OpenXCom is changing pretty fast. So you can either stay true to 1.0 or go for nightlies. I prefer going for nightlies checking for and installing new versions and always updating my ruleset to fit with latest changes (if any). But that's me. Nightlies are unstable advanced versions released every couple of days usually more features more fresh less stable. Some nightlies may have bugs but they do get sorted out really fast usually by next nightly.
OpenXCom is pretty customizable graphically but functionality/enemies/mechanics are about 90% XCom. There are some neat things added here and there but OpenXCom stays true to the original game (mostly) so do not get any wild ideas. There are a few alternative executable with more features out there for example OpenXCom Extended.
Modding for OpenXCom is pretty easy although a bit time consuming. You'd be spending about half your time writing rulsets and the othe half manipulating graphics.
So you need to learn about OpenXCom rulsets. To do that go into openxcom directory and into Rulsets/XCom1Rulsets and open any of the rul files with a text editor. I recommend Notepad++ because it kinda knows YML and rulesets are written in YML. For ruleset reference go here
https://www.ufopaedia.org/index.php?title=Ruleset_Reference_Nightly_%28OpenXcom%29 for nightlies. There are some examples at the bottom. But most examples are in the game's rulset directory I listed above.
You gonna put all graphic resources in Resources/yourmod folder. Now to learn to do graphics you kinda have to fumble around to figure out a system. I figured out a way using photoshop and act palettes. Remember that XCom uses palettes that is only 256 colors are allowed per game mode. You can make your files into gifs load them in a rulset as extraSpites and add them as any number over 100 as sprite number then use them in your mod. Don't go below 100 for any sprite number because it may cause issues with already existing game sprites. You want new ones.
An even better way to figure this out is get a mod that's very similar to what you want to make then completely change it up. Then you will learn.
Oh and I'm new to modding too. Well newish. From my little time here I kinda learned this forum is pretty friendly and active. People check these forums every days. Some people like to focus on certain things like terrain for ex Hobbes is expert terrain maker. Some are developers. Some reverse engineered the exe figuring out the original game code or AI routines. Some created editors and really neat programs that can make your own maps etc etc. There are a lot of tools out there burried in the foums. Go to Tools section. Also use forum search features such as "map editor" you will find something. Etc. I think some of the lead devs are Warboy and SupSuper . But I think there are thers. On git there are a bunch more devs. I think some may just develop and not even post on forums? So I m not too familiar with the community. But as new modder to another new modder that's my impression so far. In like 3 days I learned so much . Is not hard. Is just time consuming. And is annoying until you figure out the magic way.
Good luck!