XCOM Inspired Fantasy Game

[NancyGold]

If you don't want to use an engine, SDL2 for c++, don't know about c#. If you are fine with using an engine, Unity for c# and unreal for c++

If you don't want to use an engine, there is a high chance you won't achieve anything beside wasting time, and maybe learning a few basic OpenGL things. So I suggest you to use Unity, not Unreal or Cryengine, but Unity since it targets small teams and even single devs. With Unity you just import your assets and write gameplay code, instead of wasting man-years on all unimportant stuff. Moreover, OpenGL is slowly being pushed out by Vulkan, which is even harder to use, especially for newbies, and designed to be a part of an engine, maintained by a team professional graphics programmers.

I personally use SDL2, but that is because the game is 2d and I'm developing my programming language Symta first of all, and as of now it has barebone floating point support (i.e. no unboxed floats). Yet I still plan to integrate with Unity in the future, when the language becomes stable enough. Since I still use a few 3d effects, but software triangle rendering a bit slow, even on modern hardware.

TLDR: go with Unity.

[NancyGold]

Done implementing mine generator for now.

[Thermite]

If you don't want to use an engine, there is a high chance you won't achieve anything beside wasting time, and maybe learning a few basic OpenGL things. So I suggest you to use Unity, not Unreal or Cryengine, but Unity since it targets small teams and even single devs. With Unity you just import your assets and write gameplay code, instead of wasting man-years on all unimportant stuff. Moreover, OpenGL is slowly being pushed out by Vulkan, which is even harder to use, especially for newbies, and designed to be a part of an engine, maintained by a team professional graphics programmers.

I personally use SDL2, but that is because the game is 2d and I'm developing my programming language Symta first of all, and as of now it has barebone floating point support (i.e. no unboxed floats). Yet I still plan to integrate with Unity in the future, when the language becomes stable enough. Since I still use a few 3d effects, but software triangle rendering a bit slow, even on modern hardware.

TLDR: go with Unity.

I wanted to do something in the same graphical view as original XCom (a fixed POV with different layers causing the impression of depth), does Unity works well with 2d games?

[Bobit]

I imagine an isometric view is pretty complicated graphics-wise and unity might not be the best for it. But google unity isometric tutorials and see what you get.

But on r/roguelikedev there's a strong consensus that you don't need complex graphical engines, depending on what your plans are.  Very simple input and output things like bearlibterminal are consistent and powerful (for ASCII games, not isometric). Most of the more complicated stuff needs to be implemented/tweaked yourself anyways.

Just don't try to make something as complex to design as XCOM.

[NancyGold]

I imagine an isometric view is pretty complicated graphics-wise and unity might not be the best for it. But google unity isometric tutorials and see what you get.

But on r/roguelikedev there's a strong consensus that you don't need complex graphical engines, depending on what your plans are.  Very simple input and output things like bearlibterminal are consistent and powerful (for ASCII games, not isometric). Most of the more complicated stuff needs to be implemented/tweaked yourself anyways.

Just don't try to make something as complex to design as XCOM.

Actually, Unity has this thing called hardware Z-buffer and each sprite texel will get a camera Z, which would solve your z-sorting problem without much effort (your will still need to set up a proper orthographic projection).

I don't use Z-buffer, but instead actually sort sprites by a specially crafted Z, and then apply seveal tricks, so larger sprites, like dragon, wont have wings clipping through the ceiling, and multicell sprites, like say a dinning table, would be correctly drawn before the items stand on top of them.

Similar method (but only with neighborhood sorting) was apparently used in Ultima 8, while Magic & Mayhem had some tricky hacks. As a rule of thumb, when isometric game has more than 1 z-layer, expect a lot of ugly hacks, unless you use actual 3d hardware.

[Thermite]

Actually, Unity has this thing called hardware Z-buffer and each sprite texel will get a camera Z, which would solve your z-sorting problem without much effort (your will still need to set up a proper orthographic projection).

I don't use Z-buffer, but instead actually sort sprites by a specially crafted Z, and then apply seveal tricks, so larger sprites, like dragon, wont have wings clipping through the ceiling, and multicell sprites, like say a dinning table, would be correctly drawn before the items stand on top of them.

Similar method (but only with neighborhood sorting) was apparently used in Ultima 8, while Magic & Mayhem had some tricky hacks. As a rule of thumb, when isometric game has more than 1 z-layer, expect a lot of ugly hacks, unless you use actual 3d hardware.

So, to Tl;Dr; it: make a 3D game, with only one view point and a layering component, that it will work out better?

[NancyGold]

So, to Tl;Dr; it: make a 3D game, with only one view point and a layering component, that it will work out better?

Yes.

Consider the following ship sprite (taken from my game). In a proper z-buffered 3d engine importing the ship into the game is as simple as breaking it into elements, assign them proper Zs and just passing to the engine. But in the original XCOM or in say Magic & Mayhem, you will have to break this ship into cubes or similar boxes (together with complex geometry objects, like the rope ladders and the masts with sails). Obviously will still have to devise some way to hide occluding items, even with the true 3d objects. So most 3d strategy games (i.e. Starcraft 2) and RPGs are actually 2d, having just a single layer, to avoid the confusion. XCOM/Dwarf Fortress like games open a pandora box of problems.

[NancyGold]

Was looking for a way to import that ship without breaking it into cubes. There is actually a special algorithm to do that - hierarchical sprite sorting. Basically one has to produce occlusion forest graph, and then instead of calling qsort, just walk the forest from the roots. It works, but it is slower than qsort, since building the occlusion graph is a non-trivial task.

[NancyGold]

Researching further, most of the slow down was the result of empty spaces now now being part of the input for the occlusion algorithm. I'm sure there is a way to optimize the handling of the invisible cells, but it wont be easy.

In addition, any occlusion graph makes it easy to create circular dependency bugs, including rarely occurring ones. And then you will wonder where it happened. Especially nasty problems occur when one large multicell sprite resides on top of another, since there is a large loss of granularity. The obvious way of solve is automagically breaking large objects into isometric tiles.

That is why you use 3d engine even for 2d games.

[Yankes]

Or store accurate depths of each pixel in graphic and final surface. With this draw order will not matter but each pixel daw will take more time.
At some point I will try implementing something like that.

[NancyGold]

Or store accurate depths of each pixel in graphic and final surface. With this draw order will not matter but each pixel daw will take more time.
At some point I will try implementing something like that.

That is called Z-buffer. A thing 3d hardware implements very efficiently.

But in case of tile-based isometric game with oversized sprites, you want to be very careful with Z-Buffer, so objects wont clip through walls. I.e. you want to render your giant units with Z-test turned off, just like you render HUD in FPS games to avoid it clashing with objects on screen. Then there are also transparent surfaces, which still require tricky sorting. Unless you do physically based rendering, where actual light rays travel the scene.

Although games like Warcraft III turned the clipping bug into a feature, providing buildings with foundation, that can be unearthed on uneven terrain.

[NancyGold]

Ok. With some debugging I managed to avoid the circular dependencies (resulting in the locked-in areas not drawn), while still maintaining proper parts breakage. I.e. moving cursor behind the ship will hide its back revealing units behind it. Still a few draw order bugs, which can be easily fixed. Since the algorithm is slow, I will use it only for several special hand crafted areas, like ship boarding and boss battle arenas. At least until I find a way to optimize it.

[Yankes]

That is called Z-buffer. A thing 3d hardware implements very efficiently.

But in case of tile-based isometric game with oversized sprites, you want to be very careful with Z-Buffer, so objects wont clip through walls. I.e. you want to render your giant units with Z-test turned off, just like you render HUD in FPS games to avoid it clashing with objects on screen. Then there are also transparent surfaces, which still require tricky sorting. Unless you do physically based rendering, where actual light rays travel the scene.

Although games like Warcraft III turned the clipping bug into a feature, providing buildings with foundation, that can be unearthed on uneven terrain.

This is correct, if done poorly then big objects will be problem, but if we use tiles then on load we can check if any pixel do not "escape" 3d box where it should be.
Then if it clip then it should clip, this WC3 example is why I think this is very good function why try use this approach. In some special cases I would like if unit clip through other objects. Water, high grass, mud, waterfall, ivy, bushes. magic portals etc.

Transparency is indeed problem, but I think of work around. Instead of multiplicative transparency use additive, with it order in each surface is draw will be irrelevant. First you will draw all not transparent objects, after this transparent.

To have reasonable speed this will probably require manual use of see4 intrinsic because compilers sometimes miss some transformations (same OXCE code have 3x time speed difference based on different GCC version).

[Thermite]

Yes.

Consider the following ship sprite (taken from my game). In a proper z-buffered 3d engine importing the ship into the game is as simple as breaking it into elements, assign them proper Zs and just passing to the engine. But in the original XCOM or in say Magic & Mayhem, you will have to break this ship into cubes or similar boxes (together with complex geometry objects, like the rope ladders and the masts with sails). Obviously will still have to devise some way to hide occluding items, even with the true 3d objects. So most 3d strategy games (i.e. Starcraft 2) and RPGs are actually 2d, having just a single layer, to avoid the confusion. XCOM/Dwarf Fortress like games open a pandora box of problems.

Time to learn Blender then...

[NancyGold]

Time to learn Blender then...

Yeah. Blender or other 3d package can help. You can model even 2d scenes in it, and then export them as a set of objects for your ingame use. But I wont recommend Blender to newbies. It has a rather clunky, cryptic and generally unfriendly UI, mostly relaying on obscure hotkeys. No way you can learn it quickly and make something in a few days. 3ds Max is lot easier if you have access to it. But a room-mate artist where I live uses Cinema 4d, but I heard it is geared more towards TV and Movies, while 3ds Max and Maya were initially developed for gamedev people.