Messages

1

Programming / Re: Mods and code changes.

« on: February 18, 2016, 12:28:51 am »

Everything I'm openly planning is contained in the baseview, yes.
But the 32-8 and 8-32 conversion will be useful to have around, even just for reference.

With GPU rendering to 32-bit I don't have to lose info e.g. repeating colours. I've got an untouched 8bits that I could store a reverse-lookup for the CLUT for the entire framebuffer so you'd still know which pixel was which if you ever needed that info.  I can't think of a use case where i'd need it, but you could save it out to VRAM or even CPU addressable memory if you wanted it so the info isn't going to be lost.
More likely I'd bind a separate 8-bit render target for that, and either an RGB-24 for the colour buffer or a 32bit RGBA and use the alpha for transparency. Alpha is a little weird in an 8bit world so I'd been ignoring it.

I chose the baseview because it was easy, rather than starting with the Geoscape or Battlescape. It's self contained and the workload was low. Its a couple of hundred lines of code for the render. The colourspace conversion probably quadruples that, maybe more, but again that stuff could be useful.

If the Baseview was easy, I did have a peek at the geoscape because of the way the ugly way the world texture doesn't rotate with the world. I thought it'd be nice to fix that, but as soon as you start antialiasing you have the same RGBA-to-CLUT problems. Everything I thought of in the geoscape was cosmetic really. I thought about having clouds, waves, that sort of thing but they are just distractions.
Country highlights, clickable countries, and a few other options came to mind but nothing that grabbed me as code I wanted to tackle while I was focused on the Base.

The battlescape is a bit more of a beast. A 24 bit renderer could add transparent smoke, and variable zoom levels which are nice tweaks but not something I was too fussed over. It's too big to think about right now.

I did make a checklist of the work involved in using the BattlescapeGenerator to output a base defence mission as a vertex buffer object and then use a mesh of the base defence layout as the map screen in the BaseView. It'd be an interesting side project, generating a mesh isn't too hard, and you get nice pan, zoom, rotate on the BaseView map. It'd look cool and you'd be able to preview what the base defence would look like which might help some players, modders, or specific scenarios people came up with.
It was an interesting thought experiment, but there are a few problems to solve if you autogenerated meshes and I'm not sure I'd want to tackle those head on until I was a lot more familiar with the project.

2

Programming / Re: Mods and code changes.

« on: February 17, 2016, 07:45:12 pm »

I don't think this topic
https://openxcom.org/forum/index.php/topic,896.0.html
Is relevant to what I'm saying.

It looks like they are talking about solving a different problem, with a different solution.  I've not looked at the battlescape renderer at all yet so I'm not in a very strong position to comment - but based on what was said in the thread it doesn't look relevant at all.

I'm not doing much more than reimplementing the SDL_Surface blit on the GPU with a 32-bit internal pipeline, so not changing colours, assets or shading.

3

Programming / Re: Mods and code changes.

« on: February 17, 2016, 07:26:12 pm »

Thanks for the reply -
I *have* just joined and figured adding scroll/zoom to the baseview would be a simple gameplay feature to wet my feet with rather than diving into something big. I can't easily do that using a blit on an 8-bit surface.

The next option is CLUT-to-RGBA, draw the basescape, and then RGBA-to-CLUT to output to the existing SDL_Surface. I figured that sort of thing is worth posting up because somebody might have already thought about it, already done it, or have another future use for it.
This looks like the minimum effort to get extended bases and base scrolling working. FWIW I did consider generating smaller zoom level versions of every tile in the base view, and using those with the indexed SDL blitter. This is exactly how we would have done it in the 90s and still an option, but it's a crappy option considering the asset cost vs having the colour transform code written and available for other things.

I don't much fancy reimplementing *everything* into 32-bit, and while I jumped the gun saying there isn't a good reason not to [especially so soon after joining], it would solve a lot of problems and provide a lot of flexibility.  The point of posting it here was to get feedback and awareness. I'd rather have replies than silence. Bottom line, I can't think of a technical reason not to change. You could start with pixel-for-pixel the same game, but would have options for improving the Base, Geo or Battle screens.

The GL_EXT_paletted_texture isn't worth it, and it solves a different problem really. And by the look of it isn't widely supported anyway. The process I described needs an RGB surface of any precision, and an 8-bit unsigned buffer for output - I'm literally performing the CLUT/RGBA conversions in GPU hardware using the same technique you'd do on the CPU ... just a lot faster.

I'd need programmable shaders, so I guess thats hardware from 2003 onwards. While this doesn't add a significant increase to the minimum spec, it does raise the bar into hardware rendering and will count out some devices.
Mobiles before OpenGL ES1.2 probably won't work, but that's all the way back to gingerbread on android support earlier than that was sketchy but the pre-gingerbread market penetration is tiny. If iOS is targeted it'd be fine on a 3G or higher. PCs from the Geforce FX/5 series onwards.
The system spec argument is a strong reason not to support new hardware, but it becomes a decision on *where* to draw the line since there is a line somewhere. It's also the sort of thing I'd be happier to discuss after the fact - get the feature working and then evaluate what the minimum spec is and if it's worth the tradeoff.

If hardware support was an issue for machines from the late-90's era, we could add a forward renderer using the OpenGL fixed function pipeline and/or software implementation. It's more work, and probably means cooking the assets into a file format for that platform... I'd rather just stick a minimum spec on the modpack but there are compromises we could make I'm sure. This would get us back to early Geforce 3, maybe earlier - TNTs even.
And even then, you could write a separate software renderer using shrunk assets for the bottom end platform(s), if it really came to the crunch and extended bases was deemed important enough to need to work on every platform.

I'm a gameplay guy, but I've had a little render/GPU experience and this looks easy enough and is on the critical path for the gameplay feature I'm focusing on (extending the baseview/basescape) that it was worth approaching as one feature.

My last concern is deviation from the source material. I enjoyed UFO a lot and too much change - even a different look and feel would probably put me off. It'd have to be a definite upgrade/improvement to really justify changing what is there in the core game. I think Mods and TCs might get more use out of extended bases.

Cheers, it's a lot to think about. I'll be able to hit the keyboard over the weekend with any luck.

4

Programming / Re: Mods and code changes.

« on: February 17, 2016, 01:29:55 pm »

Sorry if I came off as naive, I'm pretty sure from a technical perspective the 8-to-32 and 32-to-8 colourspace conversion is easy on the GPU.

What I mean by RGB-to-CLUT is this:
A far as I can tell an RGB to CLUT fragment shader is reasonably simple. It's a reverse lookup which is extra work, but GPU time is cheap.
Pass in the colour lookup table to the shader as a uniform array of vec with 256 elements. You need to supersample the frame buffer to get a colour area around each pixel. You could used fixed pattern or random sampling depending on how you wanted to dither. Then you can iterate the fragment and pick an indexed colour, again there are a couple of popular options here. The result is a 32bit source image rendered to an 8-bit framebuffer with an 8bit palette.  OpenGL looks like it has a few eight bit pixel formats and the 8-bit unsigned one should be the same as the SDL_Surface pixel data.

You can pass in a second uniform parameter containing a second output palette if you want to support palette cycling. It will likely produce some artifacts if you don't mask out non-cycling areas of the original palette and are doing complex shading, but I wouldn't expect nearest-neighbour sampling to have any problems which would be a good starting point and retain compatibility with all the existing assets.
I used RGB-CLUT as a shorthand, (for the operation performed as a postprocess in the fragment shader) sorry if I didn't make that clear.

I've not written the code yet so I've not worked through all the problems, but that process can be used to output a 32-bit render pipeline onto an 8-bit render target. With the option of reading 8 or 32 bit source assets which is good for content creators.
It's the same process that photoshop goes through to palletize an image, except you can do it in milliseconds on the GPU. I'm sorry if this comes across as having no idea what I'm talking about, I am still getting to grips with the SDL and OpenGL terminology.

8-bit to RGB
Likewise going the other way is trivial since you can use the lookup table, and rendering an 8-bit source to a 32-bit framebuffer just needs the same CLUT data. You'd create a texture buffer with an 8-bit unsigned pixelformat and pass the CLUT as an uniform vec array to the fragment shader, outputting to a 32-bit RGBA target.
You could convert every asset as it loads, saving it in a 32 bit surface or more likely just pushing it to VRAM to use later. (more on this later, you'd be using the gl ETX methods from SDL_Opengl.h so I'd want to check what the compatibility is on those)

After that you can perform all render steps in 32Bit, and present back to an indexed palette using a the post-process fragment shader above.

The shading differences between RGB and CLUT are irrelevant if you maintain 8-bit source images and render to an 8-bit palette. With nearest neighbour filtering the first iteration would look pixel-perfect to the original.
But doing so opens up the option of additional effects e.g. scale/rotate/shear that just aren't possible (read impractical and slow) with 8-bit indexed surfaces. My example being the baseview, where a 32-bit render pipeline would allow easy scroll/scale to support extended base sizes and configurations.

It's also likely to be faster to draw to use 32-bit on a lot of modern platforms. Speed isn't an issue asn OpenXCom isn't really bottlenecked, but keeping things fast is always good.

I'm aware of the colour cycling that XCom uses, but it's not too difficult to emulate the effects in native 32bit. XCom seems to rarely, if ever, rely on complex cycling patterns. There are a few flashes and simple effects that are as trivial in 32bit as they are with a colour look up table.
Given that you can still cycle palettes with a 32-bit render pipeline as you are rendering to a colour look up table, palette cycling could still be supported. My opening gambit is the baseview, where complex palette cycling isn't required and that will provide a platform to iron out the problems.

Given memory capacity has increased by several orders of magnitude since the mid-90s, wanting images to look different on different screens is likely to be irrelevant as you can have a duplicate of each image customised for each context.
But given I think I can still colour cycle with a 32-bit render pipeline during the CLUT post-process step, we would have the option of multiple assets, or single assets with cook-time, load-time or runtime recolourisation. You've got the option of using original source assets but could load new 32-bit assets for new content without breaking the pipeline.

Again, this is all speculative - I've not written the code yet.
Given you can do palette cycling, and a number of other effects, in the fragment shader and we have essentially no memory limit (compared to the mid-90s) There doesn't seem to be a (technical) advantage to the eight bit render pipeline, and some reasons not to.
32 bit rendering adds up more options in the baseview, you could improve the geoscape a lot, and there are loads of battlescape options - smoke/fog for example, and the ability to have a lot more assets on screen - as well as zoom levels could be tried.

Thanks for the link to the topic about shading discussions. I think there are a couple of good points raised there and while they are mostly concerned with the battlescape, a lot of the same things apply. I'll consider those and see if they can be addressed.

Other problems I can see are hardware compatibility. The SDL_OpenGL header file looks like some of the framebuffer stuff uses the EXT methods which are old. You might need two versions of the renderer to support both generations of hardware, although TBH I suspect there isn't a target platform that doesn't support all the features we'd need but I just don't know.

I'm happy to have a discussion on any level about the render pipeline changes and requirements, and have had a little experience both with index'd palettes and modern GPU programming.

5

Programming / Re: Mods and code changes.

« on: February 17, 2016, 02:27:28 am »

Cool
Doing everything in 32 bit is going to be easier.

If AndO3131 has made the switch to 32-bit, then i'll grab and merge. It'll save me some time and generally make everything easier than wrapping everything in a 32-to-8 and 8-to-32 pipeline.

I'd probably recommend that the project switched to 32bit though for all assets and internals though. You can render with a 256 CLUT post process to capture a retro look and feel, but I can't imagine a reason to work in 8bit.

6

Programming / Re: Mods and code changes.

« on: February 16, 2016, 07:02:30 pm »

Yeah... The 8-bit thing is a real pain.

There is an 8-bit option in OpenGL EXT that will probably work, but I might want to render everything to an RGB A framebuffer than then out to an 8Bit image after all compositing is done. I've never tried it but the API documentation should be sufficient.

If not then I'll have to write a supersampling fragment shader that reduces RGBA-to-CLUT. It's not rocket surgery but will do the trick. There are probably algorithms online for dither patterns and stuff so we can pick one that matches the art style as best as possible.

It's also just possible that I can instantiate 32bit buffers for the Basescape, and neatly sidestep this sort of thing. That might be really worth considering, although it will make the basecape stand out aesthetically.

I'm planning to keep the ambition and scope down by not adding any features - just reimplementing the renderer and supporting scroll/zoom. That should make it really easy to focus on the task at hand and then all the other stuff - overground, building restrictions, national perks, roads, corridors, garages, cars, civilian buildings, police, hospital, firing range ... etc... can all wait and be done in their own separate iterations.
My goal here is just to provide support for them, not implement all the new stuff in one go, so that I can write an "Extended Bases" mod one building at a time.

7

Programming / Re: Mods and code changes.

« on: February 16, 2016, 11:56:50 am »

Hey Solarius
Sorry for the delay. I had a couple of problems with lib dependencies.  All sorted now and I started a new BaseView renderer.

In my first hack I'm using a subclass of BaseView so I can substitute it in with a one-line change in BasescapeState, and I'll revisit the interface and maybe use a factory constructor or something further down the line. The architecture will be apparent once the implementation is done.

As an interim update,
The SDL_Surface blit doesn't support scale (its crop only) so in order to get variable base sizes, scroll, zoom and multiple floor support into the Basescape view I'm going to bind the surface to an OpenGL render context and reimplement the blit to draw quads.
It's more involved than I'd expected but it shouldn't be too much work, gains tons of flexibility, and we should be able to query the system and push a texture atlas to VRAM which will reduce render time.

Longer term I'd want to refactor that to vertex buffer objects instead of manually drawing quads. It'll drop the render time to basically nothing push everything onto the GPU. I think thats all in the spec for OpenGL1.2 and 1.2ES so should be supported on every target platform, but I can confirm that and we should be able to detect for hardware support and add a toggle for it all in the options screen anyway.

It'll be a couple of days before I'm back at keyboard, so I'll think it over a bit more and try and throw something together when I can.

8

Programming / Re: #include <algorithm> usage

« on: February 16, 2016, 11:31:08 am »

Thanks!

I got boost and yaml-cpp to build, but following an unresolved external in yaml::node while linking OpenXcom I decided to download the prebuilt libs.
I'm building for Win32.

I kind of feel like I've failed at the first hurdle, but at least I can write code and run the game which is all I was after. My Algorithm #include was caused by a mismatch in my yaml-cpp version.

Either way, I'm up and running. I didn't get time to write any features yet but I'll post a separate thread.
Cheers.

9

Programming / Re: #include <algorithm> usage

« on: February 14, 2016, 12:46:40 pm »

Thanks for the encouragement.

If SDL 1.2.* is sufficient then I should be fine then. I'll read the SDL2 specifications and check there is nothing there I'm going to need, as far as I know I'll be fine. The 1.2 spec looks like it has enough input, timing, and OpenGL is already an option so I'll be fine on that side.

FYI - That build instructions link doesn't open for me. I just get a database error for everything from https://www.ufopaedia.org/
My problem is Yaml-CPPs CMake is missing a path to my boost libs - I just need to learn where to set BOOST_ROOT and I could be golden.

10

Programming / Re: #include <algorithm> usage

« on: February 13, 2016, 11:24:49 pm »

Aces thanks.
cmake 0.5.3 gives me boost errors, so I'll work through those and carry on trying. I can't be too far off.

11

Programming / Re: #include <algorithm> usage

« on: February 13, 2016, 10:58:45 pm »

OK
I must have missed the instructions
I grabbed from here
https://github.com/jbeder/yaml-cpp

Is there a fixed version for YAML, and for SDL, SDL_mixer, SDL_image and SDL_gfx I should be using?
Once I got all those built, I get an YAML unresolved external, so I was pretty sure I had the wrong version and/or a header/library mismatch.

12

Programming / #include <algorithm> usage

« on: February 13, 2016, 04:33:23 pm »

Guys, I've had to include <algorithm> to a bunch of files for min, max, transform, foreach.
Have I missed a trick?
How does this work for anybody else?

13

Programming / Re: Mods and code changes.

« on: February 12, 2016, 06:58:49 pm »

If you want bases on two floors ... there is more work involved.
I don't know if you'd need BasescapeState to have to BaseView objects of if you would repopulate a single BaseView object when the floor changed. I'd have to look at both to comment on which is more suitable. I like the idea of displaying several floors in a single BaseView object because of encapsulation.

Either way, the Base will have to count both floors for Base Defence missions, for iterating facilities, for BaseInfo, and for costs too. Base also hardcodes the base size, and BaseFacility only stores X,Y coords. You'd also have to come up with a solution for the savegame... All in all There are quite a lot of places you'll have to touch to get it working but it'd be a nice touch to have more base flexibility - especially for other mod authors. I only downloaded the project last night and was drunk at the time so that probably isn't an exhaustive list of what you'd have to change.

You also might want to restrict some facilities i.e. Hangers and Radar on the above ground floor only. Alien Containment must be underground... everything else can be freely placed. That sort of thing. This will need additional parameters for facilities.

14

Programming / Re: Mods and code changes.

« on: February 12, 2016, 04:10:51 pm »

OK, I'll think that through.
You are right - I do actually want to preserve BaseView to ensure integrating changes is easier. Mostly this is so that I can support an alternate render option at runtime.

I think my philosophy is that it should be divided into interface and implementation in the main development. That would let me make any changes without preventing integration from the main development.
"open to extension and closed to modification" style.
But that's probably a larger discussion.

I'll see if I get time to sketch something out and post back what I find.

15

Programming / Mods and code changes.

« on: February 12, 2016, 03:20:12 pm »

Hi guys.
I'm interested in making some mods, and it'll need some code changes to support them.

There are easy things like adding buildable corridors in the basescape, and adding some new rooms and vehicles, but actually I'm thinking about replacing the basescape renderer first.

I downloaded the source last night and Codewise it's not too big a task. Its an alternate BaseView class. I'm not sure if BasescapeState will have to change, beyond switching to instantiate a particular BaseView - The discussion on if this should be done in the CTOR, by injection or by factory injection is irrelevant right now - I'm just trying to learn about the project to start with.

What is the process for a change like this?
e.g. If I made BaseView an abstract class and moved the current implementation into a derived BaseViewReference class and then derived my own BaseView subclass with alternate rendering - is this the sort of change I submit and post an FYI thread about or is it the sort of thing I should start a discussion thread about first ?

The more flexible base renderer will help me support mods I'm thinking of - Garages & LandRangers, larger bases, overground bases, bases with unbuildable areas around them - fields, caves, coastlines, or civilian structures.