Statistics soldiers

[Ethereal]

Loads and save for a minute with a tail. The save file is 4 604 010 bytes in size.

Comrades, there is an offer.
To reduce the time of loading / saving, divide the save file into several parts, like in the original UFO. What would each soldier and his statistics and equipment have a separate file, which is stored in a folder with the same name as the main save file. I think this would solve the problem. Correct me, if I'm wrong.

[Ethereal]

I again about statistics. In the file with 6 megabytes of storage, the statistics takes 4 megabytes. Comrades programmers, is not it time to remake the statistics of soldiers, making it less voluminous?

[Stoddard]

I again about statistics. In the file with 6 megabytes of storage, the statistics takes 4 megabytes. Comrades programmers, is not it time to remake the statistics of soldiers, making it less voluminous?

Comrade valued customer, can you please upload such a bloated savegame? And also please state which build of the game do you use.

This way at least I can get some hard metrics on time spent parsing/loading the save game file versus other code and get some insights from that.

[Meridian]

Comrade valued customer, can you please upload such a bloated savegame? And also please state which build of the game do you use.

This way at least I can get some hard metrics on time spent parsing/loading the save game file versus other code and get some insights from that.

He already did, for example here, using latest OXCE+ (and his mod): https://openxcom.org/forum/index.php/topic,2915.msg95193.html#msg95193

He says it takes minutes to load... but for everyone else it takes only 3-5 seconds.

I believe him... but to be honest I'm not going to care too much, if it's just wrong on his side and everyone else is OK.

Still, would be nice if you could do some statistics/analysis on it...

[Stoddard]

He already did, for example here, using latest OXCE+ (and his mod): https://openxcom.org/forum/index.php/topic,2915.msg95193.html#msg95193

He says it takes minutes to load... but for everyone else it takes only 3-5 seconds.

I believe him... but to be honest I'm not going to care too much, if it's just wrong on his side and everyone else is OK.

Still, would be nice if you could do some statistics/analysis on it...

Minutes.. heh. Something must be horribly wrong with the hardware. Parsing the entire 5-whatever-MB savegame file takes on the order of milliseconds on a 10-year-old CPU, and even if we postulate that populating the varios std::vectors and stuff takes 10x time, it's still way below a second.

@Ethereal, comrade, can you please share what's the OS and hardware you're running this on? HDD SMART dump would be most interesting.

[Ethereal]

XP SP3. Equipment is old. Recently replaced the hard drive and loaded much faster, but the problem is different. 6 MB get in the middle of the game. That in the end will be - it is terrible to imagine. That's what worries me. The file will continue to gain weight and can grow to monstrous proportions.

[Stoddard]

XP SP3. Equipment is old. Recently replaced the hard drive and loaded much faster, but the problem is different. 6 MB get in the middle of the game. That in the end will be - it is terrible to imagine.

Okay, so it looks like an I/O problem.  Let's try to reproduce it.

The oldest and slowest drive I have is Seagate ST380817AS, 80Gb from 2005.

Aaand... loading the save you posted took all of about 3 seconds.
Since repeated loading of the saved game took about the same time, it's not I/O bound at all.

So, pray tell what's the actual hardware you use. CPU, MB and HDD models, what kind of RAM and how much you've got?

[Ethereal]

What about my system? I told that I replaced the broken hard drive and loaded faster. But, the save file from this increases in size does not stop. That's the main problem. And I wonder when it will reach 50 + megabytes game will be able to load it at all?

[Meridian]

What about my system? I told that I replaced the broken hard drive and loaded faster. But, the save file from this increases in size does not stop. That's the main problem. And I wonder when it will reach 50 + megabytes game will be able to load it at all?

For comparison, an end-game piratez save is about 15 MB.

And yes, the game should not have a problem with loading a save of any size.

[Stoddard]

What about my system? I told that I replaced the broken hard drive and loaded faster. But, the save file from this increases in size does not stop. That's the main problem. And I wonder when it will reach 50 + megabytes game will be able to load it at all?

The first step to fix any problem is to reproduce it. So far everyone failed.

I am absolutely sure a 60+Mb save file will load with no problems in under twenty seconds even on my crappy windows-for-testing system on that ancient HDD. You talk about minutes. No one can reproduce this. So we ask for additional information. Hell yeah, I even have the original MB+CPU+RAM combo laying around from the time I bought that HDD (2005, remember). It might even still work. But before I dust it off and assemble a test system around that I need to know if the problem isn't somewhere else, and that I'm not wasting my time. So please provide some assistance in solving your problem.

Now if you don't want to work towards solving the problem, that's fine. But then please say so out loud.

[Meridian]

He said a few posts above it loads much faster now after replacing HDD.
(don't know how fast that is)

[Stoddard]

He said a few posts above it loads much faster now after replacing HDD.
(don't know how fast that is)

Yeah, I saw that. But still if he's not disclosing wtf is his hardware, there isn't anything we can help him with.

Latest MXE builds, 32bit, 64bit with the latest YAML and LTO enabled might help a bit, but not much.

I'm not going to code support for zlib-compressed savegames on no evidence of it being a bottleneck at all.

EDIT: check out the PR

[Ethereal]

Yeah, I saw that. But still if he's not disclosing wtf is his hardware, there isn't anything we can help him with.

Latest MXE builds, 32bit, 64bit with the latest YAML and LTO enabled might help a bit, but not much.

I'm not going to code support for zlib-compressed savegames on no evidence of it being a bottleneck at all.

EDIT: check out the PR

CPU 2.93 GHz, 3 GB RAM, 32bit XP SP3. YAML has not updated for a long time (since 2017 approximately). That such "LTO" and at all not know.

[Stoddard]

CPU 2.93 GHz, 3 GB RAM, 32bit XP SP3. YAML has not updated for a long time (since 2017 approximately). That such "LTO" and at all not know.

Great, that isn't that far from my current test system - 2.0Ghz, 2Gb RAM, but win7-64 and a Q8400.

Please try this build Extended+-3.10a-2d7397146-2018-06-30-win32-lto.7z

Just get the non-debug .exe out of it, copy it into your game folder and see if it's in any way faster.

EDIT: LTO is link-time optimization. I very much doubt it'll make any difference for OXCE+, but still.

EDIT2: in before I fire up the 1.8Ghz Athlon 64 system I mentioned before, can you please share the make and model of your CPU so that we can establish a perfomance baseline? Because maybe a Celeron D at 2.66Ghz would make a better fit.

[Ethereal]

Great, that isn't that far from my current test system - 2.0Ghz, 2Gb RAM, but win7-64 and a Q8400.

Please try this build Extended+-3.10a-2d7397146-2018-06-30-win32-lto.7z

Just get the non-debug .exe out of it, copy it into your game folder and see if it's in any way faster.

EDIT: LTO is link-time optimization. I very much doubt it'll make any difference for OXCE+, but still.

EDIT2: in before I fire up the 1.8Ghz Athlon 64 system I mentioned before, can you please share the make and model of your CPU so that we can establish a perfomance baseline? Because maybe a Celeron D at 2.66Ghz would make a better fit.

Tried both ."exe " - no differences in load speed were found. The speed is the same as the standard OXCE+ latest version.

CPU Intel Core E7500 so what will... I have on this CPU World of Tanks and a Perfect World - just fly like swallows. No brakes, no lags, no traffic lights. And for openxcom it may lack performance? Not believe!