[SOLVED] Yaml Multiline with (double) quotes crashes the game on Win

[Buscher]

Only affects Windows, doesn't happen on Linux (Xubuntu)

Running Extended-8.0.0-2e70fa97b-2024-12-27-win64 a multiline set up with (double) quotes will crash the game.
Our member Mercury reported it worked fine previously on Windows builds for the release candidate for rapidyaml builds (self-built). They were using a rapidyaml branch for testing 8.0.

Code: [Select]

  STR_GENERAL_STORES_UFOPEDIA: "All equipment,
    weapons systems, munitions, recovered material and Heavy Weapons Platforms are placed in stores, including equipment assigned to craft in hangars."

It works fine with a block scalar (both literal | and folded >)

Code: [Select]

  STR_GENERAL_STORES_UFOPEDIA: >  # or alternatively |
    All equipment,
    weapons systems, munitions, recovered material and Heavy Weapons Platforms are placed in stores, including equipment assigned to craft in hangars.

We (ROSIGMA) have changed from double quotes to block scalar for now as a stop gap measure.

[Meridian]

It works fine when run from Visual Studio on Windows.

Only affects the MXE cross-build.

[Delian]

I can reproduce this on windows, if I run the 64bit release build.

However, I've gone through the code step by step, and failed to find a place in code where it could fail. It doesn't seem like there is any OS-dependent code while processing a quoted scalar.

Might have something to do with x64. Where can I get precompiled 64bit libs, so I could build x64 in VS? :x

[Meridian]

I can reproduce this on windows, if I run the 64bit release build.

However, I've gone through the code step by step, and failed to find a place in code where it could fail. It doesn't seem like there is any OS-dependent code while processing a quoted scalar.

Same happens on the 32bit build for me.

Code: [Select]

[28-12-2024_14-13-29] [FATAL] Error loading file 'c:/Users/meridian/Downloads/oxce800/Extended-8.0.0-2e70fa97b-2024-12-27-win32/user/mods/multiline_crasher.zip/multiline_crasher/Language/en-US.yml'
[28-12-2024_14-13-29] [ERROR] Rapidyaml ERROR: reached end of file looking for closing quote


Might have something to do with x64. Where can I get precompiled 64bit libs, so I could build x64 in VS? :x

SupSuper has provided precompiled 64bit libs to Xilmi somewhere... ah, here: https://openxcom.org/forum/index.php?topic=11660.msg160324#msg160324

[Delian]

Thanks. Ok so, it does crash for me if I run the 32bit build from here, or 64bit build from here

But doesn't crash if I compile it myself. Neither debug, nor release, neither 32bit nor 64bit if I build them myself in VS.

Whoever built these 32/64 bit versions, can they build a debug build?

Edit: Wait.. the "Test-Extended-8.0.0-ab0bd4040-2024-12-27-win32.7z" zip says "win32", but it's actually 64bit version inside lol. Where's the 32bit build that crashes?

[Yankes]

if I would guess I would bets on `\r\n` vs `\n`, as some confusion what version should happens and what was available. Btw is legal for yaml to have real new lines in double quotes?

[Delian]

Yes, the standard says that new lines in double quoted scalars are fine.

Debugging assembler is hard, but I'm slowly making progress lol

[Delian]

I think what's going on is, the compiler optimized
this line of code
in such a way that in subsequent loops it's not executing. However, it's critical that that line executes on every loop, because it contains the string view to the current line of text in document that's being processed.

I think removing the const (or making it const reference) would prevent this optimization from happening and fix the error.

Basically, that loop should be moving through document line by line until it finds the closing quote. But what I'm seeing in assembler is that the search keeps happening in one and the same document line (and somehow not crashing). So what I need is the MXE build with the const removed to see if it still crashes.

[Meridian]

Commit: https://github.com/MeridianOXC/OpenXcom/commit/39501f356465010bfaa3052fbcc30b30cfba5e87

Build: http://108.61.157.159/versions/win/Test-Extended-8.0.0-39501f356-2024-12-27-win32.7z

Doesn't seem to be solved yet.

If I should try also const reference, please paste here exactly how the new code should look like.

[Yankes]

This is wrong function, its handle `'` you need make similar fix in second function that handle `"`

[Delian]

Well, I tried with single quotes, and it still failed.

In ParseEngine<EventHandler>::_scan_scalar_squot()
and ParseEngine<EventHandler>::_scan_scalar_dquot()

try changing
const csubstr line = m_evt_handler->m_curr->line_contents.rem;
to
const csubstr& line = m_evt_handler->m_curr->line_contents.rem;

If that doesn't work, then... a .pdb would be helpful, so I wouldn't need to beat myself with assembler.

[Meridian]

Well, I tried with single quotes, and it still failed.

In ParseEngine<EventHandler>::_scan_scalar_squot()
and ParseEngine<EventHandler>::_scan_scalar_dquot()

try changing
const csubstr line = m_evt_handler->m_curr->line_contents.rem;
to
const csubstr& line = m_evt_handler->m_curr->line_contents.rem;

Commit: https://github.com/MeridianOXC/OpenXcom/commit/f610f9f415b5a0d546088ac6dca7345f9fe2e8a4
Build: http://108.61.157.159/versions/win/Test-Extended-8.0.0-f610f9f41-2024-12-27-win32.7z
Unfortunately still no luck.

If that doesn't work, then... a .pdb would be helpful, so I wouldn't need to beat myself with assembler.

@Yankes: can you provide that?

[Delian]

Well, that variable is still being optimized away in a weird way...
Maybe the macro is causing the compiler to misbehave, so try removing the line that reads:
_c4dbgpf("scanning double quoted scalar @ line[{}]:  line='{}'", m_evt_handler->m_curr->pos.line, line);

And if that doesn't work, then try changing:
const csubstr line = m_evt_handler->m_curr->line_contents.rem;
into
substr line{};
line.str = m_evt_handler->m_curr->line_contents.rem.str;
line.len = m_evt_handler->m_curr->line_contents.rem.len;

[Yankes]

Commit: https://github.com/MeridianOXC/OpenXcom/commit/f610f9f415b5a0d546088ac6dca7345f9fe2e8a4
Build: http://108.61.157.159/versions/win/Test-Extended-8.0.0-f610f9f41-2024-12-27-win32.7z
Unfortunately still no luck.

@Yankes: can you provide that?

I can't provide `.pdb` as this is MSVC specific file, only thing I can add is symbols for DWARF. overall I can try check it under cygwin but what version of 40k exactly crash? I have some older version that do not have fix and fail on comments.

[Delian]

You don't need to run any mod. Just put the contents from OP's zip file
multiline_crasher.zip/.../en-US.yml
into
standard/anything/metadata.yml

Technically, you could also just run this code anywhere:

Code: [Select]

YAML::YamlString yaml(
"#test\n"
"\n"
"  STR_GENERAL_STORES_UFOPEDIA: \"All equipment,\n"
"    weapons systems, munitions, recovered material and Heavy Weapons Platforms are placed in stores, including equipment assigned to craft in hangars.\"");
YAML::YamlRootNodeReader reader(yaml, "");