Porting codebases to 64 bit

[JasonX]

I started cleaning up and fixing some old engine ports i made, all based on sdlquake, on a 64 bit linux box. I'm not able to run my ports anymore, since i'm getting lot's of segfaults. When loading a new game for example, i get a segfault on the usage of the strcmp inside PR_ExecuteProgram. There are other crashes and i'm pretty sure this has to do with 64 code.

Do you guys have any tips and would like to share any caveats faced when porting your codebases to 64 bit? I don't have lot's of experience on the topic and the tutorials i read after a quick google search did not help a lot (no mentions of std lib caveats, like this strcmp one).

[Spike]

QC strings are expressed as indexes relative to the stringtable loaded from the progs.dat.
the correct way is to make multiple types of strings - make ones with the high bit set are indexes into a pointer table instead, and ones with it clear are reguar stringtable indexes.
the lazy way is to ensure that your string_t type is signed, and to allocate space for all the strings that you will use on the hunk or zone memory (as these will always be within 2gb of the progs string table).
Currently it'll crash whenever the QC tries to reference the player's netname or refer to a string pointing to a buffer inside ftos for example. There are other extension builtins of course where stuff like reading cvar strings is even more problematic, but for vanilla you should be able to get away with just netnames and ftos.

the issue is not strcmp, its with address truncation on the 'qcstring = nativestring - progs->stringtable' logic. the result of the negation needs more than 32 bits, but you can't change the type of qcstring(ie: string_t).
It means that 'nativestring = (char*)progs->stringtable+qcstring' calculates a different offset for nativestring and then you get a segfault when its read.

there's a 64bit bug in sprites too, iirc

[JasonX]

string_t type? Inside pr_exec?

[Spike]

typedef int string_t;
see. 32bit.

sv.clients[0].edict->v.netname = "some string thats somewhere in the .cdata section of your binary" - pr_strings;
overflow(OVR_DONTREALISE);
waitfortheqcakaprexectotestselfdotnetname();
c->_float = !strcmp(pr_strings+a->string,pr_strings+b->string);
crash(SIG_SEGFAULT, qcstring);

if you're using msvc, try using the 'Smaller Type Check' setting. You should see these overflows that way (it will still hide things that get cast explicitly though - this shouldn't be one of those).

[JasonX]

I have changed to a signed value, but i keep getting segfaults. Is there any good tool for aiding the port of 32-bit applications to 64-bit on Linux?

[Spike]

it doesn't matter if its signed or unsigned. if its a 32bit offset into a 64bit address space where the two pointers are greater than 2gb/4gb apart, you will have problems.
the only semi-trivial fix is to use a signed integer and to allocate *all* qc-accessible strings on the quake hunk (player names and temp strings will need to be moved). naturally this will still break many qccx hacks.
address space layout randomisation likes using the WHOLE 47-bit address space available to it - nice big gaps make it harder to guess where the actual code is, avoiding potential exploits.

I don't know about linux, but msvc has a 'smaller type check' setting that will generate some extra code that triggers a fault whenever a variable is stored into a variable that is too small for the value. this should generally be sufficient to catch pointer offset overflows, but does depend upon the result not explicitly being cast to an int. if all goes well, this gives you a fault when the subtraction happens instead of merely where the dereference happens.
linux seems to only have compile-time warnings, and will thus probably contain lots of false positives.
either way, you're actually going to have to use a debugger.

[JasonX]

I'm not sure if i follow. Instead of allocating inside the function heap, should i add the strings to the Quake hunk?

[Spike]

exactly that. instead of using globals or malloc, you should use Hunk_Alloc (or friends) for qc-accessible strings. This ensures all the strings are within the same 2gb range as the progs.dat's stringtable, and thus prevents the overflows.
its lame, but it works.

[jitspoe]

What's the point in switching to 64bit? Are you planning to address more than 2 gigs of memory?

[Baker]

[Spirit]

You just need to install 32 bit libraries on Linux.

[Spike]

for debian/ubuntu:
apt-get install package-name:i386
no need to port, but 64bit users are less likely to have 32bit versions of your dependancies installed, and if you only provide 32bit dependancy lists then they'll need to tweak them (and figure out any omissions too).
you probably still want to as a personal pride sort of thing, but its really just a dependancy issue. there's no reason you can't make a (bash) install script that downloads+installs the appropriate dependancies along with your game.

abuse of textures can result in a need for more than 2gb ram. both dp and fte were modified to make use of windows' large-address-aware thing to allow 32bit versions to access 4gb of ram on 64bit windows (or 3gb on 32bit windows), in order to avoid oom-crashing when used with the ultra version of rygel's texture package.
quite impressive for a 20-year-old game tbh.

[Baker]

[Spirit]

[qbism]