Inside3D!
     

Quake Memory Manager - Part 1: the Zone

 
Post new topic   Reply to topic    Inside3d Forums Forum Index -> Programming Tutorials
View previous topic :: View next topic  
Author Message
mh



Joined: 12 Jan 2008
Posts: 909

PostPosted: Wed Jan 06, 2010 9:36 pm    Post subject: Quake Memory Manager - Part 1: the Zone Reply with quote

I'm going to do a series of tutorials in which we tackle the most maligned of wee beasties, the Quake memory manager. This code will be based on standard Win/GL Quake and will only require changes to be made to the zone.c file. I'm maintaining the exact same interface as the original Quake memory manager so that anyone can easily use it in their own engines.

This code will be usable with both software and GLQuake, and in fact I'm developing and testing it on software Quake to ensure this. Where possible I'm writing it to be completely portable, although unfortunately the Hunk is most likely going to break this objective.

For part 1 we're going to tackle the Zone. This is the region of Quake's memory which is used for allocation of small strings and other didgeridoos, although at the end of this part you'll be able to use it for whatever you want, no matter how big.

Concept

By default the Zone is a very tight region (47 K) and shares it's memory space with the rest of Quake's memory. Sometimes Quake runs out of Zone space and crashes, so you need to restart with the -zone command-line option and replay up to the point where it crashed.

Here we're going to completely remove that limitation (the only limitation will be the address space of your OS) and also remove the Zone from the shared memory area (so that Zone corruptions don't corrupt elsewhere).

Code Changes

We're going to replace every single Zone function with standard C memory functions instead. I'll give them all here, and make comments where required.

Code:
void Z_ClearZone (memzone_t *zone, int size)
{
}


We don't need to use this function anymore, but we'll retain it for compatibility with code that might be elsewhere. This is the general pattern I'm going to follow throughout this series.

Code:
void Z_Free (void *ptr)
{
   int *zblock = ((int *) ptr) - 1;

   if (zblock[0] != ZONEID)
      Sys_Error ("Z_Free: freed a pointer without ZONEID");

   free (zblock);
}


Note the ninja-trickery in this one. I'll discuss what's happening here after I give Z_Malloc.

Code:
void *Z_Malloc (int size)
{
   int *zblock = (int *) malloc (size + sizeof (int));

   if (!zblock)
      Sys_Error ("Z_Malloc: failed on allocation of %i bytes", size);

   memset (zblock, 0, size + sizeof (int));

   zblock[0] = ZONEID;

   return (zblock + 1);
}


Right. What we're doing here is allocating an extra 4 bytes in addition to the requested size. The first 4 bytes of the buffer are set to the ZONEID value, which is used as a guard value in Z_Free. The buffer we return starts following those 4 bytes.

For Z_Free what we do is back up by 4 bytes from the address given (a pointer is just a memory address and you can go backwards from it as well as forwards), checking that the first 4 bytes equal ZONEID (in case we try to Z_Free something that wasn't allocated via Z_Malloc), then freeing it from that point.

Code:
void *Z_TagMalloc (int size, int tag)
{
   return Z_Malloc (size);
}


This function was never actually used in the engine. I guess it might have been experimental QuakeII stuff as QuakeII does use Z_Malloc with tags.

Code:
void Z_Print (memzone_t *zone)
{
}

void Z_CheckHeap (void)
{
}


Likewise we don't use these any more but we keep them to maintain a compatible interface.

Final code change is at the bottom of the Memory_Init function (go find it at the bottom of zone.c). See this block:

Code:
   p = COM_CheckParm ("-zone");

   if (p)
   {
      if (p < com_argc-1)
         zonesize = Q_atoi (com_argv[p+1]) * 1024;
      else
         Sys_Error ("Memory_Init: you must specify a size in KB after -zone");
   }

   mainzone = Hunk_AllocName (zonesize, "zone" );
   Z_ClearZone (mainzone, zonesize);


We don't need it anymore as we no longer have Zone limitations and we certainly don't want to pollute our main memory with something that is never used. Remove it. Delete it. Take it outside and kill it until it is dead.

Cleaning Up

When I'm finished I'll release a fully modified and cleaned up Zone.c, but for now you can go through the file, identify any variables or structures that are no longer used, and delete them.

The next part

For the next part I think we're going to do the Cache. The Cache is heavily dependent on Hunk allocations, so we're likewise going to break that dependency and put it in it's own memory space, giving it room to grow as large as it wants.

When this is posted (in a coupla hours time Very Happy) you'll have a very clean and tight zone.c, that looks virtually unrecognisable compared to the old one.

And then we'll do the Hunk. Very Happy
_________________
DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines
Back to top
View user's profile Send private message Visit poster's website
metlslime



Joined: 05 Feb 2008
Posts: 177

PostPosted: Wed Jan 06, 2010 9:55 pm    Post subject: Reply with quote

...This should be good Smile
Back to top
View user's profile Send private message
Baker



Joined: 14 Mar 2006
Posts: 1538

PostPosted: Thu Jan 07, 2010 12:04 am    Post subject: Reply with quote

I'm enjoying reading these, I'd like to learn more about memory management.
Back to top
View user's profile Send private message
Tomaz



Joined: 05 Nov 2004
Posts: 49

PostPosted: Thu Jan 21, 2010 11:47 pm    Post subject: Reply with quote

What codebase is this based on?
Like the comment about Z_TagMalloc not being used? and the code you pasted there? Thats NOT from the original quake code, in fact, all the Z alloc code does happen in Z_TagMalloc, since Z_Malloc pretty much does nothing but call Z_TagMalloc.
Back to top
View user's profile Send private message
reckless



Joined: 24 Jan 2008
Posts: 390
Location: inside tha debugger

PostPosted: Fri Jan 22, 2010 6:19 am    Post subject: Reply with quote

hmm aye tomaz is right Z_Malloc does indeed call Z_TagMalloc

and Z_Malloc is used in the vm and filesystem in some places ? allthough using plain mallloc or VirtualAlloc seems to do the trick just fine Smile
Back to top
View user's profile Send private message
mh



Joined: 12 Jan 2008
Posts: 909

PostPosted: Fri Jan 22, 2010 10:23 am    Post subject: Reply with quote

Tomaz wrote:
What codebase is this based on?

WinQuake. The version from ID.

Tomaz wrote:
Like the comment about Z_TagMalloc not being used? and the code you pasted there? Thats NOT from the original quake code, in fact, all the Z alloc code does happen in Z_TagMalloc, since Z_Malloc pretty much does nothing but call Z_TagMalloc.

Of course it's not from the original code, it's a code change. It seems a little unreasonable to me to expect code changes to be identical to the original code, unless I'm missing a point here.

Yes, I know that Z_Malloc calls Z_TagMalloc, but where else is Z_TagMalloc used in the engine? Nowhere.

Tomaz, I respect your work, but this isn't the first time you've made derisive comments about other people sharing their code and ideas here (I'll refer you to your outburst on the alpha tutorial a while back). Could you at least try to be constructive and contribute a little?
_________________
DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines
Back to top
View user's profile Send private message Visit poster's website
Tomaz



Joined: 05 Nov 2004
Posts: 49

PostPosted: Fri Jan 22, 2010 2:25 pm    Post subject: Reply with quote

mh wrote:
Of course it's not from the original code, it's a code change. It seems a little unreasonable to me to expect code changes to be identical to the original code, unless I'm missing a point here.

Yes, I know that Z_Malloc calls Z_TagMalloc, but where else is Z_TagMalloc used in the engine? Nowhere.

Tomaz, I respect your work, but this isn't the first time you've made derisive comments about other people sharing their code and ideas here (I'll refer you to your outburst on the alpha tutorial a while back). Could you at least try to be constructive and contribute a little?


I'm sorry that was never my intention. I respect your work as well, I just pointed this out since I found it to be false information. If I had not cared I would not have replied at all.

What I meant was that you made Z_TagMalloc call Z_Malloc, why? Since previously it was Z_Malloc calling Z_TagMalloc, so if Z_Malloc dont need Z_TagMalloc anymore, why not just remove it?

I understand that you made empty functions that does nothing when they functiosn are called from elsewhere, but in this case, there is nothing anywhere calling Z_TagMalloc, and again. I never meant to make a "derisive"?? ( don't even know what that means but it sounds bad ) comment about this code. I was going to add this to CleanQuakeCpp, which is when I found your comments about Z_TagMalloc to be.. weird.. in lack of a better word. My intention with the post was to get an answer to why it was made like that, not do disrespect your work.

And yes I know I'm not liked around here, or anywhere else for that matter. It's not that I am an evil person, I just never bother/have time to put down those extra words when i write comments that change them from just being mean pointing out errors, to be more constructive.

But I'll defenitly try to be more constructive in my comments in the future.





Edit:

But I have to defend myself, what was wrong regarding my comments on Baker's alpha tutorial? I merely pointed out 3 things that was redundant/wrong and explained why on all 3, and then gave a lengthy explenation of why an alpha sorter is needed.

What about that is bad and not constructive?
Back to top
View user's profile Send private message
Baker



Joined: 14 Mar 2006
Posts: 1538

PostPosted: Fri Jan 22, 2010 2:54 pm    Post subject: Reply with quote

Quote:
Edit:

But I have to defend myself, what was wrong regarding my comments on Baker's alpha tutorial? I merely pointed out 3 things that was redundant/wrong and explained why on all 3, and then gave a lengthy explenation of why an alpha sorter is needed.

What about that is bad and not constructive?


Your comments on my alpha tutorial allowed me to fix alpha in my engine. I needed to sort the entities. I don't believe I would have figured it out by myself very quickly.

/quick point, exits conversation ... Very Happy
_________________
Tomorrow Never Dies. I feel this Tomorrow knocking on the door ...
Back to top
View user's profile Send private message
frag.machine



Joined: 25 Nov 2006
Posts: 728

PostPosted: Fri Jan 22, 2010 3:33 pm    Post subject: Reply with quote

Guys... From what I understand, there were no intention to be sarcastic, aggressive or something like that from any part. To me, that's one good example on how having a written conversation can be... complicated, sometimes. Specially to people like me, to whom English is not the main language. That's why emoticons are so valuable: they are there to tell to the other side "hey, don't take everything I am writing here so seriously or in the wrong way".

And now, for something completely unrelated...



/me goes back to my coding hole trying to make moving liquid brushes work
_________________
frag.machine - Q2K4 Project
http://fragmachine.quakedev.com/
Back to top
View user's profile Send private message Visit poster's website
c0burn



Joined: 05 Nov 2004
Posts: 158
Location: Liverpool, England

PostPosted: Fri Jan 22, 2010 3:54 pm    Post subject: Reply with quote

frag.machine wrote:
Guys... From what I understand, there were no intention to be sarcastic, aggressive or something like that from any part. To me, that's one good example on how having a written conversation can be... complicated, sometimes. Specially to people like me, to whom English is not the main language. That's why emoticons are so valuable: they are there to tell to the other side "hey, don't take everything I am writing here so seriously or in the wrong way".

And now, for something completely unrelated...



/me goes back to my coding hole trying to make moving liquid brushes work


Engine or QC? Extras_r4 did it well.
Back to top
View user's profile Send private message Visit poster's website AIM Address MSN Messenger
Spike



Joined: 05 Nov 2004
Posts: 944
Location: UK

PostPosted: Fri Jan 22, 2010 5:01 pm    Post subject: Reply with quote

Z_TagMalloc is useful for FRIK_FILE and unclean mods. Really really useful. I would not otherwise recommend its use.
_________________
What's a signature?
Back to top
View user's profile Send private message Visit poster's website
frag.machine



Joined: 25 Nov 2006
Posts: 728

PostPosted: Fri Jan 22, 2010 8:53 pm    Post subject: Reply with quote

cOburn wrote:
frag.machine wrote:
/me goes back to my coding hole trying to make moving liquid brushes work


Engine or QC? Extras_r4 did it well.


I started using a combined QuakeC-engine solution, now I am trying engine-side only. We were discussing about this here.
_________________
frag.machine - Q2K4 Project
http://fragmachine.quakedev.com/
Back to top
View user's profile Send private message Visit poster's website
Teiman



Joined: 03 Jun 2007
Posts: 309

PostPosted: Fri Jan 22, 2010 9:51 pm    Post subject: Reply with quote

I did some horrible things in Telejano:

- Moved away from the program-managed memory to OS managed memory. Uh.
- Using a closed source proprietary module to detect memory leaks. I was very vigilant at this, maybe obsessive.

The whole game navigated away from that "16 MB" original footprint of quake to a whole universe of bloat.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    Inside3d Forums Forum Index -> Programming Tutorials All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2004 phpBB Group