Inside3D!
     

compressed pk3 support tutorial for standard quake soonish
Goto page 1, 2  Next
 
Post new topic   Reply to topic    Inside3d Forums Forum Index -> Programming Tutorials
View previous topic :: View next topic  
Author Message
reckless



Joined: 24 Jan 2008
Posts: 390
Location: inside tha debugger

PostPosted: Sat Aug 15, 2009 11:56 pm    Post subject: compressed pk3 support tutorial for standard quake soonish Reply with quote

due to work it takes a bit for me to drop this one here so if anyone feels the urge feel free to rip it from realm Smile

its not quite done yet but close only part of the old system i still need to mangle in are the demos everything else can be loaded from pk3's.

i might have to rewrite parts of it since i use newer hunk allocation system mh made so its not quite straight forward the changes are pretty massive tbh :/ but all in the name of good memory handling.

it can litterally take 1000's of textures models etc but the loadtime increases quite a bit if you overdo it it doesnt have a great impact on speed once loaded though.

it also makes it easier for modders to include special parts since only the relevant data gets loaded "if your pk3's are in the same dir as the mod" it wont load textures etc from id1 besides the mips ofc.
Back to top
View user's profile Send private message
Spike



Joined: 05 Nov 2004
Posts: 944
Location: UK

PostPosted: Sun Aug 16, 2009 1:04 am    Post subject: Reply with quote

If you're worried about loading speed, you could potentially sort the file names somehow. Then do a binary search on them whenever a file is loaded. Or you could build a hash table of them all.

It really depends how many different textures you try to load.
*.pcx, *.tga, *.lmp, *.jpg, *.jpeg, *.png, *_bump.*, *_luma.*, *_norm.*, *_gloss.*, *_glow.*, *_pants.*, *_shirt.*, textures/*.*, textures/$mapname/*.*, override/*.*, textures/exmx/*.*, etc.
id1/*, qw/*, $engine/*, $gamedir/*
And all the combinations thereof.

Really depends how many other engines you try to be compatible with. :)

But yeah, pk3 files don't have to be any slower to search than pak files, they just generally have more files inside.
_________________
What's a signature?
Back to top
View user's profile Send private message Visit poster's website
mh



Joined: 12 Jan 2008
Posts: 910

PostPosted: Sun Aug 16, 2009 1:37 pm    Post subject: Reply with quote

It's more a case that the new Hunk_Alloc system is slower, as it's quite prone to "lots of small allocations" syndrome.

This one is better. Very Happy

Code:
/*
Copyright (C) 1996-1997 Id Software, Inc.

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. 

See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.

*/

#include "quakedef.h"

/*
========================================================================================================================

      UTILITY FUNCTIONS

========================================================================================================================
*/

int MemoryRoundSize (int size, int roundtok)
{
   // kilobytes
   roundtok *= 1024;

   for (int newsize = 0;; newsize += roundtok)
      if (newsize >= size)
         return newsize;

   // never reached
   return size;
}


/*
========================================================================================================================

      CACHE MEMORY

   Certain objects which are loaded per map can be cached per game as they are reusable.  The cache should
   always be thrown out when the game changes, and may be discardable at any other time.  The cache is just a
   wrapper around the main virtual memory system, so use Virtual_PoolFree to discard it.

========================================================================================================================
*/


typedef struct cacheobject_s
{
   struct cacheobject_s *next;
   void *data;
   char *name;
} cacheobject_t;


cacheobject_t *cachehead = NULL;
int numcacheobjects = 0;

void *Cache_Check (char *name)
{
   for (cacheobject_t *cache = cachehead; cache; cache = cache->next)
   {
      // these should never happen
      if (!cache->name) continue;
      if (!cache->data) continue;

      if (!stricmp (cache->name, name))
      {
         Con_DPrintf ("Reusing %s from cache\n", cache->name);
         return cache->data;
      }
   }

   // not found in cache
   return NULL;
}


void *Cache_Alloc (char *name, void *data, int size)
{
   cacheobject_t *cache = (cacheobject_t *) Virtual_PoolAlloc (VIRTUAL_POOL_CACHE, sizeof (cacheobject_t));

   // alloc on the cache
   cache->name = (char *) Virtual_PoolAlloc (VIRTUAL_POOL_CACHE, strlen (name) + 1);
   cache->data = Virtual_PoolAlloc (VIRTUAL_POOL_CACHE, size);

   // copy in the name
   strcpy (cache->name, name);

   // count objects for reporting
   numcacheobjects++;

   // copy to the cache buffer
   if (data) memcpy (cache->data, data, size);

   // link it in
   cache->next = cachehead;
   cachehead = cache;

   // return from the cache
   return cache->data;
}


void Cache_Init (void)
{
   cachehead = NULL;
   numcacheobjects = 0;
}


/*
========================================================================================================================

      VIRTUAL POOL BASED MEMORY SYSTEM

   This is officially the future of DirectQ memory allocation.  Instead of using lots of small itty bitty memory
   chunks we instead use a number of large "pools", each of which is reserved but not yet committed in virtual
   memory.  We can then commit as we go, thus giving us the flexibility of (almost) unrestricted memory, but the
   convenience of the old Hunk system (with everything consecutive in memory).

========================================================================================================================
*/

typedef struct vpool_s
{
   char name[24];
   int maxmem;
   int lowmark;
   int highmark;
   byte *membase;
} vpool_t;

// these should be declared in the same order as the #defines in heap.h
vpool_t virtualpools[NUM_VIRTUAL_POOLS] =
{
   // stuff in this pool is never freed while DirectQ is running
   {"Permanent", 32 * 1024 * 1024, 0, 0, NULL},

   // stuff in these pools persists for the duration of the game
   {"This Game", 32 * 1024 * 1024, 0, 0, NULL},
   {"Cache", 256 * 1024 * 1024, 0, 0, NULL},

   // stuff in these pools persists for the duration of the map
   // warpc only uses ~48 MB
   {"This Map", 128 * 1024 * 1024, 0, 0, NULL},
   {"Edicts", 10 * 1024 * 1024, 0, 0, NULL},

   // used for temp allocs where we don't want to worry about freeing them
   {"Temp Allocs", 128 * 1024 * 1024, 0, 0, NULL},

   // spare slots
   {"Unused", 1 * 1024 * 1024, 0, 0, NULL},
   {"Unused", 1 * 1024 * 1024, 0, 0, NULL},
   {"Unused", 1 * 1024 * 1024, 0, 0, NULL},
   {"Unused", 1 * 1024 * 1024, 0, 0, NULL}
};


void *Virtual_PoolAlloc (int pool, int size)
{
   if (pool < 0 || pool >= NUM_VIRTUAL_POOLS)
      Sys_Error ("Virtual_PoolAlloc: Invalid Pool");

   vpool_t *vp = &virtualpools[pool];

   // if temp file loading overflows we just reset it
   if (pool == VIRTUAL_POOL_TEMP && (vp->lowmark + size) >= vp->maxmem)
   {
      // if the temp file pool is too small to hold this allocation we reset it so that it's big enough
      if (size > vp->maxmem)
         Virtual_PoolReset (pool, size);
      else Virtual_PoolFree (VIRTUAL_POOL_TEMP);
   }

   // not enough pool space
   if ((vp->lowmark + size) >= vp->maxmem)
      Sys_Error ("Virtual_PoolAlloc: Overflow");

   // only pass over the commit region otherwise lots of small allocations will pass over
   // the *entire* *buffer* every time (slooooowwwwww)
   if ((vp->lowmark + size) >= vp->highmark)
   {
      // alloc in 256k batches
      int newsize = MemoryRoundSize (vp->lowmark + size, 256);

      if (!VirtualAlloc (vp->membase + vp->lowmark, newsize - vp->lowmark, MEM_COMMIT, PAGE_READWRITE))
         Sys_Error ("Virtual_PoolAlloc: VirtualAlloc Failed");

      vp->highmark = newsize;
   }

   // set up
   void *buf = (vp->membase + vp->lowmark);
   vp->lowmark += size;

   return buf;
}


void Virtual_PoolReset (int pool, int newsizebytes)
{
   // graceful failure
   if (pool < 0 || pool >= NUM_VIRTUAL_POOLS) return;

   vpool_t *vp = &virtualpools[pool];

   // fully release the memory
   if (vp->membase) VirtualFree (vp->membase, 0, MEM_RELEASE);

   // fill in
   vp->lowmark = 0;
   vp->highmark = 0;
   vp->maxmem = MemoryRoundSize (newsizebytes, 1024);

   // reserve the memory for use by this pool
   vp->membase = (byte *) VirtualAlloc (NULL, vp->maxmem, MEM_RESERVE, PAGE_NOACCESS);
}


void Virtual_PoolFree (int pool)
{
   // graceful failure
   if (pool < 0 || pool >= NUM_VIRTUAL_POOLS) return;

   // already free
   if (!virtualpools[pool].lowmark) return;

   // decommit the allocated pool; if it straddles a page boundary the full extra page will be freed
   VirtualFree (virtualpools[pool].membase, virtualpools[pool].lowmark, MEM_DECOMMIT);

   // reset lowmark
   virtualpools[pool].lowmark = 0;
   virtualpools[pool].highmark = 0;

   // reinit the cache if that was freed
   if (pool == VIRTUAL_POOL_CACHE) Cache_Init ();
}


void Virtual_PoolInit (void)
{
   for (int i = 0; i < NUM_VIRTUAL_POOLS; i++)
   {
      // skip over any pools that were already alloced
      if (virtualpools[i].membase) continue;

      // reserve the memory for use by this pool
      virtualpools[i].membase = (byte *) VirtualAlloc (NULL, virtualpools[i].maxmem, MEM_RESERVE, PAGE_NOACCESS);
   }

   // init the cache
   Cache_Init ();
}


/*
========================================================================================================================

      ZONE MEMORY

   The zone is used for small strings and other stuff that's dynamic in nature and would normally be handled by
   malloc and free.  It primarily exists so that we can report on zone usage, but also so that we can avoid using
   malloc and free, as their behaviour is runtime dependent.

   The win32 Heap* functions basically operate identically to the old zone functions except they let use reserve
   virtual memory and also do all of the tracking and other heavy lifting for us.

========================================================================================================================
*/

typedef struct zblock_s
{
   int size;
   void *data;
} zblock_t;

int zonesize = 0;
HANDLE zoneheap = NULL;

void *Zone_Alloc (int size)
{
   // create an initial heap for use with the zone
   // this heap has 128K initially allocated and 32 MB reserved from the virtual address space
   if (!zoneheap) zoneheap = HeapCreate (0, 0x20000, 0x2000000);

   size += sizeof (zblock_t);
   size = (size + 7) & ~7;

   zblock_t *zb = (zblock_t *) HeapAlloc (zoneheap, HEAP_ZERO_MEMORY, size);

   zb->size = size;
   zb->data = (void *) (zb + 1);

   zonesize += size;

   return zb->data;
}


void Zone_Free (void *ptr)
{
   // attempt to free a NULL pointer
   if (!ptr) return;
   if (!zoneheap) return;

   // retrieve zone block pointer
   zblock_t *zptr = ((zblock_t *) ptr) - 1;

   zonesize -= zptr->size;

   // release this back to the OS
   HeapFree (zoneheap, 0, zptr);
   HeapCompact (zoneheap, 0);
}


/*
========================================================================================================================

      REPORTING

========================================================================================================================
*/
void Virtual_Report_f (void)
{
   int reservedmem = 0;
   int committedmem = 0;

   Con_Printf ("\n-----------------------------------\n");
   Con_Printf ("Pool           Highmark     Lowmark\n");
   Con_Printf ("-----------------------------------\n");

   for (int i = 0; i < NUM_VIRTUAL_POOLS; i++)
   {
      // don't report on empty pools
      if (!virtualpools[i].highmark) continue;

      Con_Printf
      (
         "%-11s  %7.2f MB  %7.2f MB\n",
         virtualpools[i].name,
         ((float) virtualpools[i].highmark / 1024.0f) / 1024.0f,
         ((float) virtualpools[i].lowmark / 1024.0f) / 1024.0f
      );

      reservedmem += virtualpools[i].highmark;
      committedmem += virtualpools[i].lowmark;
   }

   Con_Printf ("-----------------------------------\n");

   Con_Printf
   (
      "%-11s  %7.2f MB  %7.2f MB\n",
      "Total",
      ((float) reservedmem / 1024.0f) / 1024.0f,
      ((float) committedmem / 1024.0f) / 1024.0f
   );

   Con_Printf ("-----------------------------------\n");
   Con_Printf ("%i objects in cache\n", numcacheobjects);
   Con_Printf ("Zone size: %i KB\n", (zonesize + 1023) / 1023);
}


// also alloc the old heap_report command in case anyone used it
cmd_t virtual_report_cmd ("virtual_report", Virtual_Report_f);
cmd_t heap_report_cmd ("heap_report", Virtual_Report_f);


_________________
DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines
Back to top
View user's profile Send private message Visit poster's website
reckless



Joined: 24 Jan 2008
Posts: 390
Location: inside tha debugger

PostPosted: Sun Aug 16, 2009 11:47 pm    Post subject: Reply with quote

oh cool Smile

ill give it a try cheers.
Back to top
View user's profile Send private message
mh



Joined: 12 Jan 2008
Posts: 910

PostPosted: Mon Aug 17, 2009 8:42 am    Post subject: Reply with quote

A few other things you can do is to keep the number of pools small and allocate to each in batches. I have a 256 K batch size hard-coded into here ("int newsize = MemoryRoundSize (vp->lowmark + size, 256);") but with a smaller number of memory pools that could be easily increased to 1 MB. What this will mean is that instead of committing memory on a regular basis it's only committed in discrete 1 MB chunks, meaning about 7 or 8 commits per map (as opposed to potentially several thousand if you just commit memory as required). A smaller number of pools means less batch size overhead; with 6 pools and a 256 K batch the most "wasted" memory you'll ever have is 1.25 MB.

My current incarnation uses only 6 pools - permanent, this game, cached models and sounds, this map, edicts and temp. Cached objects can be easily amalgamated with this game, but I want to keep it separate so that I can demand-flush it. You could cvar-ize the commit batch size if you wanted; I'm in 2 minds about this one, it's nice to expose it but should regular users be able to mess with something fairly low-level like that?

Precalculating sizes and allocating in bulk is always good too. For example, you can figure out in advance how many glpoly_t structs (and the sizes of their vertexes) you need for non warp-surfaces, so why not just allocate a single big buffer at the end of Mod_LoadFaces and write the polys into that instead of doing a separate allocation for each poly?

You'll notice that I have fixed hard-coded size limits on the pools here, but that's OK. warpc only needs 48 MB from the "this map" pool (or about 85 MB if you don't cache models and sounds) so a 128 MB upper limit is fine. You could increase it to 256 or even higher if you want; because it's only reserving an area of virtual memory for itself there's no actual resource utilization involved in doing so (just make sure that you don't reserve more than 2 GB total though!)

Anyway, I'm getting into areas more appropriate for the Engine Coding Tips thread now, so I think I'll leave it at that!
_________________
DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines
Back to top
View user's profile Send private message Visit poster's website
mh



Joined: 12 Jan 2008
Posts: 910

PostPosted: Mon Aug 17, 2009 5:34 pm    Post subject: Reply with quote

BUG: Should be highmark, not lowmark in the VirtualFree call in Virtual_PoolFree
_________________
DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines
Back to top
View user's profile Send private message Visit poster's website
reckless



Joined: 24 Jan 2008
Posts: 390
Location: inside tha debugger

PostPosted: Mon Aug 17, 2009 9:01 pm    Post subject: Reply with quote

check thanks for pointing out Wink
Back to top
View user's profile Send private message
mh



Joined: 12 Jan 2008
Posts: 910

PostPosted: Mon Aug 17, 2009 10:37 pm    Post subject: Reply with quote

Yr welcome. Very Happy

This one should also be safe for making sure that leafs and nodes are allocated in consecutive memory (very important for node = (mnode_t *) &cl.worldmodel->leafs[i + 1]; to work right).
_________________
DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines
Back to top
View user's profile Send private message Visit poster's website
Baker



Joined: 14 Mar 2006
Posts: 1538

PostPosted: Tue Dec 29, 2009 2:27 pm    Post subject: Reply with quote

@MH re: memory manager

Is Windows specific? If not, is road tested and debugged?

Just wondering. I hate Quake's memory management but also fear changing it.
Back to top
View user's profile Send private message
mh



Joined: 12 Jan 2008
Posts: 910

PostPosted: Tue Dec 29, 2009 6:31 pm    Post subject: Reply with quote

Windows specific I'm afraid. I don't know what the virtual memory APIs for other OSs are, but anyone who does should be able to port it quite easy. Probably the biggest challenge would be dealing with Zone memory - Windows has so many memory APIs that I suspect there may not be an equivalent. You could just replace it with malloc and free and I suppose it would work (be careful though of my Ninja trickery in Zone_Free...)

As for road tested, it's pretty much the one I use in DirectQ so it's definitely been let out in the wild for a good while.

But I agree though, tackling the Quake memory manager is not a trivial task. Cache, Hunk Low, Hunk High, Zone - AAARRGGHHHH!!! Have you looked at the Q2 source code? That has another memory management system that should be easier to port and that also has quite good flexibility.
_________________
DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines
Back to top
View user's profile Send private message Visit poster's website
reckless



Joined: 24 Jan 2008
Posts: 390
Location: inside tha debugger

PostPosted: Tue Dec 29, 2009 10:29 pm    Post subject: Reply with quote

VirtualAlloc exist on linux also so should be fine Wink
Back to top
View user's profile Send private message
mh



Joined: 12 Jan 2008
Posts: 910

PostPosted: Tue Dec 29, 2009 10:34 pm    Post subject: Reply with quote

reckless wrote:
VirtualAlloc exist on linux also so should be fine Wink

Sounds good. Anyone in a position to test stuff? Question
_________________
DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines
Back to top
View user's profile Send private message Visit poster's website
reckless



Joined: 24 Jan 2008
Posts: 390
Location: inside tha debugger

PostPosted: Wed Dec 30, 2009 4:47 pm    Post subject: Reply with quote

ill see what i can do Smile

i belive VirtualAlloc support was added later looking at things i see several different support models the original i think is mmap/munmap which btw can be emulated with (yes exactly that) VirtualAlloc/VirtualFree.

funny thing is i just wrote several wrappers for some programs i was porting to my mingw compiler suite and one of them actually emulates the above.

#if defined(__MINGW32__) && !defined(QUICK)
#include <windows.h>
#define QUICK 1
#define MAP_FAILED 0
#define MREMAP_FIXED 2 // the value in linux, though it doesn't really matter
// These, when combined with the mmap invariants below, yield the proper action
#define PROT_READ PAGE_READWRITE
#define PROT_WRITE PAGE_READWRITE
#define MAP_ANONYMOUS MEM_RESERVE
#define MAP_PRIVATE MEM_COMMIT
#define MAP_SHARED MEM_RESERVE // value of this #define is 100% arbitrary

// VirtualAlloc is only a replacement for mmap when certain invariants are kept
#define mmap(start, length, prot, flags, fd, offset) \
( (start) == NULL && (fd) == -1 && (offset) == 0 && \
(prot) == (PROT_READ|PROT_WRITE) && (flags) == (MAP_PRIVATE|MAP_ANONYMOUS) \
? VirtualAlloc(0, length, MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE) \
: NULL )

#define munmap(start, length) (VirtualFree(start, 0, MEM_RELEASE) ? 0 : -1)
#endif // HAVE_MMAP

hmm time to test out windows 7 virtual machines with my suse distro Cool
Back to top
View user's profile Send private message
reckless



Joined: 24 Jan 2008
Posts: 390
Location: inside tha debugger

PostPosted: Wed Jan 06, 2010 6:45 am    Post subject: Reply with quote

eh ? i just wrote a ton of the codechanges to the pk3 tutorial and everything just dissapeared Sad ?

maybe it was to big for one page ?
Back to top
View user's profile Send private message
r00k



Joined: 13 Nov 2004
Posts: 483

PostPosted: Wed Jan 06, 2010 8:35 am    Post subject: Reply with quote

've noticed lately that as you type a [post a reply box] full and press preview sometimes it asks me to login and then poof no more message :O <- back cut/paste -> ins continue Razz
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    Inside3d Forums Forum Index -> Programming Tutorials All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2004 phpBB Group