Index | FAQ | Search | Memberlist | Usergroups | Profile | Log in to check your private messages | Log in | Register

compressed pk3 support tutorial for standard quake soonish Goto page 1, 2 Next

| Inside3d Forums Forum Index -> Programming Tutorials

View previous topic :: View next topic
Author | Message
--- | ---
reckless Joined: 24 Jan 2008 Posts: 390 Location: inside tha debugger |

Posted: Sat Aug 15, 2009 11:56 pm Post subject: compressed pk3 support tutorial for standard quake soonish |

due to work it takes a bit for me to drop this one here so if anyone feels the urge feel free to rip it from realm its not quite done yet but close only part of the old system i still need to mangle in are the demos everything else can be loaded from pk3's.
i might have to rewrite parts of it since i use newer hunk allocation system mh made so its not quite straight forward the changes are pretty massive tbh :/ but all in the name of good memory handling.
it can litterally take 1000's of textures models etc but the loadtime increases quite a bit if you overdo it it doesnt have a great impact on speed once loaded though.
it also makes it easier for modders to include special parts since only the relevant data gets loaded "if your pk3's are in the same dir as the mod" it wont load textures etc from id1 besides the mips ofc.

Back to top |

Spike Joined: 05 Nov 2004 Posts: 944 Location: UK |

Posted: Sun Aug 16, 2009 1:04 am Post subject: |

If you're worried about loading speed, you could potentially sort the file names somehow. Then do a binary search on them whenever a file is loaded. Or you could build a hash table of them all. It really depends how many different textures you try to load.
*.pcx, *.tga, *.lmp, *.jpg, *.jpeg, *.png, *_bump.*, *_luma.*, *_norm.*, *_gloss.*, *_glow.*, *_pants.*, *_shirt.*, textures/*.*, textures/$mapname/*.*, override/*.*, textures/exmx/*.*, etc.
id1/*, qw/*, $engine/*, $gamedir/*
And all the combinations thereof.
Really depends how many other engines you try to be compatible with. :)
But yeah, pk3 files don't have to be any slower to search than pak files, they just generally have more files inside.
_________________
What's a signature?

Back to top |

mh Joined: 12 Jan 2008 Posts: 910 |

Posted: Sun Aug 16, 2009 1:37 pm Post subject: |

It's more a case that the new Hunk_Alloc system is slower, as it's quite prone to "lots of small allocations" syndrome. This one is better.

Code:
/* Copyright (C) 1996-1997 Id Software, Inc. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
*/
#include "quakedef.h"
/*
========================================================================================================================
UTILITY FUNCTIONS
========================================================================================================================
*/
int MemoryRoundSize (int size, int roundtok)
{
// kilobytes
roundtok *= 1024;
for (int newsize = 0;; newsize += roundtok)
if (newsize >= size)
return newsize;
// never reached
return size;
}
/*
========================================================================================================================
CACHE MEMORY
Certain objects which are loaded per map can be cached per game as they are reusable. The cache should
always be thrown out when the game changes, and may be discardable at any other time. The cache is just a
wrapper around the main virtual memory system, so use Virtual_PoolFree to discard it.
========================================================================================================================
*/
typedef struct cacheobject_s
{
struct cacheobject_s *next;
void *data;
char *name;
} cacheobject_t;
cacheobject_t *cachehead = NULL;
int numcacheobjects = 0;
void *Cache_Check (char *name)
{
for (cacheobject_t *cache = cachehead; cache; cache = cache->next)
{
// these should never happen
if (!cache->name) continue;
if (!cache->data) continue;
if (!stricmp (cache->name, name))
{
Con_DPrintf ("Reusing %s from cache\n", cache->name);
return cache->data;
}
}
// not found in cache
return NULL;
}
void *Cache_Alloc (char *name, void *data, int size)
{
cacheobject_t *cache = (cacheobject_t *) Virtual_PoolAlloc (VIRTUAL_POOL_CACHE, sizeof (cacheobject_t));
// alloc on the cache
cache->name = (char *) Virtual_PoolAlloc (VIRTUAL_POOL_CACHE, strlen (name) + 1);
cache->data = Virtual_PoolAlloc (VIRTUAL_POOL_CACHE, size);
// copy in the name
strcpy (cache->name, name);
// count objects for reporting
numcacheobjects++;
// copy to the cache buffer
if (data) memcpy (cache->data, data, size);
// link it in
cache->next = cachehead;
cachehead = cache;
// return from the cache
return cache->data;
}
void Cache_Init (void)
{
cachehead = NULL;
numcacheobjects = 0;
}
/*
========================================================================================================================
VIRTUAL POOL BASED MEMORY SYSTEM
This is officially the future of DirectQ memory allocation. Instead of using lots of small itty bitty memory
chunks we instead use a number of large "pools", each of which is reserved but not yet committed in virtual
memory. We can then commit as we go, thus giving us the flexibility of (almost) unrestricted memory, but the
convenience of the old Hunk system (with everything consecutive in memory).
========================================================================================================================
*/
typedef struct vpool_s
{
char name[24];
int maxmem;
int lowmark;
int highmark;
byte *membase;
} vpool_t;
// these should be declared in the same order as the #defines in heap.h
vpool_t virtualpools[NUM_VIRTUAL_POOLS] =
{
// stuff in this pool is never freed while DirectQ is running
{"Permanent", 32 * 1024 * 1024, 0, 0, NULL},
// stuff in these pools persists for the duration of the game
{"This Game", 32 * 1024 * 1024, 0, 0, NULL},
{"Cache", 256 * 1024 * 1024, 0, 0, NULL},
// stuff in these pools persists for the duration of the map
// warpc only uses ~48 MB
{"This Map", 128 * 1024 * 1024, 0, 0, NULL},
{"Edicts", 10 * 1024 * 1024, 0, 0, NULL},
// used for temp allocs where we don't want to worry about freeing them
{"Temp Allocs", 128 * 1024 * 1024, 0, 0, NULL},
// spare slots
{"Unused", 1 * 1024 * 1024, 0, 0, NULL},
{"Unused", 1 * 1024 * 1024, 0, 0, NULL},
{"Unused", 1 * 1024 * 1024, 0, 0, NULL},
{"Unused", 1 * 1024 * 1024, 0, 0, NULL}
};
void *Virtual_PoolAlloc (int pool, int size)
{
if (pool < 0 || pool >= NUM_VIRTUAL_POOLS)
Sys_Error ("Virtual_PoolAlloc: Invalid Pool");
vpool_t *vp = &virtualpools[pool];
// if temp file loading overflows we just reset it
if (pool == VIRTUAL_POOL_TEMP && (vp->lowmark + size) >= vp->maxmem)
{
// if the temp file pool is too small to hold this allocation we reset it so that it's big enough
if (size > vp->maxmem)
Virtual_PoolReset (pool, size);
else Virtual_PoolFree (VIRTUAL_POOL_TEMP);
}
// not enough pool space
if ((vp->lowmark + size) >= vp->maxmem)
Sys_Error ("Virtual_PoolAlloc: Overflow");
// only pass over the commit region otherwise lots of small allocations will pass over
// the *entire* *buffer* every time (slooooowwwwww)
if ((vp->lowmark + size) >= vp->highmark)
{
// alloc in 256k batches
int newsize = MemoryRoundSize (vp->lowmark + size, 256);
if (!VirtualAlloc (vp->membase + vp->lowmark, newsize - vp->lowmark, MEM_COMMIT, PAGE_READWRITE))
Sys_Error ("Virtual_PoolAlloc: VirtualAlloc Failed");
vp->highmark = newsize;
}
// set up
void *buf = (vp->membase + vp->lowmark);
vp->lowmark += size;
return buf;
}
void Virtual_PoolReset (int pool, int newsizebytes)
{
// graceful failure
if (pool < 0 || pool >= NUM_VIRTUAL_POOLS) return;
vpool_t *vp = &virtualpools[pool];
// fully release the memory
if (vp->membase) VirtualFree (vp->membase, 0, MEM_RELEASE);
// fill in
vp->lowmark = 0;
vp->highmark = 0;
vp->maxmem = MemoryRoundSize (newsizebytes, 1024);
// reserve the memory for use by this pool
vp->membase = (byte *) VirtualAlloc (NULL, vp->maxmem, MEM_RESERVE, PAGE_NOACCESS);
}
void Virtual_PoolFree (int pool)
{
// graceful failure
if (pool < 0 || pool >= NUM_VIRTUAL_POOLS) return;
// already free
if (!virtualpools[pool].lowmark) return;
// decommit the allocated pool; if it straddles a page boundary the full extra page will be freed
VirtualFree (virtualpools[pool].membase, virtualpools[pool].lowmark, MEM_DECOMMIT);
// reset lowmark
virtualpools[pool].lowmark = 0;
virtualpools[pool].highmark = 0;
// reinit the cache if that was freed
if (pool == VIRTUAL_POOL_CACHE) Cache_Init ();
}
void Virtual_PoolInit (void)
{
for (int i = 0; i < NUM_VIRTUAL_POOLS; i++)
{
// skip over any pools that were already alloced
if (virtualpools[i].membase) continue;
// reserve the memory for use by this pool
virtualpools[i].membase = (byte *) VirtualAlloc (NULL, virtualpools[i].maxmem, MEM_RESERVE, PAGE_NOACCESS);
}
// init the cache
Cache_Init ();
}
/*
========================================================================================================================
ZONE MEMORY
The zone is used for small strings and other stuff that's dynamic in nature and would normally be handled by
malloc and free. It primarily exists so that we can report on zone usage, but also so that we can avoid using
malloc and free, as their behaviour is runtime dependent.
The win32 Heap* functions basically operate identically to the old zone functions except they let use reserve
virtual memory and also do all of the tracking and other heavy lifting for us.
========================================================================================================================
*/
typedef struct zblock_s
{
int size;
void *data;
} zblock_t;
int zonesize = 0;
HANDLE zoneheap = NULL;
void *Zone_Alloc (int size)
{
// create an initial heap for use with the zone
// this heap has 128K initially allocated and 32 MB reserved from the virtual address space
if (!zoneheap) zoneheap = HeapCreate (0, 0x20000, 0x2000000);
size += sizeof (zblock_t);
size = (size + 7) & ~7;
zblock_t *zb = (zblock_t *) HeapAlloc (zoneheap, HEAP_ZERO_MEMORY, size);
zb->size = size;
zb->data = (void *) (zb + 1);
zonesize += size;
return zb->data;
}
void Zone_Free (void *ptr)
{
// attempt to free a NULL pointer
if (!ptr) return;
if (!zoneheap) return;
// retrieve zone block pointer
zblock_t *zptr = ((zblock_t *) ptr) - 1;
zonesize -= zptr->size;
// release this back to the OS
HeapFree (zoneheap, 0, zptr);
HeapCompact (zoneheap, 0);
}
/*
========================================================================================================================
REPORTING
========================================================================================================================
*/
void Virtual_Report_f (void)
{
int reservedmem = 0;
int committedmem = 0;
Con_Printf ("\n-----------------------------------\n");
Con_Printf ("Pool Highmark Lowmark\n");
Con_Printf ("-----------------------------------\n");
for (int i = 0; i < NUM_VIRTUAL_POOLS; i++)
{
// don't report on empty pools
if (!virtualpools[i].highmark) continue;
Con_Printf
(
"%-11s %7.2f MB %7.2f MB\n",
virtualpools[i].name,
((float) virtualpools[i].highmark / 1024.0f) / 1024.0f,
((float) virtualpools[i].lowmark / 1024.0f) / 1024.0f
);
reservedmem += virtualpools[i].highmark;
committedmem += virtualpools[i].lowmark;
}
Con_Printf ("-----------------------------------\n");
Con_Printf
(
"%-11s %7.2f MB %7.2f MB\n",
"Total",
((float) reservedmem / 1024.0f) / 1024.0f,
((float) committedmem / 1024.0f) / 1024.0f
);
Con_Printf ("-----------------------------------\n");
Con_Printf ("%i objects in cache\n", numcacheobjects);
Con_Printf ("Zone size: %i KB\n", (zonesize + 1023) / 1023);
}
// also alloc the old heap_report command in case anyone used it
cmd_t virtual_report_cmd ("virtual_report", Virtual_Report_f);
cmd_t heap_report_cmd ("heap_report", Virtual_Report_f);

_________________
DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines

Back to top |

reckless Joined: 24 Jan 2008 Posts: 390 Location: inside tha debugger |

Posted: Sun Aug 16, 2009 11:47 pm Post subject: |

oh cool ill give it a try cheers.

Back to top |

mh Joined: 12 Jan 2008 Posts: 910 |

Posted: Mon Aug 17, 2009 8:42 am Post subject: |

A few other things you can do is to keep the number of pools small and allocate to each in batches. I have a 256 K batch size hard-coded into here ("int newsize = MemoryRoundSize (vp->lowmark + size, 256);") but with a smaller number of memory pools that could be easily increased to 1 MB. What this will mean is that instead of committing memory on a regular basis it's only committed in discrete 1 MB chunks, meaning about 7 or 8 commits per map (as opposed to potentially several thousand if you just commit memory as required). A smaller number of pools means less batch size overhead; with 6 pools and a 256 K batch the most "wasted" memory you'll ever have is 1.25 MB.
My current incarnation uses only 6 pools - permanent, this game, cached models and sounds, this map, edicts and temp. Cached objects can be easily amalgamated with this game, but I want to keep it separate so that I can demand-flush it. You could cvar-ize the commit batch size if you wanted; I'm in 2 minds about this one, it's nice to expose it but should regular users be able to mess with something fairly low-level like that?
Precalculating sizes and allocating in bulk is always good too. For example, you can figure out in advance how many glpoly_t structs (and the sizes of their vertexes) you need for non warp-surfaces, so why not just allocate a single big buffer at the end of Mod_LoadFaces and write the polys into that instead of doing a separate allocation for each poly?
You'll notice that I have fixed hard-coded size limits on the pools here, but that's OK. warpc only needs 48 MB from the "this map" pool (or about 85 MB if you don't cache models and sounds) so a 128 MB upper limit is fine. You could increase it to 256 or even higher if you want; because it's only reserving an area of virtual memory for itself there's no actual resource utilization involved in doing so (just make sure that you don't reserve more than 2 GB total though!)
Anyway, I'm getting into areas more appropriate for the Engine Coding Tips thread now, so I think I'll leave it at that!
_________________
DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines

Back to top |

mh Joined: 12 Jan 2008 Posts: 910 |

Posted: Mon Aug 17, 2009 5:34 pm Post subject: |

BUG: Should be highmark, not lowmark in the VirtualFree call in Virtual_PoolFree _________________ DirectQ Engine - New release 1.8.666a, 9th August 2010 MHQuake Blog (General) Direct3D 8 Quake Engines

Back to top |

reckless Joined: 24 Jan 2008 Posts: 390 Location: inside tha debugger |

Posted: Mon Aug 17, 2009 9:01 pm Post subject: |

check thanks for pointing out

Back to top |

mh Joined: 12 Jan 2008 Posts: 910 |

Posted: Mon Aug 17, 2009 10:37 pm Post subject: |

Yr welcome. This one should also be safe for making sure that leafs and nodes are allocated in consecutive memory (very important for node = (mnode_t *) &cl.worldmodel->leafs[i + 1]; to work right). _________________ DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines

Back to top |

Baker Joined: 14 Mar 2006 Posts: 1538 |

Posted: Tue Dec 29, 2009 2:27 pm Post subject: |

@MH re: memory manager Is Windows specific? If not, is road tested and debugged? Just wondering. I hate Quake's memory management but also fear changing it.

Back to top |

mh Joined: 12 Jan 2008 Posts: 910 |

Posted: Tue Dec 29, 2009 6:31 pm Post subject: |

Windows specific I'm afraid. I don't know what the virtual memory APIs for other OSs are, but anyone who does should be able to port it quite easy. Probably the biggest challenge would be dealing with Zone memory - Windows has so many memory APIs that I suspect there may not be an equivalent. You could just replace it with malloc and free and I suppose it would work (be careful though of my Ninja trickery in Zone_Free...)
As for road tested, it's pretty much the one I use in DirectQ so it's definitely been let out in the wild for a good while.
But I agree though, tackling the Quake memory manager is not a trivial task. Cache, Hunk Low, Hunk High, Zone - AAARRGGHHHH!!! Have you looked at the Q2 source code? That has another memory management system that should be easier to port and that also has quite good flexibility.
_________________
DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines

Back to top |

reckless Joined: 24 Jan 2008 Posts: 390 Location: inside tha debugger |

Posted: Tue Dec 29, 2009 10:29 pm Post subject: |

VirtualAlloc exist on linux also so should be fine

Back to top |

mh Joined: 12 Jan 2008 Posts: 910 |

Posted: Tue Dec 29, 2009 10:34 pm Post subject: |

reckless wrote:
VirtualAlloc exist on linux also so should be fine

Sounds good. Anyone in a position to test stuff?
_________________
DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines

Back to top |

reckless Joined: 24 Jan 2008 Posts: 390 Location: inside tha debugger |

Posted: Wed Dec 30, 2009 4:47 pm Post subject: |

ill see what i can do i belive VirtualAlloc support was added later looking at things i see several different support models the original i think is mmap/munmap which btw can be emulated with (yes exactly that) VirtualAlloc/VirtualFree. funny thing is i just wrote several wrappers for some programs i was porting to my mingw compiler suite and one of them actually emulates the above.
#if defined(__MINGW32__) && !defined(QUICK)
#include <windows.h>
#define QUICK 1
#define MAP_FAILED 0
#define MREMAP_FIXED 2 // the value in linux, though it doesn't really matter
// These, when combined with the mmap invariants below, yield the proper action
#define PROT_READ PAGE_READWRITE
#define PROT_WRITE PAGE_READWRITE
#define MAP_ANONYMOUS MEM_RESERVE
#define MAP_PRIVATE MEM_COMMIT
#define MAP_SHARED MEM_RESERVE // value of this #define is 100% arbitrary
// VirtualAlloc is only a replacement for mmap when certain invariants are kept
#define mmap(start, length, prot, flags, fd, offset) \
( (start) == NULL && (fd) == -1 && (offset) == 0 && \
(prot) == (PROT_READ|PROT_WRITE) && (flags) == (MAP_PRIVATE|MAP_ANONYMOUS) \
? VirtualAlloc(0, length, MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE) \
: NULL )
#define munmap(start, length) (VirtualFree(start, 0, MEM_RELEASE) ? 0 : -1)
#endif // HAVE_MMAP
hmm time to test out windows 7 virtual machines with my suse distro

Back to top |

reckless Joined: 24 Jan 2008 Posts: 390 Location: inside tha debugger |

Posted: Wed Jan 06, 2010 6:45 am Post subject: |

eh ? i just wrote a ton of the codechanges to the pk3 tutorial and everything just dissapeared ? maybe it was to big for one page ?

Back to top |

r00k Joined: 13 Nov 2004 Posts: 483 |

Posted: Wed Jan 06, 2010 8:35 am Post subject: |

've noticed lately that as you type a [post a reply box] full and press preview sometimes it asks me to login and then poof no more message :O <- back cut/paste -> ins continue

Back to top |

Display posts from previous:

| Inside3d Forums Forum Index -> Programming Tutorials | All times are GMT Goto page 1, 2 Next
Page 1 of 2

Jump to:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

Powered by phpBB © 2004 phpBB Group