I've just been putting some work into making DirectQ behave itself better with WDDM drivers. This is the driver model used by Windows Vista and 7, and is restricted to cards that are capable of running the Aero desktop. There are a few considerations with these drivers which don't apply otherwise, but in practice they can be implemented anyway.
Firstly it seems as though textures don't preload to the GPU properly. It's necessary to do a single full screen update at the end of R_NewMap to get textures loaded from system RAM into video RAM, otherwise in some circumstances the first frame of a map will look very strange indeed (partially drawn with lots of grey holes in it) and will stall the engine while textures are being demand-loaded. I'm going to extend this concept to doing a render of each texture to an offscreen buffer so as to ensure that all textures needed are kept properly "hot".
Secondly, and I'm of the opinion that this is at least semi-evil, WDDM drivers have a "stall detection" feature. If the runtime thinks that a graphics driver has hung, it will reset the device automatically. This is all fine and dandy, but the stall timeout is a mere 2 seconds. Now, and it's a matter of debate whether this is Quake's fault or not, but when loading a map Quake does not update the screen. With larger maps, and in general when loading the first map after Quake starts, the time period during which the driver is idle may be longer than 2 seconds. This will trigger the stall time out (in fact MS advise that anything running at less than 10 screen updates per second might trigger the timeout).
To work around this I've devised a function that detects how long it's been since the last screen update and triggers one if this time period is passed (if we're already in the middle of a screen update it silently does nothing). Right now I have it being called from various strategic places in the code, but I'm going to put it into it's own thread so that it's constantly running. The thread will be started suspended, kicked off by the first model load, and suspended again after all loads are completed so that it doesn't interfere with normal operation. I'm looking for a reasonably benign operation I can perform on the device that will go through to the hardware layer but won't affect what's on-screen.
Speaking of threads, I've recently been giving thought to a multithreaded DirectQ. Right now on a 2 or more core machine DirectQ just uses a single core, which is obviously not taking full advantage of the hardware's capabilities. Thinking about the logical separation points for threads, it occurs to me that the most obvious is the client/server split. Others include sound and possibly graphics and input (to ensure that input resolution is not affected by framerate). I'd consider running the server thread at a steady 72 (or whatever your value of host_maxfps is) FPS, so that we would get framerate independent physics.
I haven't even written any exploratory code for this yet, so don't expect major updates in that direction for a while. It would be a fairly large task, and would involve identifying and finding satisfactory solutions to any client/server crossover points (the server would definitely need it's own copy of the models, although they could be stripped back a lot from what the client has).
Thursday, December 31, 2009
WDDM Drivers and Multithreading
Posted by
mhquake
at
8:50 PM
0
comments
The Joy of 7-Zip
I've just discovered that 7-Zip can extract ISOs. This program gets better and better all the time. It's an astonishing wee tool, and must rank as probably my favourite Free software. Nothing dramatic about it, very unassuming, but it's just something I get genuine benefit from almost every day.
Wonder if it can do Quake PAKs...?
Posted by
mhquake
at
11:13 AM
0
comments
Wednesday, December 30, 2009
Fun with Water
I think the basics are getting towards being complete now. I've just done water in both HLSL and non-HLSL modes. The non-HLSL mode has been accelerated somewhat as it's possible to draw the entire surf (all subdivisions) with a single call, so low values of gl_subdivide_size should have lessened negative impact on performance. I recommend you start with 24 and only work upwards if you really feel you really need to.
I've decided to tune the triangle list generation a little, so on the first startup DirectQ will examine various properties of your 3D card and make some assumptions, following which it will set the maximum batch size it generates. This will be based on factors such as whether you support pixel shaders, whether you're a hardware T&L device, and how much video RAM you have. It's also clamped to the max batch sizes your card can support. Values I've chosen are purely arbitrary and based on rough guess work rather than anything grounded in reality.
Following that you can always override it and you're overrided value will take precedence. You'll use the r_trilistsize cvar to control this; higher values give larger batches but they might choke cards with low bandwidth; lower values give smaller batches but they won't handle complex scenes as smoothly. The value is approximately equal to the highest number of vertexes DirectQ allows in a single batch. I don't allow values over 65535 because I don't want to mess with supporting both 32 bit and 16 bit indexes right now. To recover the autotuned value set it to -1. You'll need to reload the map after each change, so test a bit first.
Setting it to 1 will revert to Quake's standard behavior which you don't really want to do - even ID1 maps can benefit from better batching. The option is there if you feel that you really need it however.
The real ugliness is yet to come, which is support for alpha surfaces and entities. I've been playing with this on and off, and have a basic framework in place, but overall though it's going to break my carefully put together code a little. It's all for the best though as what will come out of it is a nice generalised and extensible setup.
Posted by
mhquake
at
2:07 AM
0
comments
Tuesday, December 29, 2009
Fun with Sky
I've just moved sky across to the new "no texturechains, no glpoly_t" renderer. Sky is a little more complex than regular brush surfs as it needs to be drawn before the entire world and everything in it, so I've needed to do something disgusting to get it working OK. I'll probably be fine-tuning this as time goes on and at least looking to integrate it into the same setup as brush surfs (although it will still need to be drawn first). The intention is to meet somewhere in the middle; the current setup is rather volatile at the moment and liable to change from time to time.
Part of the grief here is instanced brush models: ammo boxes/etc. It's easy to see why ID got rid of them in subsequent engines; integrating them into the main render is just a whole barrel of hassle. Things are more difficult once you realise that you need to support sky and water surfs on them (which I do). The solution I currently have is workable for sure, but has room for improvement in code cleanliness and maintainability (wait till we get on to alpha-test surfs in mods and it will need rewriting; I prefer to at least get it so that I can have less to do then).
Of course one solution is to just do it the way GLQuake does and be damned. I don't really see that as being anyway optimal though; there are a lot of things badly wrong with the way GLQuake does things and that's the reason why Quake performs badly in certain setups.
I think I'm going to get a new laptop soon. I haven't really treated myself to a nice Christmas present yet, and the old HP dv1000 I do most of my DirectQ work on is showing it's age badly (it's gonna be 4 shortly). I still intend using it as a secondary test platform just to ensure that frame rates are maintained well enough on older machines, but compile times are starting to irritate (especially now that I'm changing headers around so much). It'll make a nice web browsing machine, and be handy to bring around when I'm travelling as I won't need to care so much if it gets bashed about a little.
Posted by
mhquake
at
10:38 PM
1 comments
Fun with Lightmaps
The "tall but narrow" lightmap approach worked perfectly. Lightmaps are now by default 32 texels wide (they can expand up to your max texture size if a surf needs it) and your max texture size high. I would have gone for 16 wide but most ID1 maps require 32 at some point, so 32 it is. This lets me pack in a LOT of surfaces sharing the same texture into the same lightmap, meaning better batching and less state changes.
They are also updated directly as modifications come in, rather than having the update deferred until all modifications are done, then bulk-updated. The latter can be faster in optimal cases, but the former avoids having a large dirty rectangle, much of which might not have been modified at all, so in the real world it's quite a bit faster.
Texture chains are dead and have been removed from the engine. Texture sorting for each model now only happens at load time; an initial qsort of surfaces by texture, then I have my filthy way with them (allocating lightmaps and triangle lists in texture order), then I qsort them again by number to put them back into the correct order.
The tall-but-narrow approach should also be viable in GLQuake. Just change some #defines, say BLOCK_WIDTH goes to 64 and BLOCK_HEIGHT to 256 and see what happens. You could also try 32 and 512 if you're not concerned about supporting 3DFX cards.
Posted by
mhquake
at
6:42 PM
0
comments
More fun with Brushmodels
I've been cleaning up and generalising my brushmodel code and now have it at a stage where the only real difference between brushmodels and the world is visible surface determination. I have a really nice single function that is used to handle all brush model rendering, and the code is clean, elegant, and makes me happy.
Texture chains are gone. They are only used for an initial sort by texture at load time now (and I'm hoping to get rid of even that). Instead there's a linked list of triangle list definitions attached to each brush model (which is allocated in texture order) with each surf possessing it's own pointer to the triangle list that contains it's verts (the old glpoly_t is almost gone too). An array of indexes for visible surfaces into this list is generated at runtime, then each entire list is blasted down to the GPU (as indexed primitives, using the array). I can draw potentially hundreds of surfaces (in practice it's more like tens, but you get the picture) in a single call using this method, which is friendlier to the API and friendlier to (reasonably) modern GPUs.
r_speeds counts are totally meaningless here. Even the busiest scenes tend to top out at a wpoly count of about 60, which equates to the number of DrawPrimitive calls used (the equivalent of a glBegin/glEnd pair). The long and winding road to the castle in Marcher is the record setter with just under 100 (this can still slow down on slower/older GPUs, but that's due to the hardware and not code inefficiency). epoly counts are strictly 1 per alias model as each alias model is drawn with a single DrawPrimitive (it will be 2 when I implement the 1 TMU path for these).
I decided to stop being paranoid about batching textures up according to whether they have fullbrights or not, and it's paid off. In practice the better drawing efficiency far outweighs the overhead from state and texture changes, so on balance it's a speed increase. It could go faster if I did batch them, but only slightly so, and it would be at the expense of adding complexities to the code, which would cause trouble for future maintainability.
One compromise I've had to make has been with lightmaps. Better drawing efficiency can be had by having as many surfaces as possible sharing both the same texture and the same lightmap (bigger batches, less texture changes, so on). For previous releases of DirectQ I'd found that the optimal lightmap size was 64: smaller and you're uploading too many lightmaps at runtime, larger and you're uploading huge unchanged chunks of a lightmap for a handful of dynamic surfaces. To get better batching I've had to increase this to 256. This means that dynamic lights have a bigger speed impact than I'd like, so I'm considering how to work around this. Obviously we don't know enough info in advance to do something like alloc lightmaps big enough or small enough for each texture batch, so clever trickery will be required (if it even happens at all).
One idea is to just upload the lightmap for each surf as it changes. A second is to use tall but narrow lightmaps, as lightmap width is a major factor here. I think a combination of these will get results, but we'll see.
Till next time.
Posted by
mhquake
at
1:18 PM
0
comments
Monday, December 28, 2009
Speeds up again (this is getting boring now)
This engine is going to totally annihilate the old cliche that "Quake and/or BSP can't handle large open or complex scenes efficiently". In fact, if it were anyway near releasable I'd release it now, but soon enough you're going to see it in action for yourself.
Case in point: the screenshot below illustrates a scene that Quake normally has serious difficulty with.
This scene had dragged an Athlon 64 X2 Dual Core 4200+ 2.21ghz, 3gigs of ram, and a GeForce8800 GTS 640 meg down to framerates in the 20s. Better primitive batching in DirectQ now gives me framerates in the 60s on a "consumer multimedia" laptop with an Intel 965!
Posted by
mhquake
at
11:45 AM
5
comments
Fun with Brush Models
I've been working on optimizing brush model rendering as per my previous post, and have something good coming along. As with alias models, it's a complete change of the in-memory format to something that's laid out in a more optimal fashion for rendering from. This time I was able to do indexed triangle lists quite effectively with the end result being that even the central arena in ctf1rq now runs very smoothly.
The setup code is a little nightmarish! I don't recommend that anyone working on their own engine just copy and paste this in without reading it carefully and fully understanding what's going on.
There's more optimization to come here too, as I can reduce the number of state changes required fairly well, so we're going to be in a situation where large complex scenes with lots of brush models (or with complex brush models, or a mixture of both) don't slow down as much (they can be over twice as fast as before).
Interestingly, I had an experimental version that used hardware vertex buffer objects but it didn't run as well; likely because the format of brush models necessitated a lot of changes between buffers. This bears on my own previous experiments with VBOs in the regular renderer, where they just didn't give that substantial a performance increase (even back when I had the entire world in a single VBO I only got about a 5% to 10% gain, at the expense of complexity, setup, loading times and loss of flexibility). At the end of the day it's down to what format and usage is most optimal for the data you have, and for Quake VBOs are firmly in the "not such a big deal" arena.
Undoubtedly if I had fully static geometry all of which shared the same textures the situation would be different.
Posted by
mhquake
at
2:03 AM
0
comments
Sunday, December 27, 2009
DirectQ 1.8 Update
It's been a short while since I posted a solid update, rather than just providing info on what I'm currently working on, so here it is. I'm dropping the bullet-point list format I usually use for these things as it tends to repeat itself over time.
The 2 main things I had in mind for 1.8 were the restructured render and alias model cleanup. Of these, alias model cleanup is done and the restructured render is still ongoing.
I've posted before on alias model cleanup, but in summary it rearranges the in-memory alias model format a good deal to enable alias models to be stored more efficiently and loaded and rendered much faster. All alias models can now be rendered with a single API call which gives better primitive batching and follows the general principle of doing very few big things being more performant that doing lots of small things (the same applies to memory allocations, disk reads and writes and anything else that interfaces with hardware). As a bonus the code is a lot cleaner and nicer to work with too.
The restructured render is close to being done as well. All that really remains is adding support for r_wateralpha modes. There is a slight dilemma here that ties into my alpha model support. If a brush entity that contains liquid surfs has an alpha attribute set, which should take precedence? The value of r_wateralpha or the alpha attribute? The problem here is that there are some mods that rely on it one way and some that rely on it the other. My instinct is to follow what GLQuake did and just set liquid surf alpha based on the value of r_wateralpha (following the principle of least surprise towards the player, as opposed to what the modder might want, which - as I've mentioned - can be mutually incompatible between different mods).
This renderer is also optimized a LOT more highly than before, allowing in many cases an entire texture chain to be submitted in a single API call (it might be more if the lightmap changes within a texture chain). Again, there are performance improvements to be had from better primitive batching here, but they're not as dramatic as I got for alias models (mostly because alias models had more room for improvement, but also because surfs sharing the same texture can't be batched as efficiently owing to lightmap changes; I do alloc lightmaps on pre-sorted surfs but it can still happen).
When this is done - or maybe before - I'm going to work on optimizing brush models some more. Brush models are currently horrible in Quake as there are two different types and they - in some cases - need different handling. The objective is to put the engine in a position where brushmodels can also be rendered in as few API calls as possible.
I wish I could say that this new renderer is as clean as the alias model setup, but it's not. This is really a legacy of Quake having so many different formats for everything (whereas with alias models you're just dealing with one format). Anything I can do to tidy it up will be done, and I suspect that even when it's "done" it won't be "finished".
The upshot of these improvements is that scenes in currently released mods that used to drag this engine (and most other engines) to a standstill can now run a lot more smoothly. The rain in the Remake Quake summer SP demo is one example (FPS used to drop to below 20, it's now quite steady at 50 to 60). The open area in CTF1RQ is another; again something that used to drop to below 20 FPS is steady at about 35 (this will increase when I do brush model optimization).
There are some other improvements coming in aside from these two big ones. Baker mentioned that I didn't support "connect host:port" so now I do. Recording demos while a server is running is another one that will happen (although I suspect that there are reasons why this was removed from ID Quake, and I'd love to know what they were). sv_progs support is done, and most of the handling of the progs.dat file has been moved over to a C++ class (in prep for CSQC, although that won't happen with 1.8 - but it's coming). Some general bugfixes, code tidy-ups, removal and clean-up of some old code that dates back to 1.0 but had never been touched since, and so on.
All in all this is shaping up to be a good one.
Posted by
mhquake
at
6:50 PM
2
comments
Saturday, December 26, 2009
Fun with Alias Models (Speeds up yet again)
This is a big one...
I've completely rewritten the alias model tri fan/strip generator to just use raw triangle lists. The code is now a whole load simpler, being just a single 12-line function (compare with the old gl_mesh.c - yes, I've replaced the entire thing with a 12-line function).
The speedup is something in the order of 20% to 30%. No joking, this one goes like a rocket. The big advantage obviously comes from being able to batch-submit an entire alias model in a single API call, and it has me thinking that I can get similar goodness from the world (it will be slightly more difficult owing to changing textures).
An earlier version retained the tri fans and strips but submitted them as a single indexed tri list, which was slightly faster again (less data in the submission). I've decided to prefer the slightly slower (by 1% or 2%) version I currently have as the code is simpler, cleaner, more maintainable, and faster to load. Indexed primitives also impose a hard limit on the number of verts in a model as some 3D cards still only support 16 bit indexes.
There is a slight memory overhead from not reusing some data, but it's relatively minor and anyway I have room to play with there as a result of keeping the vertex data indexed in memory (as opposed to expanding it out like GLQuake did).
Update:
This Christmas has been good to DirectQ. I've just switched over the main world texturechains to a similar triangle list based render, and gained some more speed. The complication is that there are two textures in each chain: the world texture and the lightmap. This makes it necessary to submit the current list and begin a new one each time either of them changes, so it can't be as optimal as alias models.
For reference, the machine I'm working on now used to get 105 FPS on demo1. It now hits near 140. For kicks I did a benchmark with r_novis 1 and got 80, so this puts us in a position where the impact of Quake's polygon fragmentation is significantly reduced.
Posted by
mhquake
at
8:12 AM
1 comments
Thursday, December 24, 2009
Fun with Particles
I moved particles across to my new render recently, and so far have hit a slow down with them. Right now though it's sub-optimal and definitely not the final code, so there is room for improvement. I have 2 main ideas that should prove effective - one relating to alpha depth sorting, the other relating to more optimal batching of particles, that will definitely bear fruit. (Update: done).
For the season that's in it there won't be any updates for a short while, so let me wish all my readers and everyone who's downloaded my engine, played it, given me good feedback, or nagged me to get a feature in, a happy Christmas/Holiday/Solstice/Larentalia/Emperor's Birthday or whatever! You've all contributed to make this engine an enjoyable experience for me to develop.
Posted by
mhquake
at
12:33 PM
2
comments
Tuesday, December 22, 2009
Speeds Up Again
The earlier version of the SDK seems to run more efficiently with the way DirectQ is coded, so there's a few extra frames from using it. I'm not quite certain what causes it, but I strongly suspect it's the D3D9 shader compiler in the earlier version vs the D3D10 (even on XP) shader compiler in recent versions.
Sky has just gone in to the new rendering framework, which is increasingly turning out to be very nice indeed. DirectQ can now properly handle sky surfs on ammo boxes, including proper depth clipping for non-hlsl modes (which includes sky boxes).
I can think of at least one situation where this might be useful, which is a sky room entity stored as a BSP model. Done well on the QC side you could in theory have changing skies in a single map. This is one for mappers and modders to explore if they want.
A report of slowdowns in certain busy scenes and I'm almost certain that this is particles. The DirectQ particle engine is fairly well optimized, but I've completely removed the upper limit on the number of particles. You could throw 100,000 particles at it and it will quite happily set them up and render them - slowly, but happily. ID Quake only lets you have 2,048 particles on-screen at any time, and I think I'm going to add a cl_maxparticles cvar to let you have some control over this. For comparison, of the ID demos, demo1 is the particle monster (yes, I've counted the number of particles in each demo) and it tops out at less than 4,000.
This is one situation where more widespread availability of D3D10 and geometry shaders would have been great, as particles - being totally dynamic - need to send quite a lot of data to the GPU each frame. In fact, of the entire scene, it's perfectly possible for the overhead from particles to be a number of times higher than everything else combined.
I have a totally crazy idea brewing in the back of my mind that, if I can pull it off, might just turn out to be something of a Holy Grail for Quake engine games. Right now I haven't even written any exploratory code, never mind proof of concept or production quality, so I'm not willing to go into much detail for fear that I might generate some false expectations (it might not even turn out to be that big a deal, depending on how it works in practice).
Imagine huge outdoor scenes with massive draw distances but no real slow down. That's what I'm hoping to aim towards. Even if I do start coding something I don't think it's likely to be a 1.8 feature; but - depending on how it goes - it could happen soon-ish.
Posted by
mhquake
at
11:50 PM
6
comments
More Nice Changes coming in
I've switched DirectQ back to an older version of the D3D9 SDK (September 2004).
The big thing about this is that it will no longer be required to upgrade your DirectX on Vista and Windows 7, and older D3D9 installs on XP should also work fine.
With hindsight I should have done this a long time ago, but better late than never.
Posted by
mhquake
at
4:44 PM
0
comments
DirectQ Speeds Up
DirectQ 1.8 is now faster on Windows 7 that 1.7.3 was on Windows XP. We're still not quite back to 1.6.x speeds, but we're inching closer each day.
This is all a side effect of the new rendering code I've been doing. I've solved several major problems, including efficient and effective state batching, handling multiple different types of input formats, multiple entities sharing the same model, and merging model rendering into the world without unpleasant side effects.
As an added bonus the code is really clean and elegant, and nice to work with. It should be possible to port this virtually unchanged (aside from the D3D API specific stuff) to just about any engine and see immediate benefit. Of course it's also reliant on the Windows API VirtualAlloc function, but if your chosen OS has a functional equivalent it Can be Done.
I'm really enjoying the work I'm doing on DirectQ at the moment, far more than at any time for a long time. It's a rare thing in any development project to be getting immediately measurable results from nice easy simple code, and also to be throwing out horrible old legacy code while doing so, and also to know that what you're getting is far more functional and robust than what you had before, but this one just keeps on giving.
I wouldn't make any claim to be particularly talented here; it was a happy accident that I stumbled across the right way, and there were 3 aborted versions of the main code along the way here. But things are on track and moving towards a good end result.
Posted by
mhquake
at
12:46 AM
0
comments
Monday, December 21, 2009
DirectQ Bug Alert!
Two bugs in current versions of DirectQ have recently come to light.
The first one is that all brush models are being drawn twice. I had suspected that there was something aberrant in my code as I had a small but significant performance decrease when I made the original 1.7; I had been putting it down to the overhead of state changes from the fixed render path. This was discovered when I had removed some of the brushmodel rendering code during the construction of my new renderer (which is coming along quite nicely).
Fixing it would involve taking a whole heap of code that is already a mess and making an even bigger mess of it. This will have to wait on 1.8, which will be a lot cleaner and more flexible (did I say it's coming along quite nicely?)
The second is a crash bug. It just happens that if when - and only when - you start up DirectQ and enter a map for the first time (and it's only on the first map you enter after starting the engine) there are no alias models in the view aside from the gun, DirectQ will crash.
This is pretty obscure; for a whole load of people the first map is e1m3 - the Necropolis demo - which has a nice big whopping torch in the view to prevent this. The majority of other first-run maps will also have some kind of models visible.
It's a one-line fix, but right now I'm not going to. I haven't had anyone report such a crash yet, so for now the fix is to not modify your autoexec to skip the Necropolis demo.
Regarding 1.8, the sands have shifted a little. My original plan was to finish off my D3D8 port of QRack and then start on 1.8, but nobody can predict the way things will turn out. 1.8 is actually well underway, and D3D8 QRack is now on the backburner.
Apologies to anyone who was looking forward to that.
The good news however is that I'm starting to see 1.8 as happening relatively soon. What I'm currently planning on releasing will primarily contain just the restructured renderer and the alias model cleanup. These two are both significant enough to justify the version number bump, and overall it seems to be the path of wisdom to release a less radically developed engine; it means fewer places to go looking for bugs, after all.
Posted by
mhquake
at
6:28 PM
0
comments
Happy Birthday to the Quake Source Code!
This is the major anniversary I wrote about a coupla days ago. It's now 10 years since the Quake source code was released. 10 years, imagine that. Not only is the game itself still going reasonably strong, but engine mods are arguably stronger than ever before these days - there may be fewer, but the stuff that's being done is well ahead of what was ever done before. It's a testament to the inspiration this rickety little old game provides to people.
Rock on.
Posted by
mhquake
at
12:22 PM
0
comments
Sunday, December 20, 2009
Cleaning out the Alias Model format
Just done some work on DirectQ 1.8 today (when I should have been working on QRack).
It's no secret that the Q1 alias model format is horrible. It's horrible to load, horrible to work with, horrible to render from. Even it's own mother doesn't love it.
Having to work with it was the main reason why I delayed doing shadows in DirectQ for so long.
The on-disk format is obviously beyond redemption but there are things we can do with the in-memory format. I've been meaning to do this for a long time, but finally knuckled down to it on a lazy Sunday afternoon.
No more commands, no more order, no more negative vertex counts, no more not knowing anything about what you're going to draw in advance. An in-memory alias model is now a linked list of polygons, each of which contains a primitive type, numverts, an array of verts and a next pointer.
Each vert contains s, t and an index into the alias verts array. The alias verts array does not need to be pre-expanded to strips or fans, but is exactly as it comes off disk.
Side effects of this include a substantial memory saving on alias models, removal of a lot of the temporary buffers that were used to shift data around, faster loading, and an extremely clean render function (it looks quite similar to a brush poly render function now). Even the loader and structs are cleaner.
Functionally it still supports everything that the old in-memory format did, but it's far more pleasant to work with and maintain. Result I think.
Posted by
mhquake
at
8:13 PM
2
comments
Saturday, December 19, 2009
Happy Birthday DirectQ!
I completely missed this, but DirectQ has recently had it's first birthday. On December 14th 2008 the first release of what became DirectQ (it was called something else then) was let out into the open. It's quite amazing to think how far it's come in that year, and equally amazing to think that this all happened after a few years during which I had released nothing.
Unfortunately time was against me and 1.7.666c didn't make it out on the same date, 2009 (it was 2 days later).
There's a major Quake anniversary also coming up in a few days time; I hope you all know what it is.
Posted by
mhquake
at
3:30 PM
4
comments
DirectFitz is Released!
Go get it.
This is a milestone release as it's the engine which I developed over 90% of the wrapper on, so I've got a nice warm glow inside from getting it out.
In the end I decided not to bother with either scissor test or r_oldwater 0 mode. I did however spot some major optimization potential in liquid surface subdivision (which should be able to be carried forward to GL FitzQuake) so that's something new for you. Basically what this means is that water surfaces will tesselate much less for a given value of gl_subdivide_size, but retain much the same quality.
The next release will be DirectRook (QRack). This is already mostly done, but there is quite a bit of work in bringing on the remaining items (I may even need to drop some QRack features, but I'd prefer not to - we'll see).
Following that I'm going to consider my work largely complete. I've had a look at the DarkPlaces code, and while a DirectHavoc is certainly possible, creating one would require considerable disruption of it's OpenGL setup code. To my mind that really defeats the purpose of creating this wrapper in the first place. I might do it sometime just as an exercise (everyone really should read more of LordHavoc's code) but I make no guarantees.
I'll probably also release updated versions of the wrapper from time to time, but nowhere near as regular as now.
Posted by
mhquake
at
2:59 PM
1 comments
Friday, December 18, 2009
DirectFitz is almost done
This is going to happen sometime over the next 2 days. All that's needed now is for me to go through the code and pick up on anything I've accidentally missed out. How many of those "accidentally"s there are will determine how long it takes.
I've had tremendous fun implementing GL_QUAD_STRIP in Direct3D (FitzQuake uses this for r_showbboxes). The OpenGL documentation could certainly be a lot better here - I was working off the basis that the documented vertex indexes were 0-based, and couldn't figure out what was going on - turns out they were 1-based all along.
Hint to OpenGL documenters - use the same array base as programming languages do, please. You're writing for programmers, and programmers don't actually care if you're technically correct according to some criteria that's irrelevant to them.
Sheesh, I know I harp on a bit about the quality of D3D documentation, but here's evidence that OpenGL ain't perfect either.
For the most part I like to try get a port as complete as possible, but this will be one where I might fall short. Two main things will likely be missing; the first will be on account of the fact that D3D8 doesn't support scissor test (although I might try to hack something up), the second will be on account of the fact that I didn't feel like bothering with implementing FitzQuake's render-to-framebuffer water (although I will need to implement the API calls used for other engines, so I might knuckle down and do it yet).
I also haven't done the stencil buffer; FitzQuake 0.85 requests a stencil buffer, clears it if it has one, but doesn't actually do anything else with it.
Polygon offset was also fun. D3D8's implementation was very basic (it would have been much better if I had used D3D9, but I wanted to retain compatibility with as much horrible old hardware as possible), just being a single Z bias factor which is only capable of biasing towards the viewer (sometimes we need to bias away from the viewer). In the end I hacked a little and set the default to the midpoint of D3D's range, thereby allowing me to bias towards or away at the expense of some range (which in practice wasn't really used).
More next time (which will hopefully be the release).
Posted by
mhquake
at
12:22 AM
0
comments
Thursday, December 17, 2009
DirectTonik is released!
Baker started this one, but hit some issues with texturing and then ran out of time, so I finished it off for him.
It's a bit rougher around the edges than some of the other ports, but all the same, it's a QuakeWorld engine which is always a nice thing to have more of.
Get it here!
Posted by
mhquake
at
10:08 PM
0
comments
DirectFitz Update
Following the release of 1.7.666c I've been doing some work on DirectFitz; just finishing off the OpenGL support. I've decided to not do a complete implementation of everything in FitzQuake (that means render-to-framebuffer waterwarp, mostly) as it would delay a release and is not really necessary for regular gameplay.
I do however want to implement glPolygonOffset, as other engines use it for other features. That's about all that's left to do (there's also GL_QUAD_STRIP), and it's just really a matter of mapping the OpenGL calls to the D3D ones. D3D8 has a somewhat simpler implementation of polygon offset than OpenGL (it's better in 9) and there doesn't seem at present to be a one-to-one direct mapping, so it's over to research, experimentation, and finding out which parts of the documentation are telling the truth and which parts are lying (and which parts are missing...!)
Hoping for a release in not too much time (days rather than weeks).
DirectRook will take a bit more work, there are some 9 calls that need to be implemented. I think I might skip display lists; they're not really that big a deal and we can just call the function that generates the list directly instead. In the vast majority of cases these days display lists give zero performance improvement; they were really only useful back in the early/mid 90s.
There are however a few interesting things there like 1D textures and glCopyTexImage2D, which will need a bit of work. Hopefully I'll still make the target of a release before 2010, but I'm not going to punish myself too much to get there; if it happens it's great, if it doesn't, well it'll just have to happen in early 2010.
Posted by
mhquake
at
1:08 AM
3
comments
Wednesday, December 16, 2009
DirectQ 1.7.666c is out
Get it from the link on the right.
If you downloaded it before this notice arrived, please download it again as I had accidentally packaged a wrong version. Ooops.
Full list of changes:
- Increased quality of sound scale table (less abrupt transitions between volume levels).
- Increased sound paintbuffer size to help prevent sound stuttering, breakup or drop-out.
- Added ProQuake NAT fix (use cl_natfix to toggle, default 1 (on)).
- Fixed bug where BeginScene would not get called in time if only liquid surfs were visible in the world.
- Rationalized sound buffer locking and usage a little better.
- Fixed bug where a player head model skin would get replaced with a full player model skin.
- Begun generalising the effect system for both HLSL and non-HLSL.
- Added fallback for cases where L8, A8L8 and X8R8G8B8 texture formats are not supported.
- Added rudimentary HalfLife BSP Support (not too robust, yes I'm aware of the bugs).
- Added exception handling around main loop to try help with restoring gamma correctly after a crash.
- Added HLSL classic sky warp - no depth buffer trickery, no perspective trickery, just looks right.
The HalfLife BSP support is quite rudimentary at the moment; don't expect to be running your favourite HalfLife maps using it! It's good enough if you want to just load a HalfLife map and look around it, but nothing more. There are bugs with hulls and clipping, and it may occasionally crash to the console. It gives you a LOT of warnings about missing spawn functions when you load.
If someone releases a good Quake SP map pack (and it needs to be SP for offline testing and monster-sized hulls) using HalfLife BSP format I'll probably firm it up, but right now there's no real requirement for it (aside from to add to the list of engines that support this format, and thereby provide extra motivation to release such a pack).
Before starting on 1.8 I'm going to finish and release my D3D8 ports of FitzQuake and QRack, which will give me time to ponder more on the work that needs to be done for that version, as well as respond to any bugfixes 1.7.666c might need.
Posted by
mhquake
at
7:47 PM
4
comments
Tuesday, December 15, 2009
DirectTomaz is Released
It's been a short while since I did a D3D8 port, so in a fit of nostalgia I decided to go for DirectTomaz.
This is the legendary TomazQuake 1.48 engine, which pioneered so many advances back in the day. You will need to be aware however that it was very much a product of it's time, and that time was before we understood a lot of things as well as we do today. I haven't checked but it's quite possible that there may be some wacky behaviour with it that just doesn't happen on a more modern source port.
You will also need the original TomazQuake data pack to play this. All available from the link above.
Posted by
mhquake
at
10:32 PM
2
comments
Fun with Sky
Just been spending some time doing a pixel shader implementation of the classic Q1 skywarp. This has been really really nice to do, and I was able to get it up and running in next to no time at all, and have an immediate worthwhile result in that it's incredibly fast, rock-solid, doesn't heave or buckle, doesn't require subdivision or any depth buffer trickery.
I'd post a screenshot only it looks identical to the current DirectQ sky (at least in screenshots).
Posted by
mhquake
at
8:17 PM
0
comments
Monday, December 14, 2009
Plan for next releases
The current plan for the next 2 releases seems at present to be as follows:
There will be a relatively fast-happening (expect it possibly over the next few days) release of 1.7.666c, which will be a minor update. Features for this one will include some small bugfixes, some cleaning up I've been doing on the sound code, the ProQuake NAT fix, and possibly a few other items. It's not going to be a huge advance on 1.7.666b, just an incremental upgrade. I'll remove the original 1.7.666 roundabout the time I do that release.
That's going to be the last of the 1.7 line. The following release will be 1.8, and will be a big one. I've been talking about doing a restructuring of the world render, and the design of this has been coming together in my head over the past coupla months to the extent that I'm now fairly happy that I'm going to have something solid and flexible. I'm going to defer the second HLSL based render path until that time, as I will have to rework it again anyway; the less I need to rework when I restructure the main render the cleaner, faster and easier I'll be able to get it done.
I had moved alias models to a toggleable HLSL path for 1.7.666c, but I've disabled that code for release. It's still there, and you'll be able to see it, but it won't do anything unless you recompile yourself.
In between those two releases I'm going to put in some time to bring my D3D8 wrapper up to the full QRack feature level and release DirectR00k. That will happen before the new year. 1.8 won't be happening until sometime in 2010.
I'm fairly excited about the way things are panning out for 1.8, by the way, and am getting eager to start bringing on some code. This has been overdue for a while; the 1.7.666 line of releases wasn't even supposed to happen. But it's all worked out for the best and it's going to be a pretty damn cool codebase, even if I say so myself.
I'll post some more on the changes that are going to happen as things progress.
Posted by
mhquake
at
12:11 AM
1 comments
Saturday, December 12, 2009
Fun with HLSL
Following the recent success with waterwarps, I've started becoming interested in going back down the HLSL route for the next release. This time however I intend providing a fixed path as well, and you will be able to toggle between the two paths in realtime. It's going to be nice to get that done because HLSL does genuinely enable a much leaner and more optimal setup, with a huge number of state changes simply removed. I want to put DirectQ in a position where it can take advantage of this, but also be able to run on older hardware that doesn't support HLSL (especially hardware that doesn't support Shader Model 2).
Other differences with the old 1.6 and earlier versions include that I'll be intending to replicate the non-HLSL look exactly, and won't be providing a means for supplying your own shaders. I thought about the latter option, but I'm convinced it's the right way now. A Quake engine isn't a general purpose scene graph, it's intended to do precise effects in a known and predictable way. I don't think it's too contentious as I suspect it's something that only a very small number of people would actually want, and those who want it would likely also have the means and ability to recompile the engine themselves anyway. The fixed path of version 1.7.x certainly didn't hurt the number of downloads that the engine got, and it's hard to argue with that.
Posted by
mhquake
at
8:23 PM
0
comments
Wednesday, December 9, 2009
1.7.666b is out
Get it here.
I expect that there will likely a 1.7.666c as things shake out of this one, but overall I'm quite a bit (lot!) happier with the code than I was for the previous release.
Posted by
mhquake
at
8:22 AM
7
comments
Tuesday, December 8, 2009
Fun with Player Skins
I've just been making the player skin translation system on DirectQ actually work properly. This is a problem on account of the way Quake handles player skins. We basically have a situation where a player skin is translated in one of 3 circumstances: (1) when a player enters the game, (2) when a player's model is changed, and (3) when a "color" command is issued at the console (or via the Setup menu).
What this means is that any busy deathmatch will have lots and lots of runtime translations as people are killed and respawn (which counts as entering the game, oddly enough). Run the bigass1 demo (look for it on Google, but be careful what links you click on...) to see the effect. Lots and lots of hitching and stalling as skins are repeatedly updated.
A 16 player deathmatch in a confined space will always result in lots of killing, so this is something of an extreme example - or is it? Even with 4 players things will get fairly busy in some of those maps, so it seems a sensible thing to do.
I had previously implementing a caching system where skins were retained based on the player number, but that didn't quite work out as I had failed to take account of the fact that the player number might change. So I've now changed it to retain skins based on the colour, which means a cache of up to 256 skins.
The plus side is that it's now utterly reliable, the negative side however is that it can cause some fairly heavy video RAM usage. To alleviate that I'm clearing down the cache at each map load, and also clearing individual entries if a skin colour changes (via the "color" command or the Setup menu) and no other client is using that skin.
Overall it's a nice result and a good compromise.
Posted by
mhquake
at
9:38 PM
4
comments
More Direct3D 8 - DirectBengt has arrived
I've just released "DirectBengt" (same link as before: http://quakeone.com/mh/, which implements my wrapper on Bengt Jardrup's "Enhanced GLQuake" engine. This was a bit more involved as I decided to retain the Win and GL Project configurations and also update the assembly language to work properly on Visual C++ 2008 Express.
There was also quite a bit more work involved owing to the fact that Bengt didn't update ID's original mishandling of OpenGL. Being ultra-trad is one thing, but when it means bringing forward mistakes and malpractices it gets a little silly.
In many ways this is now a "reference implementation" as it demonstrates how to co-exist peacefully with existing projects, as well as how to correctly update the assembly language files (and also provides a set of .asm files that can be used by other folks).
If you're registered on QuakeOne.com there is a thread for feedback on these engines (clicky) which I would appreciate you post any or all feedback - especially notifications of problems - as I'm not the only one who needs to know.
Posted by
mhquake
at
6:58 PM
0
comments
Direct3D 8 Quake Engines
This is something I've been working on for a while - it even held up the release of 1.7.666 of the main DirectQ line. To understand we need to go back in history a little. Once upon a time there was a Direct3D 7 wrapper for OpenGL, which used D3D7 to emulate OpenGL calls used by Quake. It was pretty basic, pretty buggy, pretty slow, and no longer maintained. However, I did notice that ProQuake was using it, and I thought: "I can do better!"
So I went and wrote one of my own. It's been growing over the past few months and is now at a stage where it can do everything required by standard GLQuake, JoeQuake, and possibly a few other engines.
Time to go public!
The first release of an engine using it has now been accomplished, and it can be obtained here (thanks to the QuakeOne folks for the webspace, although I've certainly remembered how much I hate HTML).
This is JoeQuake, and I've decided to retitle it "DirectJoe" for the purposes of this implementation. Bear in mind that this is a very early implementation, the first public release, and as such it hasn't been widely tested yet. I've tested it, and so has Baker from QuakeOne (who deserves MASSIVE credit for his part in making this happen), and now so can you.
Some things might work, some might not, but the point of the whole thing is to have fun, so enjoy it.
Further engines will follow - I'm in the process of doing FitzQuake (almost done) and QRack, and would also like to do Enhanced GLQuake and maybe even DarkPlaces (if it plays nice). Baker's doing ProQuake himself. I don't want to release GLQuake (which is done and working) as it's too much work to get it to a state where it's usable.
This doesn't mean that DirectQ is dead - far from it! I've already used code and things I've learned from this in the forthcoming 1.7.666b release. This is more of an entertaining side-project for me, and it enables me to focus on totally 100% behind-the-scenes work while leaving the engine enhancement to others.
Posted by
mhquake
at
12:04 AM
1 comments
Sunday, December 6, 2009
More fun with Waterwarps
OK, decision time.
I've dug out some of my old HLSL code and have re-implemented waterwarps in a pixel shader. It's been a little fine-tuned in that it is now properly capable of falling back to the old (non-HLSL) way if your card doesn't support the required version of pixel shaders. I've also cleaned up the old code a little (lot!) and rewrote the shader so that it gives a result that's virtually identical (higher quality in some places) to the old way.
I guess that this is now the best compromise; pixel shaders can be used for a better result (with much higher performance) if you support them, but you don't actually need them.
Posted by
mhquake
at
10:40 PM
1 comments
Latest Fixes and Changes
Here's what we have so far for 1.7.666b:
- Set minimum far clip to 4096 so that it plays nice with maps/mods that expect it.
- Fixed menu shortcut keys/commands not allowing you to exit from menus properly.
- Fixed vid_restart and state restore at fullscreen console.
- Cleaned up game changing a little.
- Restored mouse pointer is visible in menu and console screens with windowed mode.
- Forced r_wireframe to always off in multiplayer (potential wallhack).
Waterwarps are a slightly contentious issue, and I want to take the opportunity to explain why they are the way they are (a fixed subdivide size of 24 with no option to change it).
We all know that default GLQuake waterwarps are horrid. Now, there are a few different possible solutions to this, and I have explored the following:
Using a bigger subdivide size
Huge performance improvements, but water surfaces break up horribly again.
Using a pixel shader
Previous releases of DirectQ did this, and it worked fine and performed well, but it did end up locking out anyone who's 3D card didn't support pixel shaders.
Using render to texture
I had this in once, and once again it worked fine, but it could consume up to 80MB of video RAM for a set of 4 textures, and it was really really slow (especially if you didn't have hardware mipmapping available).
Abandon classic Quake warps and do it another way
DarkPlaces does this, any my honest opinion is that it's incredibly ugly. There might be other ways as well, but we're getting into the territory of creating new art, which is something that can't be manufactured on-demand.
Restore a variable subdivide size
I thought about this, I really did, but it falls into the territory of offloading responsibility to the user. There's also the question of what the default should be. I know that whichever I pick someone somewhere will report it as a bug, and continue reporting it as a bug even after 6 months when I've already explained what's happening and put up posts everywhere explaining it (that's already happened with other things, by the way). I prefer a more general purpose solution if possible.
The position is that we have to accept a tradeoff of some kind, and the factors involved include more than just performance; there's also compatibility with a wide range of systems, resource usage and visual appearance to take into account. Where things currently stand is that DirectQ's chosen tradeoff is - to my mind - the lesser of many evils, so that's the way it does it.
It's at the back of my mind that I do need to get back to it and make it better, but it's something that will need to wait until the correct idea comes to me; it can't be done on-demand.
This is actually a good example of the kind of behind-the-scenes thought process that goes into a lot of DirectQ features. If something is changed from the way GLQuake does it, there is nearly always a reason, and it's not lightly done: alternative options are always considered.
Posted by
mhquake
at
4:59 PM
5
comments
Thursday, December 3, 2009
First 1.7.666b bugfixes are in
Quick and easy, but so far the sky and the menu/shortcut key things are fixed.
In the end I decided to impose a minimum of 4096 for the far clipping plane. It's what GLQuake uses, so it's what maps and mods will expect. If geometry in the current PVS is greater than 4096 units away the far clipping plane will be extended as required.
One "new" feature I want to add is showing the mouse cursor in windowed mode if the menu or console are active. GLQuake did this, but it's been missing from DirectQ for some time.
Posted by
mhquake
at
9:50 PM
5
comments
Wednesday, December 2, 2009
First 1.7.666 bugs are in
The big one seems to be a problem with rendering sky. In some maps we have a situation where the sky (or parts of the sky) ain't being drawn.
What's wrong with this seems to be cases of the first thing every mapper should learn when sealing a leaky map - do NOT put a great big box around the map.
Here we actually do seem to have a great big box around these maps, and it's drawn at a distance that makes the dynamically set far clipping plane go wrong.
There are other possible causes, but I'm going to download some maps and mods tomorrow and start testing this theory for starters.
There are a few solutions here, but the one I think I'm going to go for is just extending the minimum size of the far plane to the standard max extent of a Quake map, which is about 14188.
A second bug is flickering instanced brush models (ammo boxes, etc). I must admit that I haven't observed this myself in any of my testing with Intel or NVIDIA hardware (I don't have any ATI), but what I am curious to know is if it is a new thing or if it also happened in 1.7.3; at least that way I can start identifying what changed to possibly cause it.
Thankfully there seems to be no crashes this time... (fingers crossed, and hoping I haven't tempted fate by saying that!)
What this means is that there will probably be a bugfix release - 1.7.666b most likely.
Posted by
mhquake
at
11:21 PM
0
comments