Release 1.7.666.
How could I have called it anything else?
There's a lot of new stuff in here, not enough to justify a bump to 1.8, but too much for 1.7.4, so something in between seems right. There may be problems for some people with some of this, but right now the fix is "use 1.7.3 instead". I am interested in hearing of any issues, of course.
I've dropped the status back to "Beta" to reflect this, but it should nonetheless be good for general use in most cases.
1.8.0 is not likely to appear this side of 2010, and the plans I have for that (primarily a restructure of some parts of the renderer to work better) are still on the cards.
Oh, and apologies to the people whos server I connected to while testing the Multiplayer menu. That wasn't nice of me...
Monday, November 30, 2009
And it's out!
Posted by
mhquake
at
10:02 PM
10
comments
Updates for Monday 30th November 2009
Once again I haven't been doing anything for the past week or so. Just resting up, really. I hope to do the last testing, finalize what needs to be finalized, and release by the end of the week.
Posted by
mhquake
at
6:06 PM
0
comments
Monday, November 23, 2009
Updates for Monday 23rd November 2009
I'm on the brink of releasing this, but I'm not going to get anything done today. Been typing all day in work and my fingertips are quite sore, so a rest is in order.
It's really just tidying up a few things and making certain that the "Join Game" multiplayer menu does what it's supposed to at this stage though.
Posted by
mhquake
at
6:46 PM
0
comments
Sunday, November 22, 2009
FitzQuake Protocol Network Testing
Network testing of my implementation of the FitzQuake protocol has now - finally - been carried out. This comprised connecting DirectQ clients to FitzQuake servers and vice-versa. Happy to report that all went well.
One or two new bugs got thrown up by this in other areas however; one to do with skin changing (I just hadn't updated the model name check to reflect the fact that I am now retaining ".mdl" on loadmodel->name) and the other relating to multiplayer menu layouts.
Fixing the multiplayer menus is a definite requirement for pre-release. Now, I know that there are people who have been playing Quake for 13 years, and none of this is difficult to them, but there are also people who haven't.
I have to say that Windows 7 networking is horrible. I can appreciate all the "homegroup" stuff for certain types of user, but sometimes all you want is to put in some IP addresses on the same subnet and have it Just Work.
Posted by
mhquake
at
6:36 PM
1 comments
More on "Relieving the bottleneck"
Since I wrote my original "Relieving the bottleneck" post, I have come across another post here ("Why is Present so slow?") which discusses similar issues to those I addressed.
The basic problem occurs with, and only applies to, a combination of a fast CPU but a slow GPU. The CPU is sending data and commands faster than the GPU can handle, and from time to time the GPU needs to stall in order to catch up.
The theory is to deliberately slow down the rate at which the CPU sends data and commands so as to remove, or at least reduce, the need to stall.
As I said in my previous post, this is a feature that is off by default, but that can be enabled if you are experiencing symptoms. It's not a magical cure all as stalls could be coming from many other potential sources, but it might help.
One (related) thing that needs attention drawn to it is that there is an untruth in the Wikipedia Comparison of OpenGL and Direct3D article. This makes mention of a mode transition (user mode to kernel mode and back) which it intimates (but rather cleverly so, and without saying it directly) occurs with every API call. This is not the case, it actually occurs when the command buffer is full and needs to be emptied.
This article has been noted as being biased in the past. Beware.
Posted by
mhquake
at
12:00 AM
4
comments
Saturday, November 21, 2009
Update on new release
It's been a while since I've updated, so here's a dump on where we're currently at.
I had intended to be releasing roundabout now, but there is still some outstanding testing work. It's relatively minor stuff, and it's really just a matter of getting round to it.
(Going into rant mode now...)
One feature that almost made it in but that I ended up backing away from is the use of D3D9Ex for the renderer. This is an extended version of Direct3D 9 that is only available on a Vista or 7 machine, and even then only if you have hardware that's capable of running the Aero desktop. The intention was to support both regular D3D9 and D3D9Ex, running a test for availability at startup and deciding which to use then.
The neat thing is that the interfaces are almost 100% compatible, such that DirectQ could be ported to 9Ex in about 5 minutes. There are also a lot of really NICE benefits, such as elimination of device loss, extra functionality, sharing of resources across devices, and so on. Unfortunately getting both to co-exist peacefully in a single code-base is a little more involved, and since D3D9Ex is not, and apparently never will be, available on earlier OSs (or on Vista/7 machines that don't have a WDDM driver) the people who end up suffering are the developers and end-users.
This would not be a sustainable situation for me. I really REALLY wanted to use 9Ex, but the peaceful co-existence issues have for now defeated that impulse. The irritating thing is that all of the extra 9Ex functionality seems as though it could easily have been implemented in a software layer in the runtime. The new Direct3DCreate9Ex and CreateDevice9Ex APIs would have made it a non-breaking change. Microsoft did The Right Thing by implementing a software layer for vertex shaders on hardware that doesn't support them, for example, and I know that their programmers aren't stupid, and could have done it with 9Ex. Evidence: all of the 9Ex features (at least that I've read about so far) are things that otherwise need to be programmed around in the application.
This comes back to a point I made a while back where I suspected that Direct3D 10 was being (mis)used as a Vista marketing tool. And the 9Ex situation seems to support that point.
Oh well, I suppose it could have been worse. At least it's not Unix.
Posted by
mhquake
at
10:55 PM
0
comments
Tuesday, November 17, 2009
FitzQuake Protocol Is Even More Done!
Just added in the necessary server-side support for > 8192 entities. This was semi-supported previously in that a DirectQ server would just not send the messages, whereas a DirectQ client would just blindly gobble them up.
The major reason why I left this until last was because I needed to do some work on memory allocation in other functions in order to better support the bumped number. Some of the physics code was allocating large arrays of edict_t * and vec3_t of length MAX_EDICTS on the stack, with would have totalled 1MB, so I had to move that to a proper one-time-only dynamic allocation. There was also a static allocation in the world code which I wanted in one of my memory pools in order to give a better reflection of memory used with heap_report. I also needed to tweak the initial sizes of some memory buffers (to cut down on run-time allocations) and bump the MAX_VISEDICTS value client-side.
Getting the protocol itself working had taken priority; I always knew that I was coming back to this stuff but I needed to be certain that everything else would work fine (and coexist peacefully with the other 5 protocols) before doing so.
The remaining item that's not fully supported is the U_LERPFINISH bit. Again, servers don't even send it and clients just blindly gobble it, so it's supported sufficiently to not cause a crash. There's a lot of auxillary code about this that's specific to the FitzQuake interpolation system which would need careful studying and translating into DirectQ-language (I don't just blindly copy 'n' paste other folks code!) before I could tackle it properly. Most likely is there's a one-to-one mapping of almost everything, but I need to be certain.
I think right now that sufficient support to run without crashing is preferable to risking ripping apart some of my alias model drawing architecture (even though I'll be first to admit that there are parts of it that need ripping apart, but let's do it in a careful, controlled and deliberate manner).
Did I say "testing" a while back? I've done the ID1 run with protocol 666 up as far as e4m3 now, and no problems at all have occurred. I've also done some brief WarpSpasm testing. RemakeQuake is still required for .alpha tests, and once I know all the SP stuff is solid I'm going to do a network test (mental note: also need to test SDQuake).
All going well I'm now inclining towards releasing with protocol 666 as the default. There are enough kludges in the BJP protocols (which he admits freely in his comments) to make replacing it a desirable goal, and 666 just fixes these in a fairly elegant manner.
Posted by
mhquake
at
7:46 PM
0
comments
Getting rid of evil
I know I said that I was done with updates and going into test/debug mode, but another essential one just came through.
Quake uses the Windows QueryPerformanceCounter and QueryPerformanceFrequency functions to control it's in-game timers. It turns out that these functions are evil on x64 and/or multi-core machines. Plenty of nightmare stories about timer drift, different cores running at different frequencies, and so on - just search on Google for more info.
This explains some of the issues I had experienced on a multi-core x64 machine, so I've switched over to the timeGetTime API family. This is the Windows built-in multimedia timer, and it can be configured to have a 1 millisecond resolution - perfect! (Side note: ID did the same for QuakeWorld and QuakeII, but for different reasons.)
Unfortunately I won't be able to confirm if this is a definitive resolution to the specific issues I had for at least a week or so.
I've also switched from using doubles for timing to using floats. This has finally enabled me to remove the D3DCREATE_FPU_PRESERVE flag from my video startup, and I now have an integer timer that's converted to a float every frame, so I'm effectively immune to all forms of timer drift caused by D3D switching to single precision mode unless you specify this flag (this was a serious problem in the old days while bringing up 1.0 for the first time).
The only issue with this timer is, because it's a 32-bit DWORD value, it's subject to the old wraparound bug after 49.7 days of continuous running. I guess this won't be happening too often though, and anyway the wraparound will only affect DirectQ if DirectQ happens to be running at precisely the moment in time it occurs, so I'm not going to worry too much about it.
All in all a very productive hour or so.
Posted by
mhquake
at
5:49 PM
2
comments
Relieving the bottleneck on slow Gfx cards
This one's going to be a little bit difficult to explain, so bear with me. If it seems bad at the start, remember that it's not and try to get through to the end.
By way of introduction, all versions of Quake by default limit your FPS during play to 72. This is overridden in timedemos, and can also be overridden by some engines that provide a cvar for it (DirectQ's is called "host_maxfps" and defaults to 72).
Now, there are a lot of things going on in a frame. Key events and other input, command processing, network transmissions, screen updates, sound, and so on. Of these, screen updates is actually the major bottleneck on a lot of hardware. Even a 4 year old machine is perfectly capable of running everything else at a coupla thousand frames per second (try setting r_norefresh 1 then doing a timedemo and you'll see what I mean).
To relieve some of this bottleneck I've added an optional (off by default) cap equal to your refresh rate multiplied by a user-configurable factor on how many times per second the screen update portion (and only the screen update portion) of each frame can run.
The next bit is very important.
This is not the bad thing it might sound like to some.
The maximum number of times per second that your screen can be redrawn is equal to your refresh rate anyway, so one effect of this cap is to prevent damage to your monitor.
A second effect is that it relieves pressure on your graphics card. If you're running a 60 Hz refresh rate and you don't have enough fillrate to do the Quake default 72 FPS, you'll get better performance all round (yes, there are still cards like that, even in modern machines).
A third effect is that it allows other frame events to run at something more like their natural speed, meaning that even if you're getting poor graphics performance you're less likely to suffer from framerate dependent physics and less likely to choke input and sound. Even those with low framerates can get a more enjoyable game.
And finally remember that it's off by default, so unless you actually enable it you will notice nothing different at all.
This is not the same thing as vsync; vsync (which will also be available in the next release) blocks everything until the next refresh interval, this just skips drawing but lets everything else run normally and unaffected.
A side effect is that timedemos are stupidly fast, and are in fact largely meaningless.
If you're not happy with the idea of this, you can always forget that it ever existed; it's off by default, remember. If on the other hand you're getting uneven input and/or choppy sound, or if your CPU meter in Task Manager hovers very low despite DirectQ running flat out, you're heavily GPU-bound and this might be the solution you need. You can even experiment with different values; set it to 0.5 to refresh at 30 FPS (assuming a 60 Hz refresh rate, and remembering that everything else runs at normal speed), for example.
It's not mandatory that you use it (chant: "off by default, off by default, off by default"), but for the primary target audience of this engine (those with slow Intel cards) it does provide benefit.
Posted by
mhquake
at
5:04 AM
6
comments
Monday, November 16, 2009
FitzQuake Protocol Is Done
...ish. Current position is that I can successfully play a demo recorded in FitzQuake using sv_protocol 666, FitzQuake can successfully play a demo recorded in DirectQ using sv_protocol 666, and DirectQ can connect to itself using protocol 666. I have yet to test a network connection from each engine to the other.
There are some parts of the FitzQuake protocol I've chosen not to fully implement yet. What this means is that my server won't set the flags or send the messages, and my client just tests for the flags and blindly gobbles up the messages. These are parts that rely on code changes elsewhere, such as alpha support and support for > 8192 entities.
If I want to make protocol 666 my new default I'm going to need to implement alpha fully on the server at least; I don't think support for > 8192 ents is that important (not even Warpspasm needs it).
In other news I've cleaned my monitor and it's been eye-opening in more ways that one. There was 4 years (!!!) of accumulated gunge on it which I had just gotten used to and hence never really noticed. The brightness from it now is pretty extreme, and actually HURTS to look at. I've had to turn down the brightness a little as a result. Ouuufff.
Posted by
mhquake
at
7:58 PM
2
comments
More fun with Protocols
No updates for a short while as I've been busy with Real Life. It's getting close to the Christmas Party season too, so time to work on DirectQ will be slightly restricted.
I'm currently in the throes of implementing the FitzQuake protocol 666, and it's kinda fun. Client side it's working fine, and I can play demos recorded in FitzQuake without any bother; server side there is a whole barrel of nastiness to sort out, mostly arising from getting interoperability with the BJP protocols and assorted vile, evil, ugly, disgusting and nauseating Nehahra hacks to work cleanly.
I'm thinking of making the FitzQuake protocol the new default at some stage. I'll probably release with the old default still in place as I think my implementation might need more testing. You'll still be able to change it at the console via the sv_protocol command (complete with autocomplete!) or in the Multiplayer menu, and whatever protocol a demo was recorded with will be autodetected and Just Work.
Overall this was an unplanned extension to the code, and as such will mean a small delay on the release, but hopefully not too much. I do think it's the Right Thing to do though, as the FitzQuake protocol is overall a LOT cleaner than the BJP protocols ever were.
Posted by
mhquake
at
1:22 AM
5
comments
Wednesday, November 11, 2009
Lightmaps go OOP!
If you've read the code for previous releases, you may have noticed that for some time now I've been intending to switch my lightmap struct to a class. This has now been done.
I'm still not a huge fan of Object-Oriented Programming - it feels very alien to me. There are however some very clear cases where it's a definite win, and this was one of them. Never mind structuring of design, etc, the big advantage to me is that you can set up an object to manage it's own internal state with clearly defined boundaries, so that some of the more - shall we say - delicate items (such as textures) become effectively immune from any meddling coming from elsewhere.
The code remains broadly the same as before (throwing away and rewriting working, debugged code is never a good idea), just that it's now wrapped in OOP shenanigans. It's another opportunity to use one of my favourite constructs too - the self-destroying linked list.
Some other changes include the removal of gl_flashblend (again!), restoration of the original 256 normal brightness (I had it at 264 to be consistent with ('m' - 'a') * 22), a resizing of initial memory allocations to be able to fit the peak usages for a full run through all of ID1 in (without requiring extension of memory used; about 32MB total) and the addition of cmdlist, cvarlist and cmdcvarlist. These last can dump to the console or to a text file (if you supply a file name as a param).
A new bug crept into this version which was quite nasty. Alias model caching was basically broken, and as a result all alias models were reloaded from scratch each map change, and re-cached every time. I had noticed quite a slowdown in map loading recently, and this was it. Run the automap for more than a few seconds and you could use up all your memory - ouch! Nice to get that one before release, woulda been embarrassing. I think I'm going to cache BSP bmodels (ammo boxes, etc) too.
More next time.
Posted by
mhquake
at
6:26 PM
1 comments
Tuesday, November 10, 2009
W_LoadWadFile: couldn't load gfx.wad - the final word
This one sucked a lot. Anyway, on release here's the way it will work:
- It will check the directory that DirectQ.exe is in for the presence of certain files needed to run Quake (pak0.pak, etc).
- If it finds Quake there, job done; ignore the rest of the list. Goto 200, run DirectQ.
- If it doesn't find it, it checks in certain "well known" folders such as C:\Quake, etc.
- If it finds Quake in one of these, job done; ignore the rest of the list. Goto 200, run DirectQ.
- If it doesn't find it, a meaningful error message is popped up.
- There is no HD scan or anything like that, the error message is terminal.
I'd like to encourage any nay-sayers to at least download and run the engine before succumbing to fits of paranoia, the proof will be in the exe (and the source code).
Posted by
mhquake
at
8:54 PM
5
comments
Fun with vid_restart
I mentioned a while back that I had a bug with the vid_restart command when issued from a disconnected (fullscreen) console. I've been doing a small bit of work on trying to track down and fix it, but have not had much progress.
Because I don't want to hold things up too much while trying to track down a stubborn bug that only happens in certain circumstances, I've decided to prevent the situation that causes it from occurring instead. So for the next release, if you try to issue a vid_restart command from a disconnected console you'll receive a message telling you that it won't work and that you should do it while running a map instead.
I know it's evil, but the alternative is to either spend time (and I don't know how much time) trying to trace it, or release with a bug that could trap people.
It is something that I intend fixing at some time, and the video code is continually getting a working over, so there's a chance that a fix could fall out of something else.
Posted by
mhquake
at
8:27 PM
1 comments
Fun with Software Quake
Been doing some comparisons of DirectQ with software Quake to see how close the look is. Here's some screenshots.
You can't see it too well in these, but overall it's very similar now. There's the obvious brurring from linear filtering of course, but still a few other differences.
Alias models are brighter in software Quake, but seem to integrate into the world better in DirectQ. I worked hard enough to get that so I'm keeping it.
The big surprise is that software Quake doesn't actually seem to have 2 x overbrighting. An examination of the slot in the floor of the Easy hall indicates that it seems more in the region of 1.5 or thereabouts.
This is an interesting dilemma. The purist in me is telling me to reduce my lightmap scale to match, but at the same time it's not something I really want to do.
Opinions are welcome.
Update:
Decided not to clamp the lighting ramp. I actually put the code in, and it's not really that big a deal. Also, I have concerns that it would not play nice with my "gl_overbright 0" mode.
Posted by
mhquake
at
1:31 AM
1 comments
Monday, November 9, 2009
Monday Updates
I think I'm going into bugfixing and testing mode on the new release; pretty much everything I want to achieve from it is now done.
What didn't make it is the full PQ messaging system, so that means no scoreboard ping, no iplog, etc. The code is a lot more involved than you would think, and has all kinds of interesting tentacles inserted into all kinds of interesting places. Fragments of it have been added, and I'll probably work over it again post-release.
The surface refresh wasn't touched too much. I know it's a slightly horrible mess right now, but it would require some major work to clean up. This release was originally intended to be a fairly minor incremental update, so I never really got the chance to schedule time for it.
I did do some optimization in some places, so speed should be up on most machines. Some of this is potentially at the expense of some entities showing through when crossing water boundaries (just like GLQuake - sigh) and I'm interested in hearing if anyone notices. It should only be an issue on maps that haven't been re-vissed for translucent water. In the end the code I had was suboptimal and wasn't working right in places so I did a quick-n-dirty replacement that should work fine in 90% or more cases.
I do consider it a quite major graphical glitch but I'm also not willing to trade the framerate hit I had for it.
Code has been tidied up in some other places, but the big offenders remain. This isn't going to be remedied overnight but I am working towards it.
Hopefully I'll be releasing within a week.
Posted by
mhquake
at
11:32 PM
0
comments
More fun with Lumas
Been doing some more work on Luma textures. Previous releases had a restriction whereby an external texture (from a texture pack) that had a luma available wouldn't have that luma loaded if there wasn't also luma texels in the native texture equivalent. Example: your pack might have a wbrick1_5_luma, but because the native wbrick1_5 does not have luma texels that texture wouldn't have been loaded.
I've now relaxed that restriction, but what it does mean is that all textures will go through the loader twice: once for the native texture, and once to check for a potential external luma (returning NULL if one is not found). This slows down map loading somewhat, so I'm going to need to do some optimization. I kinda like the sound of a gl_noexternal cvar, but it does mean that - owing to the fact that textures are cached - it will require an engine restart. Another option is to do a one-time check if the external texture dir exists and disable external loading if not. Not certain how well this will work with stuff like PAK files and the DarkPlaces non-standard alias model texture paths though.
I'm thinking that the cvar is the way to go for now at least, and maybe keep in mind that a more optimal solution will be required at some later date.
Posted by
mhquake
at
1:06 AM
0
comments
Sunday, November 8, 2009
wbrick1_5 differences
I had mentioned that the new release of DirectQ has some differences in the appearance of the wbrick1_5 texture, and that it was more accurate to the original software Quake look.
For comparison, here's how it looks with GLQuake (-no8bit):
And here's how it looks with DirectQ:
You'll also notice that DirectQ looks brighter; this isn't on account of overbrights (which I had enabled in the GLQuake build I used) or any nasty artificial palette scaling hacks (which have now all been removed) but rather on account of the corrected lighting ramp and a palette derived more accurately from the colormap.
Posted by
mhquake
at
10:15 PM
1 comments
Thinking about D3D8
I posted a while back about my experiences in porting DirectQ 1.7.3 to Direct3D 8. Since then I have done some more work with this version of the API, and have some things to report.
Firstly is the fact that I successfully resolved the speed issue. As I had commented before, this was caused by mipmaps not working, and the reason why they didn't work was a bad selection of filtering modes. It seems as though D3D9 was a little more forgiving than D3D8 in this respect. It's nice to find and fix a bug in the main codebase in such a manner, and also nice to get D3D8 speed up to par.
Secondly, and quite importantly, I have been working part time on a D3D8 wrapper for OpenGL. The intention is to make it available for use with ProQuake, but it's also potentially suitable for more general use. It's not a full wrapper, but has more in common with the old 3DFX mini OpenGL DLLs in that it restricts itself to the subset of features used by GLQuake (but can be compiled directly into the engine exe). No reason why an enterprising programmer couldn't extend it all the way however, although shaders would require a full parser/lexer/converter implementation which is well outside the scope of what I'm trying to achieve with it. Think more OpenGL 1.3 to 1.4 level and you'd be roundabout there (although I haven't done multitexturing or combines yet).
Performance of this wrapper is actually quite good with GLQuake, easily hitting 100 FPS on a machine that gets 110 FPS with DirectQ. This is with a 2 pass full render of everything, but misses out on RGBA lightmaps and compressed textures. I would guess in an exact like-for-like scenario it would be a very close call.
Bottom line is that it has convinced me that D3D8 is viable. The one issue I had considered a sticking point (lack of being able to set a constant colour at TextureStageState) has turned out to be not so important after all as I pass in an additional D3DCOLOR with each vertex which is either modulated with the texture or left alone (this can be optimized by switching the struct depending on the TexEnv but Quake isn't a graphics pig so it's not too much a big deal).
I'll probably do some further porting tests on DirectQ itself, but any final decision on a full port will be held until I release the current version. There are advantages however, specifically an ability to run on even older hardware, and without the requirement for annoying DirectX updates even on Vista and 7. Since DirectQ itself uses a D3D6 or 7 level of technology (but with the D3D9 API) there's not much to lose by doing this.
Writing this has definitely opened my mind to how weird OpenGL is. I mean really really weird. It's still nice to use, but some of the contortions behind the scenes seem like some very strange design decisions were made, and it's roots as a non-textured gouraud-shaded API that could have had a much more graceful evolution really come forward. Stuff like how the glVertex calls really work (and this isn't an artefact of my wrapper, just read the spec) is quite bizarre.
Posted by
mhquake
at
9:32 PM
0
comments
How near to release am I anyway?
The short answer is "I don't know". I could probably release it now, but there is a small list of things I want to either tighten up or finish off, and I haven't really done as much testing as I would like. The current code just doesn't feel ready yet.
One concern is that I have subjected some parts of the engine to quite significant reworkings, and that normally means that there is a possibility that there may be some edge cases in mods that I need to account for.
Overall though I think I'm getting there with it, but it will be a while yet before I start getting the feeling that it's close to done. Best guess at the moment is a week or two still.
Feature Update
The "gl_texturemode" command is back. I've tried to make it operate the same as in GLQuake, but there are subtle differences between how OpenGL and Direct3D handle texture filtering, so there may be some consequent differences in the end result. For the most part it should be quite similar though.
I'm toying with the idea of bringing back sRGB colour space correction. A previous release had it but I had removed it as it didn't play nice with 64 bit lightmaps. As I removed 64 bit lightmaps a while ago it's now an option that's back on the table, and in fact I had added in the code for most of it a short while ago, before removing it again. Right now it's not playing nice with 4 x overbrighting, but as this option is (a) unreleased as of yet, and (b) not present in software Quake, I think I can safely drop it.
Two problems with sRGB. Firstly it requires a 4 x modulate blend in overbright mode, and 2 x in GLQuake mode. Not too difficult, but it does mean more code. Secondly it requires some adjustment of cases where a modulation by colour (rather than by texture) is done. Overall I'm not sure if the gain is worth the pain, but I'll think on it.
"r_overbright" has been renamed "gl_overbright" for consistency with FitzQuake and possibly others. Some other cvar names have been renamed a little. Right now my change log is in slight disarray, with some out of date info in the earlier entries, and some more minor changes not added to it at all.
I really must dump a cvar and cmd list and document what each one does.
Running with white light
For the past few weeks I've been running DirectQ with coloured light disabled, and it has been an interesting experience. What kicked it off was the desire to get the lighting ranges back to proper Quake, and since then I've just stuck with it. It's odd, but I'm actually starting to feel that I prefer it to coloured light. There's something about the stark simplicity and purity of it that's quite attractive, and Quakes lighting effects just seem to look so much nicer without any eye-candy to distract from them.
I recommend giving it a try.
DirectQ design goals
I indicated a while back that I wanted to write a little on what the design goals for DirectQ are. This is something that can probably be gleaned from previous blog posts over the past year or so, but I've never actually summarised them in a single place. I know deep inside my head what I want and where I want it to go, but getting it out in the written word is trickier.
I suppose the primary goal was to have a version of Quake that would run well on my laptop. GLQuake just ran slowly, and occasionally sent the graphics driver into an infinite loop (or at least what it thought was an infinite loop), thus requiring a hard reset of my machine. I had spent most of 2008 working on a new OpenGL engine that was never released, and finally - in frustration at OpenGL - I switched over to D3D. That this was a good and right decision is supported by how far DirectQ has come.
Secondly I wanted any engine I released to have a reason to exist. I didn't see my abandoned OpenGL engine as having such a reason; it did much the same things as QRack and others did, only not as well. By being D3D I automatically fill a gap in the market for engines that can run well on a wide variety of laptops and business-class PCs.
Thirdly I wanted it to be a player-friendly engine. There are plenty of engines that are designed to be useful to mappers, or to give a wide variety of options to modders, but I wanted this engine to be aimed at people who actually played Quake. In many ways this is a much more difficult goal, and I'm still learning.
Fourthly I wanted it to be flexible. This meant going down the huge capacity route, and it meant coding for a certain lowest common denominator of hardware. I'm not sure how close to the ideal it is in either of those, but I hope it gets better all the time.
There are probably other important things I haven't touched on here, and probably places where the ones I have described could have been described better, but for now I guess it will have to do.
Posted by
mhquake
at
1:33 AM
1 comments
Saturday, November 7, 2009
Fun with Lighting
The differences between lighting in Software Quake versus GLQuake are well known, but for anyone who isn't aware of them a good summary may be found here.
One of the things that's not so well understood however is that Software Quake's lighting, as well as being overbrighted, is also non-linear. We're going to need to dive behind the scenes a little for the full explanation, so take a deep breath.
In summary, Software Quake uses a lookup table to determine the final colour of a pixel. This table is called "colormap.lmp", is contained in your pak0.pak file, and consists of 64 rows (each representing a different level of lighting intensity) and 256 columns (each representing a palette index). The result of a lookup is a final palette index which is then used for the final colour.
GLQuake just does a straight modulation of texture colour multiplied by light intensity, with the light intensity being in a range from 0 to 255. However, in software Quake the brightness scale ramps slightly differently, and rather than each of the 64 levels being equal in brightness it's more of a curve, with brightness accelerating slightly up to medium/high levels, then slowing down towards the end. The chart below illustrates the approximate difference.
This is one reason why Software Quake can look a lot more vibrant and alive than GLQuake (there are others). In order to achieve this in DirectQ I've added another step to video startup where it will read the colormap and derive the lighting curve from it. In practice it's not 100% correct as each entry in the Quake palette has it's own individual lighting curve, but it is an average of the curves across the entire palette, excluding fullbright entries.
Another thing that's recently come in is slightly corrected texture colours. I love a bit of wbrick1_5 (the brick texture in most medieval maps) but in GLQuake it just looks very green, whereas if you export and view the original texture you'll find that it's really quite brown. This has now been corrected.
Posted by
mhquake
at
7:51 PM
2
comments
Fun with Texture Filtering
One key item I'm working on is considerably more robust hardware support and fallback modes. It's quite often erroneously believed that D3D is capable of software emulation in cases where a feature isn't supported, but this is actually not so. In fact it's OpenGL that does the software emulation thing (but without providing a means of testing capabilities to see if it's going to do so) whereas D3D will normally fail silently and do nothing.
Texture filtering is on the list, and has now been extended to fallback gracefully from anisotropic to trilinear to bilinear to nearest neighbour to no filtering for all 3 of magnification, minification and filtering between miplevels. I haven't yet re-implemented the old "gl_texturemode" command, but I hope to do so (and in a way that makes it work identically to in GLQuake).
A consequence of texture filtering is that the hardware accelerated Quakes have always appeared blurred, smeared and somewhat washed out when compared to software Quake. I'm not yet certain of a good approach for mitigating this (fixing it definitifely would be impossible) but I'd like to try do something about it. The real problem is that filtering down miplevels causes a lot of it, and that software Quake's colormap was set up to provide a certain look which isn't reproducable in a hardware engine. It's a case of "nice to have" rather than "must have" though.
Posted by
mhquake
at
5:54 PM
0
comments
Thursday, November 5, 2009
W_LoadWadFile: couldn't load gfx.wad (slight return)
Advance warning: this will only make sense to you if you've been following QuakeOne.com
OK, so it looks like I got a little too ambitious and underestimated the extent of the negative reaction my original post and plans revealed elsewhere would get. I had also somewhat underestimated the extent to which people would read the wrong/bad thing into it, and cling on to that impression in the face of all explanation to the contrary. The only reasonable criticism was from a security/trusted app perspective, but that is the one that I'm going to take seriously.
I do want to set the record straight however and say that a full HD scan was never the default behaviour. If you go back to my original post on this topic you'll see that has been the position from the outset. Unfortunately things developed the way they did, and even more unfortunately I got annoyed and posted a rant here, which I have now removed. No need to make a public trainwreck of things.
All that aside, the security/trusted app perspective (something along the lines of "I would be very suspicious of an application that does a scan of my HD if I hadn't asked it to") is a perfectly reasonably and rational one, and it does apply even to the reality of a full HD scan as a last resort. As a consequence I'm going to disable the code for steps 2 (check of well known folders) and 3 (full HD scan) in release builds, so that the version you get won't do this. I'm leaving it in debug builds as it's still useful to me for development and debugging.
It seems a pity to see something that was originally designed with the intention of being helpful to the user going slightly down the plughole, but even the best of intentions don't always work out; road to hell and all that.
Posted by
mhquake
at
1:16 AM
5
comments
Wednesday, November 4, 2009
Fun with Lumas
I've been playing around with the internal format used to store luma maps (for fullbright colours) in DirectQ. A coupla factors are at work here; firstly, DirectQ has been storing them in full XRGB format meaning 32 bits per texel, meaning a whole heap of wasted memory owing to the blend mode used for them ((light + luma) * texture). Secondly, the new r_overbright cvar needs to rescale the lumas on change to prevent them from becoming over-saturated, so in order to do this properly it needs somewhere to stash the original value. I had been storing it in the unused 8 bits, but - and in order to achieve memory savings (not to mention faster texture uploads) - I want to reduce the number of bits per texel. So I chose D3DFMT_A8L8 as a possibly viable alternative (just plain old _L8 is not an option as there is no extra space to store the original value), which is the equivalent of GL_LUMINANCE_ALPHA.
Unfortunately the conversion from raw RGBA data to a luminance format is accomplished by either averaging or adding the 3 channels, rather than doing something sensible (I believe OpenGL does the same so don't bash D3D over it please). Obvious problem here is that some texels have values of stuff like 255/255/128 whereas others have values of 255/0/0. So rather than a single uniform brightness to add to the lightmap where a fullbright colour is rendered, we have a mixture where some colours are dull and others are heavily over-saturated.
It seems simple and obvious now that I've been actually getting in behind the scenes on it, but I've been writing code using this blend for almost 2 years now and had seriously never cottoned on to it.
A complication is the fact that for external textures I use D3DX helper functions where possible, so that I can load a wide variety of image formats without having to write, test and debug code for them. These don't give me access to the raw RGBA data in order to actually do something about it (like take the brightest of the 3).
I think the approach I need to take here is twofold. For the native textures I need to create a second palette at startup time which contains values of 255/255/255 for all fullbright components, and use that for the upload (I upload textures direct as _P8 format so this is easy: just detect if it's a luma and switch the palette pointer). For external textures I'm thinking that I need to initially load them into a surface, then manipulate that, then load that into the final texture. A bit more longwinded than I had hoped, but it seems like the right way.
Update:
Just did the native palette switch trick and it's working beauifully. Lumas now look virtually identical for all values of r_overbright, and also look almost identical to software Quake; no more oversaturated colours - yayyy!!!.
While I was at it, I added an r_lumaintensity cvar, which you can use to control the intensity of lumas in the game (or set it to 0 to switch them off).
What this means is that I've also relaxed my policy on fullbright colours. I suppose it's a victory of that "pragmatism over purity" thing I talk about sometimes; unfortunately the reality is that we have over 13 years worth of damage by texture artists who didn't fully understand the format they were working with to contend with.
Posted by
mhquake
at
10:54 PM
0
comments
There seems to be some confusion...
...over where this post has gone!
Posted by
mhquake
at
5:21 PM
1 comments
Tuesday, November 3, 2009
Barrels of fun
Today it's mostly been menus and a nasty vid_restart bug.
The menu work was totally behind the scenes, and involved a lot of cleanup and standardisation of ways of doing things. The end result will hopefully be simpler, more maintainable, and less potentially buggy code for me. If you've ever looked at my HUD menu code you'll know what I mean.
The vid_restart bug caught me by surprise. I was adding toggleable vsync and testing it out, when I noticed some seriously weird behaviour when doing a vid_restart from a full-screen console. Funny thing is that it works perfectly when in a menu or an in-game console. First you get some bad graphical corruption and it looks as though the engine has hung. Press ESC a few times and the menu will come up, but the Q cursor will be spinning veeeery slow. After a few seconds it picks up speed, and eventually settles back to normal speed.
I suppose it's probably always been there, but I'm damned if I can find the root cause right now. To temporarily resolve it I've put in what must be the cheapest and cheesiest hack I've ever seen; I fire a togglemenu command, then a wait, then togglemenu again to remove the menu. This works but it doesn't hold well over more than 1 consecutive vid_restart, when the whole thing just happens again.
I must go back to some of the older code and see if I can discover what point it came in at. I may also rework vsync changes to only take effect if you have a 3D view up (which is the only place it really matters), but that still won't resolve the more general bug.
Anyway, you heard me say "vsync" (twice now), so yes, vid_vsync is now in (default 0) to enable you to switch vsync on or off as you wish. Previous releases always had it off, and I always wanted to include the option. Yes, changing it does require a vid_restart as vsync settings are specified at device Creation or Reset time. I've set up the engine to do this automatically for you so all should be nice and fine.
I'm not certain about the internal workings of OpenGL or why it's able to do vsync in a much simpler manner via wglSwapInterval (or equivalent), but I suspect that it may encapsulate a lot of behind-the-scenes nastiness that D3D shoves in your face.
Posted by
mhquake
at
12:47 AM
6
comments
Monday, November 2, 2009
Fun with Multithreading and Overbrights
Two main topics today.
I've posted before on the difficulties I've had with the multithreaded C runtime in Visual Studio 2008, and recently I've been working on a multithreaded .NET application (nothing to do with Quake, this is my day job). Multicore/multithreaded environments and applications are no longer just the future, they are the now, and it's something that everyone has to adapt to. The big problem though is that most threading APIs (or at least the ones I've used) are just horrible. I've written multithreaded code before in C/C++ and I was astonished by the ancient terminology in use, by the awkwardness of the API, and by all the bear traps that exist that you need to avoid. It's quite amazing that the .NET System.Threading namespace doesn't really attempt to hide any of this.
There's been a lot of talk about potential things that could definitively swing the desktop market away from Windows and to Linux, but in my opinion the one crucial and critical factor would be the provision of a good API that makes it easy and painless to write safe and performant multithreaded code. Because whichever OS gets that API first will be the one that the besp apps are going to become available for, and with the shortest turnaround time.
On to DirectQ.
From the start, DirectQ has supported and actually enforced overbright lighting. Have a read here for more info on this feature. One consequence however is that some maps - in particular those made by mappers who are slaves to the great god GLQuake - are just not made with overbright lighting in mind. I had been fairly bullish in my attitude towards this: "it's an original and intended software Quake feature, it's the way Quake is meant to look, and tough luck, the mappers should have gotten it right". However, this attitude means that players suffer as maps that are lit without overbrights in mind can look at best weird and at worst bad in an engine that supports overbrights. I've relaxed my opinion on this, and will now be providing the ability to disable overbrights via an r_overbright cvar. The default value will be 1 (overbrights enabled) but you will be able to switch them off any time you like, so if you're playing a map that looks bad (or if you just prefer the look of Quake without them) you can set it to 0, and the change will take effect immediately without needing to reload your map.
One other unintended feature of DirectQ overbright support is that my fullbright colours have always been oversaturated. With hindsight I should have been aware of this from the start, but I only really noticed it while toggling between different values of r_overbright. This is something else that is going to be fixed.
Posted by
mhquake
at
11:10 AM
0
comments
Sunday, November 1, 2009
I'm guessing this is something a few of you won't like
I've just restored the original GLQuake gamma range. What this means is that DirectQ will now apply a gamma of 0.7 to the actual loaded palette entries on startup, which can then be further adjusted by differing values of the "gamma" cvar.
I've been unhappy with DirectQ's handling of lighting and colour for a good while now; while everything did look bright and vibrant it also had a tendency to look kinda washed-out, with dynamic lights in particular looking really poor.
So I was doing some work with the old 1.0 codebase when I observed that the overall look was far more to my taste. The same with some further work on the GLQuake codebase. Some experimenting revealed that the culprit was my use of GDI gamma ramps, which just did not give the same result. Plugging the old gamma back in did it though.
Now, the default value of the "gamma" cvar in DirectQ is and always has been 1. I've personally been adjusting it myself (to 0.7) in order to bring up my brightness, but from the next release onwards there will be no more need to. If you have it adjusted in your config you'll probably need to reset it to 1 and then experiment with values until you find a level you like.
A further consequence is that dark areas in DirectQ will now look a lot moodier, while bright areas likewise look brighter. This isn't some cheap and cheesy HDR effect I've hacked in, this is Quake's intended look as designed by ID. It might take some short time to get used to, especially if you're accustomed to engines (and I include DirectQ in this) that have been doing it wrong.
I'm sorry about that, but I view it as a small price to pay for getting the correct look back.
Fullbrights and overbrights are still supported and are unaffected by this, of course.
Posted by
mhquake
at
11:39 PM
5
comments
Release 1.7.4 Status Update
I haven't been doing any DirectQ 1.7.4 for a while as other things have distracted me, but right now I'm getting back to it. I've almost decided that it won't actually be 1.7.4, as there are too many deep changes to warrant a minor version number bump. What the final version number will actually be remains to be seen.
Right now I'm doing some much-needed cleanup of the video startup. A lot of that code has been in since very early versions, and has never really been overhauled properly. I'm also going to work towards making it more robust.
I've also switched the projection matrix from right-handed (OpenGL and Quake default) to left-handed (Direct3D default). I have a sneaking suspicion that my use of a right-handed matrix was causing some trouble with drivers which assumed that a D3D app would never use one (despite there being support for right-handed matrixes in several parts of the API).
One other thing I've decided is to drop the ProQuake messaging system for now. Implementing it properly would involve some fairly detailed analysis of the ProQuake code and running a debug build against a ProQuake server to test my support. Both of these are things that I can do, but other priorities would take a backseat if I did. LOC support is still there however.
No news on a possible release date as of yet.
Posted by
mhquake
at
10:18 PM
0
comments