Tuesday, April 28, 2009

Sorry, no release today

There's some small outstanding glitches I'm not happy with, mainly to do with slipgate light colours. Each slipgate light brush generates up to 5 colour sources (normally 4 - 3 visible sides and the top), and there are normally 4 of these brushes in a typical configuration, meaning that we get a LOT of heavily saturated red with a fairly sharp transition line at the edge. It only really affects maps where these are present without any compensating colour sources being nearby, but it's ugly enough to warrant fixing.



Now, I could fix this by just hard-coding special cases, but I want to make the program as generally useful as possible, so that's an absolute last resort. The real challenge is to find a solution that doesn't wreck standard isolated colour sources. So far it's been a coupla evenings of bashing at it, but if there is a solution I hope it won't take much longer.

Now the really bad part - I'm off on holidays/vacation for the next week, so there's another gap in work coming up. Maybe it'll be good for solving these problems to get away, hopefully some kinda Zen effect will happen.

I appreciate that it's a bit of a pain having all these delays, but it WILL be worth it in the end. I'm VERY conscious of the fact that the first version of this utility I ever wrote, many years ago, ended up being used to generate LIT files which had some embarrassing glitches in them, but which were subsequently circulated quite widely. I intend to avoid such a situation with this one. Trouble is, once LITs generated by this do get out, the damage will have been done and we'll be past the point of no return with it. Maybe I'm fretting needlessly, but I don't believe that's in anyone's best interest.

Worst case is I just release the program with no LITs, and let anyone who wants light their maps themselves. This isn't so bad as it sounds, the Windows interface and PAK file support make it very newbie-friendly, and the speed (about 1 minute for an ID1 episode) helps a lot. Not much use to anyone on Linux or OS X who wants to join the party though.

Monday, April 27, 2009

MHColour 2009 Update 11

I think I'm really really close to done now. I have a set of LIT files for all of ID1 which appear viable, and which don't seem to have any issues with anything anywhere. So far I've done a run through all of e1 and good chunks of e2, e3 and e4 with solid results. There is some small colour balancing left to do, which is more of a finishing touches thing, just putting on the final coat of polish, rather than absolutely essential. Tomorrow (evening, GMT) looks on for a release!



I'm going to release a full LIT pack for ID1 together with the program and source code, and I'll be posting it on QuakeTastic (so that everyone can get it easily) together with announcements on OuakeOne and Inside3D. If anything happens to delay it I'll announce so here.

One remaining bug is to do with Hipnotic rotating brushes (why did it have to be Hipnotic again?) Depending on the brush orientation they don't seem to light correctly, so no colour at all is picked up by some faces. I'm not even remotely familiar with the horrible hacks they used to get these working, aside from knowing that they were horrible hacks. I'd still like to find a solution that might be at least sorta-kinda-workable here, but it won't hold up the ID1 release.

Rogue should work fine, as should any other mod that keeps things sane and sensible.

Regarding the program, the source code is an utter utter mess; not even remotely pretty. You have been warned. A major improvement could be made by switching it to C++ and building classes around some of the stuff in it, but other messy areas might be terminally so. There's a mixture of standard light.exe code, some of the brush model loader from the engine, R_RecursiveWorldNode is even in there, and a lot of stuff of my own that evolved organically and was removed and rebuilt many times over. Bring goggles.

Typical time to light an ID1 map is from 5 to 15 seconds, some take slightly longer (I think e3m1 is the big one, owing to all the slime surfs which need to be subdivided to generate light points), some shorter (dm1 runs in a second or two).

There are no hard-coded limits on map sizes in this.

Sunday, April 26, 2009

MHColour 2009 Update 10

I think I'm coming close to a release. One (pretty major) bug has surfaced: I obviously haven't been as careful with my mallocs and frees as I should have been, with the result that things have a tendency to crash in a pretty dramatic fashion. I'm temporarily working around it by just allocating 5 times as much memory as I need, but I think we'll all agree that's not a reasonable solution.

Visibility acceleration has just gone in; this is similar to the reason Quake II needs visibility before it can light a map - speed. Typically, compile times are quartered compared to what they would have been otherwise. If the map has no visibility it'll light it anyway, but as every surf must be checked against every light it takes significantly longer.

I've even picked up some work on DirectQ lately too; nothing dramatic but it was nice to do a little bit there.

Saturday, April 25, 2009

Colour mapping in action

I've shown lots of screenshots of the new UI, but here's the first of (one part of) the output from the program. This is the colour map that I've been taking about previously, showing how computed light sources generate colour.

You should note the shot for the e1 entry hall in particular, as it demonstrates the precise light source positioning.









Friday, April 24, 2009

MHColour 2009 Update 9

I think I'm at the "tweaking till it looks right" stage with things now. I've removed the lighting parameters from the right side of the UI, as it appears as though there's no real need for them. Also, I went back to a semi radiosity inspired solution. Now I have textures casting light to create a colourmap, which is then blended with the original lightmap to produce the final result. Surfaces are subdivided into a 64x64 (max) grid, so that lighting is evenly distributed across larger surfs. For solid surfaces, the light is then projected out 8 units along the surface normal (adjusting for backfacing) to get the position. It looks great, and also gives precise positioning of colour sources. Because it creates much more clour sources than there were lights, it means more processing to get the final result, but you can't win 'em all. I think the end result of this one is going to be well worth it.

Tuesday, April 21, 2009

MHColour 2009 Update 8

I've fairly successfully pulled back from my botched radiosity implementation, and now have just one remaining annoyance, which is that the little slipgate lights don't seem to be contributing at all in certain circumstances. Fix that and I'm going well.

Other advances include alias model loading and deriving torch colours from the skins (instead of having them hard coded). Haven't done sprites (yet) - the format seems a little messy compared to alias. I've also had to do some palette mangling as Quake's washed-out colours don't exactly give great results. Halving the lowest of the 3 components seems to do the trick rather nicely.

Also, I've added in some "fake fullbrights" to cope with all of the lights in Quake maps that have a name containing the word "light" but that don't use fullbrights (e1m1 has a few). This scales the textures so that brighter colours accumulate more, and adds in a colour factor for those.

Anway, barring utter catastrophe I now have a fairly solid deadline again; we're at most one week away.

Monday, April 20, 2009

MHColour 2009 Update 7

I started on the radiosity implementation yesterday, and very quickly made an unholy mess of the code, so right now I'm looking to pull back and go back to the old standard method. I got far enough to satisfy myself that there was no way it was going to work out well, so not too much time lost, thankfully, and at least I've satisfied my own curiosity.

Also, I've decided that rather than set torches to a fixed orange colour, I'm going to open the model and derive a light colour from it's texture. This means a fairly heady dive into progs.dat (with a mini progs interpreter built into the program!), but I've already completed that part. Model loaders come next (I'm thinking sprite and alias first, possibly followed by BSP - you never know what a mod might do!!!) This was prompted by the realisation that there's nothing to stop a mod having a blue torch if the author wants one.

All of that will require an enhanced PAK loader similar to the Quake engine, rather than the fairly rough and ready one I have at the moment, and of course it also means even more delays, but the end result will be a much better program.

Friday, April 17, 2009

Good news! Bad news! Good news! Bad news!

Slightly mixed at the moment.

WinMHColour 2009 is now working perfectly with classic Quake BSPs; LIT files that will work with unmodified BSPs are being generated and look great.

However, while it could be released (together with a LIT pack), there is some slight room for improvement in terms of handling lava, slime and torches. Right now the lava and slime just add colour ro lights that are in leafs with the relevant content types; this needs to be modified so that nearby lights are done instead.

Torches need a kind of radiosity added. This will prevent situations where a torch in an alcove (like in the episode room in start.bsp) only colours the alcove itself.

So to experiment and as a proof of concept with radiosity, I've adapted HalfLife's version of QRAD to work with Quake 1 maps (relatively trivial, you can do it in 10 minutes or so). Works beautifully and has the desired result.

Unfortunately, I can't just drop in code, as the Halk Life SDK's license is not very permissive at all (in particular the restriction that modifications derived from it may only work with HL). Fortunately, it's close enough to Q2's version that I can pull code from there. Alternatively, I have some ideas of my own.

What all this means is that something that works well enough right now won't be released for another while yet, but when it does happen it will work even better!

Wednesday, April 15, 2009

MHColour 2009 - RC1 (Internal Testing)

I'm at Release Candidate 1 which is currently for internal testing only. In other words, I think I'm done but I need to light the BSPs and ensure that all is well.

Here's the final UI. I've left plenty of space down the right-hand side for (hypothetical) future releases which might (if they ever happen) have more options (if I can find useful things to expose as options).

Tuesday, April 14, 2009

MHColour 2009 Update 6

Option 2 (original lightmap multiplied by a "colour map") is now up and running, seems to be going well so far. Advantages are that it's almost totally bug proof (as it uses the original light as a baseline), doesn't need extra sampling (so it runs fast), and simplifies the code a hell of a lot. Disadvantage (of course) is that it's not really usable (not at all in fact) as a general purpose light replacement utility.

Overall I'm quite happy with it, as a general purpose replacement wasn't an original goal, but just something that happened along the way.

So now I'm back to code cleaning and polishing things up.

Bugfixing

OK, I've identified the cause of the corrupt lightmap bugs. Unfortunately it's as I feared; ID changed the lighting algorithm slightly between that used for the released Quake maps and that used in the released light.exe source. What this means is that on occasion the released source will generate a slightly different number of lightstyles to those which are in the otiginal maps; usually one less. I've confirmed that this is the case by doing a diff between the lightstyles generated on faces from a fresh release build of the earliest source available (on the ID FTP) and those contained in the original maps, using e2m1 for the comparison.

Options?

The most obvious (and easiest) appears to be to abandon the effort. Realistically, I'm never going to get an exact match with the original here. A proper replacement that's usable for all LIT supporting engines will require a full relight of the maps and a release of new BSPs as well as LITs. This seems preferable for a number of reasons, but it's unrealistic too, as expecting people to replace their BSPs seems a step too far. There are also legal implications; while the GPL will allow a release of the compiled BSPs, it doesn't cover the textures in them.

A second option is to drop the general light utility functionality and focus purely on colouring. This would involve building a "colour map" for each surf, which would just contain an amount (in R/G/B) to multiply the original lightmap by, then just multiplying the original lightmap to derive the LIT. Fast and accurate, but would require some fairly radical restructuring of the code.

A third option seems to be to do something to approximate the data. Using a combination of the original light data, the new light data and style info, it should be possible to get reasonably close to what the original had. I'm not certain how well it would work, or if the effort required would be worthwhile, but this seems to be the a reasonable second-best approach, assuming that a release of some kind is the objective.

Other options involve various brute force methods, which are not really up for consideration.

I think what I'm going to do is take the "colour map" approach on a forked copy of the code and see how far I get with that, and save the data approximation as a fallback.

Monday, April 13, 2009

Fun with the ListView Control/MHColour 2009 another update

You've just gotta love Microsoft documentation. Well, actually you don't. In fact you must despise it with every fiber of your being until it becomes a defining purpose for your life. Or something.

Anyway, in working on the UI for MHColour 2009, I wanted to add some features like groups and a highlighted first column. Seems simple enough, so here we go; use ListView_InsertGroup for the former and ListView_SetSelectedColumn for the latter. Compile it; oops - unresolved external. OK, check them up on MSDN again. No joy, no indication of what's up. Right, shove unresolved external Listview into Google; no joy.

Long story short, opening up commctrl.h reveals that it needs a #define of _WIN32_WINNT to pick these up. THANKS A LOT GUYS. Pop this into the MSDN search and there it is, under the heading of "Using the Windows Headers". But - unless you knew to look for these - there is NO WAY ON EARTH that you are going to find it. What's wrong with adding one teensy weensy line to the documentation for ListView_InsertGroup etc? Just one line, "you need to define _WIN32_WINNT to 0x501 or higher to use these"?

This is an exact repeat of the experience I had learning Direct3D; all the info was there, but in isolation. There was no feel for how things fit together. I know that Win32 is semi-officially "dead", and that MS would like app developers to switch to .NET, but let me ask them a few questions. What if one has a legacy codebase consisting of perfectly good (and debugged) code that one just wants to spruce up a little? Where are the next generation of low level native programmers going to come from? How many hours of programmers time have been wasted so that MS can save a few minutes?


Enough of that. I'm only getting to pick up the code again now, so - assuming all goes well with the bugfixing - it'll be late this week or early next week at the soonest.

Sunday, April 12, 2009

Fixing a broken MSDN

This has been bugging me for a good while, and I note from various forum posts and Google searches that it is a fairly common problem. Microsoft seem to have brushed the whole thing under the carpet; now, this may or may not be sinister, but there are a lot of search results that give links to pages on the MS website, but when you click on them you get a generic "page cannot be found" type message.

The primary symptom is that your MSDN collection comes up with a blank table of contents, an empty index, and a "This program cannot display the webpage" message on the MSDN start page.

Other people who have gone looking for help have been sent round in circles, with no clear solution that works for all.

Today I fixed it. No, it doesn't require reinstallation, no, it doesn't require re-registering of DLLs. Quick, clean and simple.

  • Search for NameSpaceSharp on the web; this is an MS tool that lets you view and edit the namespaces registered with the help system used by MSDN (and others). Download and install it.
  • Open MSDN, go to the "Search" page, and type something; I used CreateWindow.
  • You'll get a "Namespace not specified" message, followed by something beginning with ms-help:// - take a note of everything after that (in my case it was MS.MSDNQTR.2005APR.1033)
  • Open NamespaceSharp, and expand "Registered Havana Namespaces"; it will be blank so right-click and select "Register a new namespace".
  • Enter the name from above, the description is optional, and browse to where you have MSDN installed and select the msdn.HxC file, then hit the "Register" button.
  • Now right-click on the new namespace and select "Register Help Files", browse to where you have MSDN installed again, and do a Ctrl-A to select all files.
  • This will take a minute or two to register; when done just open MSDN one more time and you should see the magical "Help is updating..." message pop up - bingo, fixed!

Wednesday, April 8, 2009

MHColour 2009 Update 5

I've changed the GUI a little bit again; just some reorganising of the controls so that it looks cleaner. The list box of selected BSPs has been replaced by a list view giving you some useful info such as the number of faces and entities, as well as the current light data size, in a BSP. This can help to guess how long a given BSP may take to light.

Extra Sampling! I now have up to 32x Extra Sampling available! I haven't tested it all the way up to this, but I have done trials of 8x, which looks sweeeeeet but is very slow to process. Some day PCs will be fast enough to do 32x.

There's really no limit on how high this can go, by the way (aside from factors such as available memory and CPU speed), but I decided that 32x was well into the realms of stupidity and stopped there.

Bugs! There is a bug that seems to occur when switchable lightstyles trigger; lightmaps go corrupt. Obviously it's unreleasable in that state, but at least I have a definite test case now, so I can work on fixing it.

That's about all for now; off to Real Life for a few days!

Tuesday, April 7, 2009

MHColour 2009 Update 4

The PAK integration is now working fully, but unfortunately time is not on my side and I won't be able to release tomorrow. If I was an hour or two earlier I would have done so, but Real Life has me fully in it's clutches until Friday at the earliest, following which I'll be short on spare time for another few days. I hope to use whatever chances I do get between now and then to finish cleaning up some things, then by Sunday or Monday do a full ID1 LIT file build, bugcheck the maps, and all going well a release shortly after.

The UI hasn't changed much since the last screenshot, just some icons have been replaced. Another nice feature I have is that the TreeView will automatically open up pre-expanded at where it guesses your Quake game is installed. Right now it's just C:\Quake, C:\Program Files\Quake and C:\Games\Quake (but not only C, it checks each drive), but I can easily extend this one. Something I might do is read them from a config file.

One other required feature - I've tried generating LITs for Hipnotic maps before but they come out incredibly dark, so I'm guessing that the good people at Hipnotic used some of the -range or -dist options when lighting their maps (another filthy Hipnotic hack that makes things that otherwise work fine blow up with that pack - sigh...), so I might rebalance the generated coloured light so that it's as close as possible to the original intensity.

Monday, April 6, 2009

MHColour 2009 Update 3

The tree view is in; nice, eh?



About all that remains is to amend the BSP loader so that it's capable of loading from PAKs (I already have generic code for this in the palette loader, so it's just a matter of doing some string splitting), then do a bit of a clean up on the code.

Sunday, April 5, 2009

MHColour 2009 - Feature List So Far

This is a direct copy and paste from a Readme that will accompany the release, and gives a nice overview of the cool things this program is gonna be able to do.

Bugfixed
The original version made an attempt to write the lit data into the same filespace as the original data, but was not always successful, leading to corrupted lightmaps on a few surfs. This one is successful (I can't say "always", but the ID1 maps I've tested work fine).

Windows UI
I've replaced the old DOS UI with a proper Windows UI. It looks nice. No more messing with paths, or trying to make sure that you've got filenames right; just double-click the icon and browse for your maps. It even makes a (kinda) reasonable stab at guessing where you have Quake installed. Welcome to the 21st century!

Multiple file support
Think of it as a kind of batch processing mode. If you want to light multiple BSPs at once you can now do so.

Large BSP Support
All map limits have been removed. As a bonus, we're now a *LOT* more memory-efficient with regular size BSPs that ID light was.

Custom Palette Support
Fullbright colours in the Quake palette are used for helping to determine light colours (a light close by a texture which has fullbrights in it will pick up it's colours from those fullbrights). Previous versions used a hard coded palette in the executable, but no more! You can now use this with mods that have a custom palette. It will even pick up a palette from a PAK file, and searches relative to the location of the current BSP it's working on. If it can't find a palette on disk it falls back on it's own internal copy of the Quake palette.

4 x Oversampling
This can give better results (although at the limited resolution of Quake lightmaps there's only so good you can get), but unfortunately it doesn't work so good with pure LIT generation, so it's only available as an option if you choose to also regenerate the light data in the BSP.

Lights BSPs in PAK Files
You can open a PAK file, browse the BSPs in it, and select to generate LITs for them. If you have selected to replace the light in the BSP it will work slightly differently (obviously), and will instead write the BSP out to your /maps directory (rather than stomping over the data in the PAK).


Right now I'm thinking that I'm going to miss my Tuesday deadline; all was well until I started adding in PAK file support, but I'm of the opinion that it's an important enough feature to justify the delay.

Another UI change might happen - I'm liking the idea of an Explorer-like tree view incorporating BSPs from both the file system and from PAK files.

Saturday, April 4, 2009

MHColour 2009 Update 2

We're now fully functional with the UI working well.





I didn't bother with getting the 4 x extra sampling done: far more important to get the correct colouring functionality in place. What I might do before I release is to only make the option available if we select "Overwrite BSP Light" - at least that way something is guaranteed to work correctly.

The next thing however is to take a large power tool to the memory handling of the light utility. I'm based on standard ID code, so I need to look out for mallocs that aren't free'ed, as I'm allowing multiple BSPs to be lit in a single exe session.

Informal tests indicate that even if there are leaks happening, they ain't too bad; I do need to be certain though.

The Dist and Range Scale options don't work well with writing the lightmaps into the same file space as the original (I might force them disabled unless we select to overwrite the BSP), and at times certain values cause a crash. I haven't checked out the old Error function yet, so that's also on the to do list.

Friday, April 3, 2009

MHColour 2009 Update

I have MHColour 2009 working now, writing the LIT file data into the same file space as was in the original BSPs but without the old bugs, at least in -extra mode. -extra4 mode still has bugs in the light data, which go away when you overwrite the BSP.

Right now what I'm working on is a Windows GUI for it - it is 2009, after all.





This is standard Win32 written in C, none of yer fancy stuff, which was quite an interesting experience to write. For all that Win32 has a reputation for being "difficult" and "awkward" (at least compared to .NET or Java), it was actually very pleasurable to write and has surprisingly little code (just over 300 lines in my "winui.c" file, which includes a lot of non-Win32 stuff, like setting the default folder for the Open File Dialog).

I can see myself doing some more work with the basic API as time goes on.

Anyway, enough of that - once I finish polishing the UI I'm going to see can I track down the -extra4 bug, but it's low enough priority, and if it holds things up too much I'll just remove the option.

Another thing I want to add is a "batch mode", letting you process more than one file at a time (think multi-select in the Open File Dialog).

Once I'm complete I'm going to build a new LIT file pack and release the whole lot under the GPL (the Q1 map sources are already GPL, and as the LITs are technically derived from those they can be GPL too). Hopefully next week - I've a deadline of Tuesday, beyond which Real Life is going to take over for a week or so.