Index | FAQ | Search | Memberlist | Usergroups | Profile | Log in to check your private messages | Log in | Register

NQ / QW Split Goto page 1, 2 Next

| Inside3d Forums Forum Index -> General Discussion

View previous topic :: View next topic
Author | Message
--- | ---
avirox Joined: 16 Aug 2006 Posts: 109 |

Posted: Thu Dec 31, 2009 2:17 am Post subject: |

I know this probably deserves a new thread, but.. The one most major incompatibility across modern quake engines is the fact that both Quakeworld and Netquake cannot run the same mods, and cannot join each others' servers. FTE is the only exception to this rule. Darkplaces can join QW servers, but cannot play them in singleplayer nor host them.
I won't go into the technical details on how either works or how such a project should be undertaken. I am neither qualified nor familiar enough with both to make such observations. However, this has been the single most polarizing factor in quake over the past 10+ years. The lack of multi-compatibility across quake engines for NQ and QW protocol support has made the QW community unaware of NQ mods, and vice verse.
It has also separated the player base in a bad way. People still play netquake and quakeworld. However, if you ask players from either community - 9 out of 10 will tell you "what, people still play netquake/quakeworld?". Yes, we take it for granted on these forums that both iterations of quake 1 exist, and that people still play both. However, we here but make up a small fraction of that player base (and I'm guessing most here are on the NQ side).
ZQuake had some NQ compatability going on at one point iirc, but that has not crossed over to FuhQuake/EZQuake. Speaking strictly as a Quakeworld player, if we can reach out to the EZQuake devs (some of whom are open to new ideas) and if the NQ community can reach out to the devs of their favorite engines, we can really breach a fissure that has persisted for far too long.

Back to top |

LordHavoc Joined: 05 Nov 2004 Posts: 243 Location: western Oregon, USA |

Posted: Thu Dec 31, 2009 4:17 am Post subject: |

There are a few major distinctions between NQ and QW: different progdefs.h - this requires a significant amount of work on the progs.dat loader to adapt the fields to match what the engine code uses internally player movement - there is some QuakeC code used in NQ that is not used in QW (instead handled by the engine, in a way that matches the prediction), so there is a question of correctness here (a QW client would mispredict all the time, unless the NQ player jumping/ water movement code in qc is bypassed). some mods simply can not operate properly in QW regardless (this is why there was never a QW version of qrally for example)
less particle effects - NQ has several customizable particle effects that are not present in QW (this is why the QW port of hipnotic mission pack has blood effects for forcefields for example - the custom particle() function simply does not exist, the protocol has no svc_particle for it, etc)
differences in QuakeC builtins - in particular the monster ai code is missing in the QW side, a lot of code to sync up, or both versions being used according to the type of mod loaded
differences in network layer - the net_dgrm.c code has no similarity whatsoever to the netchan.c code and it is a fairly significant amount of work to unify the two and add back singleplayer support to the netchan framework.
There are some more subtle differences, but those are the main things that take significant time to code.
P.S. there are also some less technical issues to consider, such as:
named teams - QW uses 4-letter team names, NQ uses pants color, supporting both is not hard on the engine side but the differing functionality can be unexpected by players (team being ignored on an NQ mod, or an NQ client being unable to choose a team on a QW mod)
differences in deathmatch/teamplay cvars, death messages, etc - these differences are entirely in the qc code but may be unexpected by players
differences in entity effects - QW has a different set of entity effect flags, including red and blue glow effects, CTF flags being carried by a player (where an NQ player would not see the CTF flag unless the engine attempts to network an additional entity for it, and a QW player on an NQ CTF mod would see the player flag lagging way behind the player), and other features that do not translate well
Last edited by LordHavoc on Thu Dec 31, 2009 4:28 am; edited 1 time in total

Back to top |

metlslime Joined: 05 Feb 2008 Posts: 177 |

Posted: Thu Dec 31, 2009 4:18 am Post subject: |

avirox wrote:
The one most major incompatibility across modern quake engines is the fact that both Quakeworld and Netquake cannot run the same mods, and cannot join each others' servers.

I agree this seems like a seperate project from what was proposed above, but you have a valid point that this is an issue.
What would it mean for an engine to support both? I guess the client needs to be able to accept both protocols, and the server needs to be able to handle both types of progs.dat and adopt a protocol that is compatible with the chosen progs.dat. Both of these are not trivial, but seem like they can be done in any current engine without any unexpected trouble. (There are probably additional details missing from my list... does quakeworld have any special UI stuff like server browser or anything? New console commands or features that are required for quakeworld?)
However, what engines would need to adopt this support in order to solve the "fractured community" issue? Darkplaces alone obviously hasn't solved it. Perhaps you would need to hit proquake, fuhquake (or ezquake?), joequake, and fitzquake in order to target all known communities? And would even that level of support solve the problem?

Back to top |

LordHavoc Joined: 05 Nov 2004 Posts: 243 Location: western Oregon, USA |

Posted: Thu Dec 31, 2009 4:34 am Post subject: |

metlslime wrote:
However, what engines would need to adopt this support in order to solve the "fractured community" issue? Darkplaces alone obviously hasn't solved it. Perhaps you would need to hit proquake, fuhquake (or ezquake?), joequake, and fitzquake in order to target all known communities? And would even that level of support solve the problem?

The proquake code is nearly unmaintainable, especially the network layer which would have to be overhauled for QW support, there is a ton of screwy stuff in there for "cheatfree" mode in proquake.
Regarding darkplaces and running QW mods - I never attempted this because QW already had a standalone server (qwsv) which the mods are designed for, and QW was never aimed at singleplayer in the slightest, so being able to run a qw mod in singleplayer did not occur to me, besides being very difficult to add that functionality (supporting the protocol in the client was relatively easy but the server side differences are far more significant).
It was simply never a goal for DP, and it is a difficult one.
Now running NQ mods in a way that supports QW clients, does have some appeal to me, but the network protocol is quite poor for what NQ mods expect to be able to do.
This is a valid point, just a very difficult one to solve in any meaningful way, overall I'm still fond of the teamfortress approach (progs.dat and qwprogs.dat included, built from the same source using some preprocessor directives).

Back to top |

scar3crow Inside3D Staff Joined: 18 Jan 2005 Posts: 837 Location: Las Vegas, NV

Posted: Thu Dec 31, 2009 4:39 am Post subject: |

I split this from Urre's standards thread, agreeing with avirox's opening comment, and seeing that it had received suitable attention already, I didn't want either discussion to get interrupted. _________________ ...and all around me was the chaos of battle and the reek of running blood.... and for the first time in my life I knew true happiness.

Back to top |

metlslime Joined: 05 Feb 2008 Posts: 177 |

Posted: Thu Dec 31, 2009 5:07 am Post subject: |

Lordhavoc: Some of your objections might go away if we just assume that the mod (progs.dat or qwprogs.dat) determines the protocol: i.e. if a server runs a NQ mod, it will be using a NQ-derived protocol, and likewise for QW mods. That would solve, for example, particle efffect issuesl.
However it sounds like you still raise some difficult problems. For physics i guess you would also run QW or NQ physics depending on the mod. For some of the other issues i don't know enough to know if they are solvable.

Back to top |

leileilol Joined: 15 Oct 2004 Posts: 1321 |

Posted: Thu Dec 31, 2009 6:19 pm Post subject: |

The biggest suck to me is QW's predicted, mandatory PlayerJump function. That alone, kills all sorts of creative fantasy for QW. _________________

Back to top |

Baker Joined: 14 Mar 2006 Posts: 1538 |

Posted: Thu Dec 31, 2009 6:53 pm Post subject: |

avirox wrote:
I know this probably deserves a new thread, but.. The one most major incompatibility across modern quake engines is the fact that both Quakeworld and Netquake cannot run the same mods, and cannot join each others' servers. FTE is the only exception to this rule. Darkplaces can join QW servers, but cannot play them in singleplayer nor host them.
I won't go into the technical details on how either works or how such a project should be undertaken. I am neither qualified nor familiar enough with both to make such observations. However, this has been the single most polarizing factor in quake over the past 10+ years. The lack of multi-compatibility across quake engines for NQ and QW protocol support has made the QW community unaware of NQ mods, and vice verse.
It has also separated the player base in a bad way. People still play netquake and quakeworld. However, if you ask players from either community - 9 out of 10 will tell you "what, people still play netquake/quakeworld?". Yes, we take it for granted on these forums that both iterations of quake 1 exist, and that people still play both. However, we here but make up a small fraction of that player base (and I'm guessing most here are on the NQ side).
ZQuake had some NQ compatability going on at one point iirc, but that has not crossed over to FuhQuake/EZQuake. Speaking strictly as a Quakeworld player, if we can reach out to the EZQuake devs (some of whom are open to new ideas) and if the NQ community can reach out to the devs of their favorite engines, we can really breach a fissure that has persisted for far too long.

Over time, I have drifted to the viewpoint that the NQ/QW fissure is a good thing.
I like the independent directions that the "4" communities have pursued:
FitzQuake/aguirReQuake = conservative single player enhancements for old school mapping
DarkPlaces + FTEQW = very, very strong modding capabilities and radical ideas
ProQuake/Qrack = mostly fine tuning and ease of use enhancements
ezQuake = Highly oriented towards DM play and competitive features
If there wasn't a fissure, there would be less experimentation and R&D.
I think the idea of closing the fissure is an interesting thing and worthy of discussion, but downloading another exe isn't really a lot of work.
I used to believe it would be great if there was just one single engine that did everything, but I have come to think that is a bad idea over time.
I personally believe that the Quakeworld community as of lately hasn't fostered the idea of "diversity" which used to be what made the Quakeworld community an exceptionally powerful driver of change and new features.
Note: ZQuake if it had a few dozen moderate changes (video mode switching, HL alpha texture support, my NQ physics "tweak", animation interpolation, support for giant maps, FitzQuake rendering option) could probably be my "one and only" engine for NetQuake, strangely enough.
I don't mind prediction, but what I really cannot stand is the QW bunnyhop. ZQuake is both conservative (feels FitzQuakey to me) and forward-looking.
@ LordHavoc: ProQuake cheat-free is dead! There are no servers running it remaining (except one in Edmonton that no one ever uses).

Back to top |

avirox Joined: 16 Aug 2006 Posts: 109 |

Posted: Thu Dec 31, 2009 8:14 pm Post subject: |

Baker: you make valid points, but the time when such a fissure has been "beneficial" has come to an end. Indeed, the reasons you listed above are probably in part why no such engine has been made until recently. However, now - with the vastly diminished size of both communities - such fissures have little merit in my opinion. Combining the two would strengthen the online player base. It would also get people interested in the different mods each side has developed.

Back to top |

Baker Joined: 14 Mar 2006 Posts: 1538 |

Posted: Thu Dec 31, 2009 9:22 pm Post subject: |

avirox wrote:
Baker: you make valid points, but the time when such a fissure has been "beneficial" has come to an end. Indeed, the reasons you listed above are probably in part why no such engine has been made until recently. However, now - with the vastly diminished size of both communities - such fissures have little merit in my opinion. Combining the two would strengthen the online player base. It would also get people interested in the different mods each side has developed.

I'd like for there to be NQ and QW support in every engine.
Eventually I'd like to do this in ProQuake but first I'd need to understand the network code in both NQ and QW better. And that's time consuming. I think Rook took a stab at adding Quakeworld support and gave up on it for being too much world. ZQuake has "nqconnect" as I understand it, but I haven't tried it.
FTEQW server is definitely a better option with all the functionality, but there are a lot of obstacles.

Back to top |

mh Joined: 12 Jan 2008 Posts: 909 |

Posted: Thu Dec 31, 2009 9:41 pm Post subject: |

I wonder would it be easier to backport NQ support to QW? Of course, if your engine is already significantly modified the answer is "no", but all the same, it's an interesting question. _________________ DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines

Back to top |

Spike Joined: 05 Nov 2004 Posts: 944 Location: UK |

Posted: Fri Jan 01, 2010 2:17 am Post subject: |

Actually, most qw engines nowadays already support many of extra twiddles of the mission packs, and a little more. FTE servers are able to run either NQ or QW mods via either the NQ or QW protocol, converting and emulating missing parts both ways. At least that's the theory.
Note that the FTE QTV proxy is able to accept connections from NQ or QW clients, but can only connect to QW servers. Admittedly its not bug free, but its one possible solution. I wanted to build it into a pluggable module at one point that could potentially be easily added to an NQ engine, and this would give easy qw protocol support (and qtv+mvd playback).
Strictly speaking, QW clients don't need predicion. The entity deltas could be made to feed through to the existing entity system instead easily enough. Tbh, there arn't all that many differences. Usabilitywise, the only serious difference is the team cvar/setting. The rest should be just additions.
_________________
What's a signature?

Back to top |

r00k Joined: 13 Nov 2004 Posts: 483 |

Posted: Fri Jan 01, 2010 7:17 am Post subject: |

LordHavoc wrote:
Now running NQ mods in a way that supports QW clients, does have some appeal to me, but the network protocol is quite poor for what NQ mods expect to be able to do.

I have a few netQuake mods I would like to port to DP, as a server,which could support FTE/QW clients via dp7, i woul love some sort of rough outline for DP7 support for netQuake strictly as a baseline non qw prediction layer that is truely netquake compatible.

Back to top |

LordHavoc Joined: 05 Nov 2004 Posts: 243 Location: western Oregon, USA |

Posted: Fri Jan 01, 2010 3:36 pm Post subject: |

r00k wrote:

LordHavoc wrote:
Now running NQ mods in a way that supports QW clients, does have some appeal to me, but the network protocol is quite poor for what NQ mods expect to be able to do.

I have a few netQuake mods I would like to port to DP, as a server,which could support FTE/QW clients via dp7, i woul love some sort of rough outline for DP7 support for netQuake strictly as a baseline non qw prediction layer that is truely netquake compatible.

I wish I had time to write up a tutorial for DP7, but you can find everything dealing with it by searching for PROTOCOL_QUAKEDP in the darkplaces source (PROTOCOL_DARKPLACES7 is never checked for, because it is the "default" mode, the code represents differences from DP7 rather than differences from PROTOCOL_QUAKE or PROTOCOL_QUAKEDP).
PROTOCOL_QUAKEDP is darkplaces' extended quake protocol from way back, it was used by nehahra for multiplayer, it's the most similar to stock QUAKE protocol (it simply added more entity flags and flags for extended sound info and such).
But the important part is that PROTOCOL_QUAKEDP will show you all the places that DP7 DIFFERS from QUAKE/QUAKEDP protocol.
The major differences are in these areas of networking:
cl_input.c CL_SendMove - the clc_move message this writes has higher precision, and a great deal of extra data (for the prydon gate cursor extension, reporting what is at the crosshair), it also has a "move sequence" which represents the current move number in the game for prediction, this is 0 if cl_movement is off (no prediction), otherwise it is used to synchronize the prediction to what the server has parsed from the client (how many moves have been executed on the server side), so that the prediction knows what data has been used and what data is still unconfirmed (and thus needs to be predicted).
After clc_move is sent it sends some clc_ackframe messages which correspond to all the server frames it has received since the last clc_move was sent out, this is critical for the server to function properly (it is how it knows what data the client has received).
cl_parse.c CL_ParseClientdata - SU_ITEMS and other stats are not networked here in the case of DP7, all stats are updated by the svc_stat* messages which are sent as unreliable messages (their receipt is confirmed by the clc_ackframe handling).
common.c MSG_WriteCoord* and MSG_WriteAngle* - these functions represent alternate encodings of coordinates and angles used by the rest of the DP7 protocol code, and have matching MSG_ReadCoord* and MSG_ReadAngle* functions.
netconn.c - this contains some packet layer functions for the QW-like networking that is used instead of the legacy quake protocol for connecting to servers, do a search for CCREQ to find things here, basically connect messages in darkplaces begin with 0xFFFFFFFF and then a text string (like in QW, Q2, Q3) rather than the specialized CCREQ_ numbers, so a different connect process is required.
The DP7 connect process begins with a getchallenge message and the server replies with a challenge response, then the client gives back that challenge code to the server in its connect request, then it proceeds normally like in Quake, the reason for this is to prevent qsmurf attacks (where someone slams a list of quake servers with connect requests spoofed to point to an intended victim of the flood, and the servers spend 5 minutes trying to talk to this person, which also makes it hard for legitimate players to connect), this connect process was inspired by Quake3 network documentation.
You can find the code for this process in NetConn_ServerParsePacket and NetConn_ClientParsePacket, where getchallenge, challenge, connect, and other commands are handled.
protocol.c EntityFrame5_ functions - this is the heart of DP7 networking, it is the loosely synchronized Tribes-like entity network code, and stat code which operates in the same way.
The heart of the Tribes-like entity/stat networking is this:
Each frame a packet is emitted containing some differences from the previous frame (I say some because often not all data fits, so this is only a partial update, this is why there are no packetoverflow problems in huge levels full of monsters in one room), along with these updates being put into the packet, it also builds a "packet journal" describing the same differences (such as which fields on an entity were sent this frame, and also which stats were sent).
Overall this is a straightforward delta of the entity/stat database for this player, and each frame it makes an attempt to fit all the differences in the database into the packet, thus nullifying the differences in the database.
Some may call this "dirty flag" networking.
Now the reason for the packet journal is simple - when the client input comes in, it contains some clc_ackframe messages, each one of these refers to a packet journal by frame number, any messages for packet journals that have been deleted (already confirmed as received) are simply ignored, any messages that correspond to packet journals will cause those packet journals to be deleted (as they have been confirmed), if a packet journal is older than the frame number specified it is now known to be a LOST packet, and its differences (entities sent, what fields were sent on each entity, and the stats sent) can now be flagged as dirty again, then the packet journal for the lost frame is deleted as it is no longer useful.
So server->client packet loss is handled by this clc_ackframe mechanism, where any lost update is restored so that it will be sent again.
There is an extra detail to this process - when a packet journal is lost, it only needs to restore the entity field flags and stat flags that have NOT been sent in any later packet journal in the database, because any later update would have already arrived by the time this repeat does.
It is worth noting that all networking uses the latest data available on this entity, which is why packet journals do not save the value sent, only the flag for the lost data itself.
If there are too many packet journals, the server can simply mark them ALL as lost (rolling their dirty flags back into the database) and start over, this mostly happens when someone is experiencing a multi-second lag spike and not confirming any packets at all.
sv_main.c SV_SendServerinfo - this calls EntityFrame5_AllocDatabase for the client, initializing the database for networking entities and stats for this level to this player
sv_main.c SV_WriteClientdataToMessage - this does not send the various SU_ flags when using DP7 protocol, because they would be redundant, however it does build a stats array which will be networked later by the EetityFrame5_WriteFrame function.
sv_main.c SV_SendClientDatagram - this uses different packet size limits based on the protocol used, for DP7 1400 byte unreliable packets are used instead of Quake's 1024 limit, and for a local client (singleplayer/listen server) the limit is 65536 bytes, so that singleplayer has the best experience in crowded levels.
sv_main.c SV_ReadClientMove - this is where the upgraded clc_move message is parsed, the cursor details could be ignored along with the move sequence (but you still get better aiming precision), the client will not predict as long as the server tells it that its last received move sequence is 0.
Now an overview of the entity networking, NOTE: all of this is in protocol.c and can be used as-is, this explanation is only informative.
The client plays dumb and simply reads updates, and echoes back frame numbers received, the updates are applied immediately to the entities, and any entities not mentioned in this packet are assumed to be identical to before (where as in QUAKE protocol they are assumed to be removed), interpolation is done between the two latest states received.
The server keeps track of which frames have been sent but not yet confirmed by the client, essentially this boils down to 3 states for each frame in the packet journal: pending (sent, no confirmation yet), lost (sent, later frame was confirmed by a clc_ackframe message but not this one), received (sent, this frame number was specifically referenced by a clc_ackframe message)
Whenever a packet is lost the server restores the update flags it represented (but first clearing any update flags that match later packet journals, no point in sending it redundantly), so that the next packet will attempt to fix the state.
For each player an entity_state_t array (snapshot) is built representing the entities the player can see, the player also has an entity_state_t array representing the current state of everything, and an update flags array (which fields need to be sent), and an entity priority array, the snapshot is compared to the entity database for the player and update flags are set accordingly for any differences detected.
Entities are prioritized by distance from the player and other attributes, how this is done is not particularly important, but the priority must continually increase - in DP it increases priority by 1, it also checks if the entity is within 1024 units and increases priority by 1, it also checks if it is a full update, attachment, has a colormap change - like a team change on a player - or changed model, and increases priority by 1, the net result is that priority is increased every frame by 1, 2, or 3, it is capped at 31.
Entities are added to lists by priority number and the lists are walked from highest to lowest, any entity that is put into this packet has its priority drop to 0, and it is added to the packet journal (to handle packet loss).
Stats changes are simply networked individually, but their receipt is confirmed by the same packet journal system, this is done by comparing the stats array built for this player against the stats database for the player and update flags are set.
There you go, that is an overview of the DP7 protocol code in DP.

Back to top |

Spike Joined: 05 Nov 2004 Posts: 944 Location: UK |

Posted: Fri Jan 01, 2010 3:57 pm Post subject: |

LH, I'm not entirely sure that that's what he was asking about. r00k: As far as mods are concerned, DP7 is meant to be backwards compatible (just not for the engine). As far as prediction is concerned, the defaults are meant to match regular quake physics so as long as you don't change the player movement code (just the fields+cvars) prediction will work fine, this is the same problem you'd have with QuakeWorld prediction, just that it feels more like NQ.
_________________
What's a signature?

Back to top |

Display posts from previous:

| Inside3d Forums Forum Index -> General Discussion | All times are GMT Goto page 1, 2 Next
Page 1 of 2

Jump to:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

Powered by phpBB © 2004 phpBB Group