Inside3D!
     

Documenting protocols
Goto page 1, 2  Next
 
Post new topic   Reply to topic    Inside3d Forums Forum Index -> Engine Programming
View previous topic :: View next topic  
Author Message
metlslime



Joined: 05 Feb 2008
Posts: 177

PostPosted: Sun Nov 15, 2009 12:41 pm    Post subject: Documenting protocols Reply with quote

I took a shot at documenting the fitzquake protocol here: http://quakery.quakedev.com/qwiki/index.php/Fitzquake_Protocol

My approach was start with the nontechnical feature list at the top, useful for mappers/players/quakec peeps, then have section explaining the specific changes for engine coders, then finish with raw #define type stuff (also for engine coders.)

Any thoughts?
Back to top
View user's profile Send private message
mh



Joined: 12 Jan 2008
Posts: 909

PostPosted: Sun Nov 15, 2009 1:59 pm    Post subject: Reply with quote

Nice idea. Very Happy

I've wanted to add the FQ protocol to my engine for a while, maybe as the default, and now it's just so much easier than hacking around with the BJP protocols was.

The only thing I see lacking is that you don't seem to extend the positioning, so it's still quantized into shorts. It's not that big a deal (how many maps actually use positions beyond +/- 4096 in all directions? how many map editors support them?) but it would have been nice to see a U_LARGEPOSITION also.

One other thing I haven't yet checked in your code is whether or not you've increased the message buffer size, which will probably be needed at some point.
_________________
DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines
Back to top
View user's profile Send private message Visit poster's website
Spike



Joined: 05 Nov 2004
Posts: 944
Location: UK

PostPosted: Sun Nov 15, 2009 5:51 pm    Post subject: Reply with quote

one thing I would personally be interested in is if there is some way to detect if a client supports this stuff, rather than instantly crashing it if it does not, even if none of the extra bits is useful.

Also, U_LARGEPOSITION on its own is not enough. It needs to potentially apply to all coords.
_________________
What's a signature?
Back to top
View user's profile Send private message Visit poster's website
mh



Joined: 12 Jan 2008
Posts: 909

PostPosted: Sun Nov 15, 2009 8:08 pm    Post subject: Reply with quote

Ideally when the client receives it's PROTOCOL_VERSION from the server there would be a mechanism where it could say "nope, don't know about that one, here's a list of the protocols I do know" and the two of them could then work out something in common (with the option for the server to be able to specify certain things as mandatory, so as to ensure that all clients are on an equal footing so far as critical gameplay matters are concerned). Seems better than the current Host_Error, but it would need to be retrofitted to even current protocol 15 implementations and you just know that everyone's gonna do it differently. Laughing
_________________
DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines
Back to top
View user's profile Send private message Visit poster's website
metlslime



Joined: 05 Feb 2008
Posts: 177

PostPosted: Sun Nov 15, 2009 9:00 pm    Post subject: Reply with quote

Quote:
Quote:
The only thing I see lacking is that you don't seem to extend the positioning, so it's still quantized into shorts. It's not that big a deal (how many maps actually use positions beyond +/- 4096 in all directions? how many map editors support them?) but it would have been nice to see a U_LARGEPOSITION also.


U_LARGEPOSITION on its own is not enough. It needs to potentially apply to all coords.


Yeah, i spent some time trying to invent various tricks to get large coords to be only used on a per-need basis, to save packet size. The problem is you can't do it cleanly because quakec hand-generates some svc_ messages.

I'm mostly convinced to simply have a "large coords" variant, say 667, and just accept that packet sizes will be big, then. My other potential idea is to have a svc_coordssize message which can come multiple times during a packet, and resets the coord size for any messages that follow. So you could just set it to "large" before quakec starts writing out messages.

Quote:
Ideally when the client receives it's PROTOCOL_VERSION from the server there would be a mechanism where it could say "nope, don't know about that one, here's a list of the protocols I do know"


Another idea that would be good, but is there a way for a custom server to detect this without the client being custom too?
Back to top
View user's profile Send private message
mh



Joined: 12 Jan 2008
Posts: 909

PostPosted: Sun Nov 15, 2009 10:41 pm    Post subject: Reply with quote

metlslime wrote:
Another idea that would be good, but is there a way for a custom server to detect this without the client being custom too?

None that I'm aware of. A potential hack (and it's a hack) would be to send the protocol number and wait to see if you get any messages back from that client, but that would fall down as the client would disconnect following the Host_Error. Unfortunately it seems that it needs both; that Host_Error is pretty final.
_________________
DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines
Back to top
View user's profile Send private message Visit poster's website
Spike



Joined: 05 Nov 2004
Posts: 944
Location: UK

PostPosted: Mon Nov 16, 2009 12:28 am    Post subject: Reply with quote

stick an extra bit on the end of CCREQ_CONNECT, in VENDOR:VALUE 4-byte pairs, imho. Extensible and can co-exist between different engines. Eg:
"FITZ" 0x666
would say the client supports fitzquake's protocol version 666 and below.
Or you could use bitfields to advertise an individual feature that could work with any base protocol...
The protocols the server accepts would then be echoed back as a prefix (VENDOR:VALUE pair) before the protocol version.
Demos would work with any client that supports the protocol that was actually used (ie: either the client or server engine (assuming the server's engine supports everything as a client that it supports as a server), but not always anything else).
This is what I did with FTE's protocol. But being quakeworld it fully completes the handshake before the actual connection and serverinfo bits are purely for telling clients what they need to enable to play demos.
But yeah, postfix the CCREQ_CONNECT packet and then FTE can potentially use various extension bits.
Tbh, I'm not sure any other server would use that information, but it does seem a shame to kick clients purely because they're not XYZquake.

The documentation already up there is muuch clearer than I would ever write. :)

Additional thoughts:
svc_clientdata's stats should probably be sent as actual stats. add an optional svc_updatestat_byte to compact them slightly, and take the ammo/armour/etc counts out of the clientdata. Really you're not going to be shooting every single weapon every single frame... plus its a more extensible way.
svc_spawnbaseline2 should, imho, use the same format as entity updates.
svc_spawnstatic2: see spawnbaseline2, but note that the entity number is irrelevent.

That is, if you want the protocol itself clean, ignoring ease of implementation.

And a final note:
U_LERPFINISH is tied to nextthink? When deltaing, it would be more network-efficient to send the interval rather than the lerp end time. And by delta, I mean baselines, although true (qw/dp style) deltas would be preferable. But this would be sent a heck of a lot and even if the interval is 0.1 (because the server doesn't tic only at 0.1 sec intervals). And as its present within an extension byte, this extension byte will always be sent.
_________________
What's a signature?
Back to top
View user's profile Send private message Visit poster's website
metlslime



Joined: 05 Feb 2008
Posts: 177

PostPosted: Mon Nov 16, 2009 4:41 am    Post subject: Reply with quote

Spike wrote:
U_LERPFINISH is tied to nextthink? When deltaing, it would be more network-efficient to send the interval rather than the lerp end time. And by delta, I mean baselines, although true (qw/dp style) deltas would be preferable. But this would be sent a heck of a lot and even if the interval is 0.1 (because the server doesn't tic only at 0.1 sec intervals). And as its present within an extension byte, this extension byte will always be sent.


It's actually just the interval, and it's determined from inside SV_RunThink, which sets the time to exactly the current thinktime before executing the think function (so that the entity will think the time is actually the time it intended to think) and this adjusted time is used for determining the interval. So it really is exactly* 0.1 almost all the time (crucified zombies are the only exception in stock id1.)

It's also only sent for MOVETYPE_STEP entities or entities whose .frame has actually changed in the latest think function.

*not exactly 0.1 since that can't be represented in a float, but after encoding to a byte i ignore values 25 and 26 since they are the closest to 0.1 using the mapping 0.0-1.0 -> 0-255
Back to top
View user's profile Send private message
metlslime



Joined: 05 Feb 2008
Posts: 177

PostPosted: Mon Nov 16, 2009 4:56 am    Post subject: Reply with quote

Also @ Spike:

The handshake idea is interesting, i don't know much about how that connection stuff works, but it sounds like a stock client would send normal stuff, a modified client would send some extra data -- question is would a stock server crash on the extra data?

Or would the server have to ask for it, and in that case, would a stock client crash when it gets asked?

This does seem like a good way to sniff out the supported protocols, if we can figure out a way not to break when one of the parties is a stock engine.

---

As for the specific feedback about the protocol, of course this version is done since a version already shipped with it, but i may consider some of these ideas for future protocols (which i hope there will be very few of.)

My choices were mainly made with the goals of ease of implementation first, and minimizing packet bloat second, which is why it's basically just netquake with new flags and extra bytes as needed.
Back to top
View user's profile Send private message
Spike



Joined: 05 Nov 2004
Posts: 944
Location: UK

PostPosted: Mon Nov 16, 2009 11:53 am    Post subject: Reply with quote

if it is the interval/duration, then lerpfinish is a misnomer. :P
Which, yes, would only be sent when properly required.

Regarding connection stuff, the client asks for a connection, and the server sends an ack packet and instantly starts sending 20+ packets every second from a single potentially-spoofed packet... Which is my most major gripe about NQ protocols.
The problem is that with that you can spoof ip addresses and then fully clog up a server with X dead clients that the server is constantly sending packets to. So either a denial of service attack on the server, or use a few servers and you have a denial of service for someone else. Because the CCREQ_CONNECT packet can be spoofed, its not possible to ban the DoSer.
QW+Q2+DP onwards use 'challenges', where the client has to respond with a magic cookie before any client slots are actually consumed, so you know they actually have control of that ip before you start spamming it. Also, by having the client acknowledge a challenge, you can easily route through client-side NATs (so long as the client sends the reply to the server's new port).
But now I'm babbling.

I had a quick peek at the server source, and it looks like it would be just fine with extra data tacked on the end. From memory, proquake does just that to carry the extra info to get it through NATs. From memory, proquake include a magic code to avoid bad things from happening if others add stuff too, but I've not checked that.
_________________
What's a signature?
Back to top
View user's profile Send private message Visit poster's website
mh



Joined: 12 Jan 2008
Posts: 909

PostPosted: Mon Nov 16, 2009 5:20 pm    Post subject: Reply with quote

To my mind, the ideal protocol is one that's designed from the outset to be extensible. The basic version would be almost identical to protocol 15, but after sending the protocol version the server would send a number of flags.

First is "mandatory flags", a single int specifying which extra features are required (a server admin could be given the ability to toggle these via cvar), with a MOREFLAGS optional bit specifying another int of mandatory flags to follow (which can also have it's own MOREFLAGS bit, and so on). A client that doesn't meet these requirements is going down in smoke, either with a Host_Error immediately, or with an "illegible server message" at some point during gameplay, as the server will always send messages based on these.

Second comes "optional flags", which clients can choose not to support if they wish. The same MOREFLAGS pattern applies. The client reports back to the server which of these is supported (as part of the signon reply), and we're off. The server can filter out which of these items it will send messages based on, depending on the report back.

This would allow a single standard protocol to evolve without breaking compatibility, and would put control of protocol features in the hands of server admins (it would also enable mod-breaking features to be toggled as required). The downside is that every engine would likely end up with a partial implemenation, but at least things would have been designed to mitigate that.
_________________
DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines
Back to top
View user's profile Send private message Visit poster's website
mh



Joined: 12 Jan 2008
Posts: 909

PostPosted: Mon Nov 16, 2009 7:50 pm    Post subject: Reply with quote

Anyway, all that aside, I think we can now say that there's another engine that implements protocol 666 - or at least it will in it's next release. Wink
_________________
DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines
Back to top
View user's profile Send private message Visit poster's website
metlslime



Joined: 05 Feb 2008
Posts: 177

PostPosted: Tue Nov 17, 2009 9:03 am    Post subject: Reply with quote

mh wrote:
Anyway, all that aside, I think we can now say that there's another engine that implements protocol 666 - or at least it will in it's next release. Wink


Nice. I was suprised to read on your blog that you had trouble implementing edicts > 8192, i thought everything in your engine was dynamically allocated.
Back to top
View user's profile Send private message
Baker



Joined: 14 Mar 2006
Posts: 1538

PostPosted: Tue Nov 17, 2009 9:56 am    Post subject: Reply with quote

Nothing really to say except that I find the fact that metlslime is documenting the protocol to be epitome of the advancement of Quake.

Aside: If only this were done for ogg vorbis support (argh! more complicated than I thought. Streaming! Memory!), zip instead pak support, dynamic memory management (DP, FTE) and things like LHNET.
Back to top
View user's profile Send private message
mh



Joined: 12 Jan 2008
Posts: 909

PostPosted: Tue Nov 17, 2009 11:21 am    Post subject: Reply with quote

metlslime wrote:
mh wrote:
Anyway, all that aside, I think we can now say that there's another engine that implements protocol 666 - or at least it will in it's next release. Wink


Nice. I was suprised to read on your blog that you had trouble implementing edicts > 8192, i thought everything in your engine was dynamically allocated.

I wouldn't say trouble, just that I decided to leave it alone for now as there is other code outside of the protocol-specific areas that I need to look over first. I also wanted to get the basic communication up and running properly before adding in this one. I wouldn't really anticipate any bother with edicts though as I have them in a completely separate walled-off buffer that I just need to increase the max size of, and add in the checks for non-Fitz protocols.

Biggest trouble though was integrating it with the BJP protocols and setting things up so that I can switch cleanly between what's now 6 protocols in the engine. But that's more a problem coming from my engine than it is one from your protocol. Laughing
_________________
DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines
Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:   
Post new topic   Reply to topic    Inside3d Forums Forum Index -> Engine Programming All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2004 phpBB Group