Inside3D!
     

Proposal - Protocol Extensions

 
Post new topic   Reply to topic    Inside3d Forums Forum Index -> Engine Programming
View previous topic :: View next topic  
Author Message
mh



Joined: 12 Jan 2008
Posts: 909

PostPosted: Sat Nov 29, 2008 12:54 pm    Post subject: Proposal - Protocol Extensions Reply with quote

OK, lot of things floating around that need a new protocol number, and it's a one-way ticket to a horrific mess. Engine #1 ends up supporting protocol 15 and 16, engine #2 ends up supporting protocol 15, 16 and 17, but the two implementations of protocol 16 are incompatible, so a client/server session between the two engines will crash and burn.

We need to clean this up, so my proposal is that we define a "protocol extensions" mechanism.

Rather than bumping to a new protocol number every time, instead we implement a single "protocol 666" which tells the engine "I'm using something other than stock protocol 15 here, so watch out".

Each supported extension is a #define, so the items we need to standardise are (1) that this is the way we'll do it, and (2) the list of defines.

If using protocol 666, we read a series of ints from the server in CL_ParseServerInfo (it could be only one, but I prefer to build this to support more than 32 extensions from the outset) and store them in an extension flags member of the client struct.

The last thing we need is a way for the client to tell the server what extensions it supports. This can be set up in a similar manner to the above, and sent during CL_SignonReply.

So now we have a client which knows which extensions the server supports and a server which knows which extensions the client supports, so we take the common items which gives us which extensions we actually use. The server checks with a bitwise and when sending the data, and adjusts it's behaviour accordingly. The client checks with a bitwise and when reading the data and adjusts it's behaviour accordingly. Both can fall back to protocol 15 if any given extension is unsupported by both.

This then puts us in a position where the proposed "new standard protocol" can be implemented as quickly and easily as possible in as many engines as possible, doesn't have to be an "all or nothing" implementation (engines can pick and choose which features they want), preserves compatibility between protocol version numbers, and will always have a fallback to standard protocol 15 behaviour if something isn't there.

Comments welcome. Wink
_________________
DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines
Back to top
View user's profile Send private message Visit poster's website
frag.machine



Joined: 25 Nov 2006
Posts: 728

PostPosted: Sat Nov 29, 2008 1:57 pm    Post subject: Reply with quote

I've been playing around with this idea, and implemented a special message to send a extension string very similar to the DP_CHECKEXTENSION mechanism (which in turn was cloned from the OpenGL extension mechanism). The integer list is a smarter, network-wise variant. There are a couple of problems here, though:

a) somebody needs to centralize the extensions codes/string names, or the whole thing won't work. Since LordHavoc already volunteered to do a similar work with the QC extensions, I vote on him to assume the task Very Happy;

b) everyone needs to agree about the meaning and behaviour of every extension, what can become a real problem. We can end up with either radically different versions of the same extension, or with lots of very similar extensions, polluting the definition table.

Otherwise, the extension check during handshake and selective fallback is great idea.
_________________
frag.machine - Q2K4 Project
http://fragmachine.quakedev.com/
Back to top
View user's profile Send private message Visit poster's website
mh



Joined: 12 Jan 2008
Posts: 909

PostPosted: Sat Nov 29, 2008 3:03 pm    Post subject: Reply with quote

On (a), it's a pity that we don't have a "Quake Standards Group" anymore; would have been perfect for such a central registry.

I'll concede (b) with the observation that everyone is going to end up doing their own thing anyway. Laughing It's something of a miracle that there is (almost) unanimous agreement on even things like use of a "textures" directory... Evil or Very Mad

(b) is also where proper documentation can make it live or die; if an extension is poorly documented or not documented at all, there's more danger of multiple variant implementations.

Anyway, in the spirit of "show, don't tell", i'm going to build a very simple "sample implementation" on stock Quake which should demonstrate things better.

_______________________

Some further thoughts - the server absolutely must control the protocol. My suggestion above of letting the client downgrade extension support is not a good one; it's feasible that a hypothetical protocol extension could be for anti-cheat mechanisms, for example.
_________________
DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines
Back to top
View user's profile Send private message Visit poster's website
Spike



Joined: 05 Nov 2004
Posts: 944
Location: UK

PostPosted: Sun Nov 30, 2008 2:24 am    Post subject: Reply with quote

Extension flags are good, just don't forget to allow negotiation of supported extensions.

FTE supports something like this, but tuned for QuakeWorld obviously.
In the connection protocol, the client sends a vendor/flags pair. The server masks it with what the server supports and sends the resulting vendor/flags back to the client in the svc_serverdata (svc_serverinfo is the nq name). It actually inserts the vendor/flags before the version, so the list is terminated with the protocol version that its extended from.
This mechanism allows future expansion and independant expansion.
vendorids would be stuff like LittleLong(*(int*)"FTEX") for example.
_________________
What's a signature?
Back to top
View user's profile Send private message Visit poster's website
metlslime



Joined: 05 Feb 2008
Posts: 177

PostPosted: Sun Nov 30, 2008 10:19 am    Post subject: Reply with quote

an issue with this negotiated protocol is that a demo is going to be frozen on whatever featureset the client and server in that one instance agreed upon. Now, i suppose the worst case is that only the original client should be able to play it, and any other client that supports a superset of that original client's feature set would be able to too, depending on how it's done.

------

An alternate approach I've been thinking about and might as well add here for discussion purposes:

Basically each protocol extension would be a new SVC or CLC number, rather than messing with existing SVC contents, and the second byte of any new message would be a "size" so that engines that don't understand the message type can skip it by reading the correct number of bytes.

That's the easy part. For extensions to the entity update message, we can reserve the last flag of each flags byte as a "more flags in the next byte" indicator, giving us plenty of flags. Each flag could indicate the presence of some new chunk of data, in the same order as the flag bits are read, and the only issue is how to indicate the size of that chunk to engines who don't know what it is. If we didn't care about bloat, each new chunk could have a size byte in front of it. On the other hand, we could simply say that all chunks are the same size and then there's no need to encode the size. (probably 1 byte, but 2 or 4 might be better depending on how people plan to use it.) If you need to send more than that amount of data, you'd just need to reserve multiple flag bits for that extension. (on a side note, choosing 1 byte might be a good way to encourage people to be frugal. I can't believe some protocols are sending floats for alpha, for example, when 255 gradations of alpha are more than enough.)

The advantages of this approach:

1. it's dead simple to implement "ignore stuff i don't understand", therefore any engine could be patched to ignore all of it in a few lines of code.

2. the demo contains all of the features the server supports, and a different client can play it later, and even unlock some of the latent goodness in it that the original client might have ignored.

Disadvantages:

1. Not the most efficient use of bandwidth, since more data is sent than the client may use. Also, a size byte on every SVC could be wasteful (depends on how many SVCs are generally sent in a frame.)
Back to top
View user's profile Send private message
metlslime



Joined: 05 Feb 2008
Posts: 177

PostPosted: Sun Nov 30, 2008 10:32 am    Post subject: Reply with quote

one more general note:

there are some extensions that, I think, would have to be required for the game to be meaningful. For example, if you have 2-byte modelindexes, how do you send that to a client that doesn't support that extension? At best, the client will have invisible entities, or entities with the wrong model.

So, should there be a way to tell the client to abort if the server decides one of the extensions is non-optional, but the client doesn't have it?
Back to top
View user's profile Send private message
mh



Joined: 12 Jan 2008
Posts: 909

PostPosted: Sun Nov 30, 2008 12:49 pm    Post subject: Reply with quote

metlslime wrote:
one more general note:

there are some extensions that, I think, would have to be required for the game to be meaningful. For example, if you have 2-byte modelindexes, how do you send that to a client that doesn't support that extension? At best, the client will have invisible entities, or entities with the wrong model.

So, should there be a way to tell the client to abort if the server decides one of the extensions is non-optional, but the client doesn't have it?

That's pretty much where I was coming from too, although I was thinking more along the lines of anti-cheat and ensuring all clients are on a level playing field.

The client would have to tell the server which extensions it supports, and following from that the server could either decide to kick the client or to rely on the good old "illegible server message" to cause to kick itself.
_________________
DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines
Back to top
View user's profile Send private message Visit poster's website
frag.machine



Joined: 25 Nov 2006
Posts: 728

PostPosted: Sun Nov 30, 2008 6:36 pm    Post subject: Reply with quote

mh wrote:
Some further thoughts - the server absolutely must control the protocol. My suggestion above of letting the client downgrade extension support is not a good one; it's feasible that a hypothetical protocol extension could be for anti-cheat mechanisms, for example.


Agreed. Also, the server must have some way to tell the client which extensions are mandatory (like large coordinates support, sound/model messages using short/int values, etc), and which could be ignored (fancy particle effects and other non-vital eye candy).
_________________
frag.machine - Q2K4 Project
http://fragmachine.quakedev.com/
Back to top
View user's profile Send private message Visit poster's website
goldenboy



Joined: 05 Sep 2008
Posts: 310
Location: Kiel

PostPosted: Thu Dec 04, 2008 7:11 pm    Post subject: Reply with quote

whatever you do, please document it.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    Inside3d Forums Forum Index -> Engine Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2004 phpBB Group