Help with Quake Bot

[Tremor]

[LordHavoc]

You need to respond to each svc_signon message with the appropriate reliable command (prespawn, spawn, begin), or the server drops you for having a half-open connection (which has been used as a network flood exploit in the past - hence the "brief" 30 seconds - note that upgraded server protocols like darkplaces DP7 do not suffer from this exploitable connection process).

The keepalive messages should work fine (clc_nop), but be aware that the server tells you which port to talk to, the original connection port won't accept clc_nop and other packets (unless hosted by darkplaces servers running quake protocol where it uses a single port for quake servers, or fteqw running quake protocol which will do the same thing).

[Spike]

the protocol is built on spam.
you need to spam unreliables or you'll time out. unreliables contain some timing information which is used to determine pings etc.
either 60 or 72 per second is a good number. more than that is excessive and/or cheaty.
servers that support player prediction will require a continuous stream of movement messages or the player is likely to freeze in place despite having a velocity.
reliables require that you wait for a response before you send the next. you cannot trivially spam these, and must be prepared to resend the same message multiple times.
vanilla waited for a server message before sending an unreliable. this is unsafe with certain dodgy NAT routers. its generally safe nowadays, but your movement then becomes doubly-dependant upon server->client stalls.

each packet contains multiple svcs, both reliables and unreliables. once you've delt with reliables fragmentation, you can then just read the list of packed svcs from the message. each begins directly after the previous. there's no padding after the final byte (ethernet padding will not normally exist beyond the ip layer, so there shouldn't be any padding in your code).

the engine source is open. read protocol.h for the svc list. IIRC the simple ones contain a comment saying what data there is in the svc. for more complex things, go read cl_parse.c

you're probably getting dropped because you're not doing sequences with reliables correctly. this means the server will reject the message and then think you're not sending any (because you didn't send any VALID ones). again, reliables require that you wait for an ack before you can send the next, or to timeout and resend the previous one. pay attention to your sequences!

[Tremor]

LordHavoc,

I believe that I am doing the 'signon' process correctly and successfully. The first reply after the initial connect request gives me the new port to talk with the server, and I am certainly using that. After having done this sequence, the server sends me player informations etc. Before having sent the prespawn, name/color, spawn, and begin messages, it would certainly do just as you said - drop the half connected state.

Spike:

As far as the initial connection sequences go, I am waiting on a reply before sending the next. However, after the connection sequence, I don't currently have any reliables to send. Of course, I will have in the future when I build some functionality into it. What I am taking from what you and LordHavoc said is to simply send unreliables at the rate you specified and the server should not drop the connect? I was previously sending unreliable keepalives but LH seems to think that the noop will work? My goal (at this point) is to simply maintain a game connection and have the bot sit in observer mode indefinitely.

Also, I will most certainly check out protocol.h and clparse

[frag.machine]

I'd suggest you to use the source code as your primary reference, since the QSpecs are result of reverse engineering efforts back in 1996/1997.
Also I remember there were protocol changes between versions 1.01 and 1.06 (or was 1.08 ? Can't remember fro sure), even though the protocol ID was the same.

[Tremor]

[Tremor]

[Spike]

there is no such thing as 'observer mode', unless its some 3rd-party mod-specific thing - but that has nothing what so ever to do with the protocol.
by sequences, I mean the sequence numbers in the packets. if you repeat the sequence number, the packet WILL be dropped.

if you're testing with fte as a server, use sv_port 26000 (to default it for nq instead of qw), sv_listen_nq 1 (so you don't have to implement svc_stuffcmd properly yet), showpackets 1 (so you can see the packets the server receives without needing to resort to wireshark), developer 1 (so you can see stale/dupe sequences), showdrop 1 (so you can see dropped sequences). if developer or showdrop result in any network-related prints then it means that the server is dropping packets for some reason, on a lan this is always a bad sign. I expect DP has some equivelent settings, but I can't help with those.

note that a timeout error only occurs if no valid messages arrived - it really doesn't matter what's in the message, nops, moves, reliable, unreliable, whatever. It just has to be valid and complete. individual fragments don't count, but *any* unreliable does, and any reassembled reliable does.
timing out thus proves that your packets are getting dropped by the server's network layer (ie: before the clc parsing layer), or they're simply not arriving at all (wrong address or dodgy NAT or firewall or something out of quake's control).
also, remember that the sequence value (bytes 4-7) is big endian, just as the flags+length value is.

vanilla nq has some annoying quirks about exactly when particular messages are allowed. send things at the wrong times and you can easily crash clients. be glad that you're writing a client and not a server.

also, clc_nop is 1. 0 is clc_bad, and should never be sent.

[Tremor]

[LordHavoc]

You might want to start sequence at 1 but I don't recall off hand.

Note that reliable and unreliable sequence numbers are independent, and unreliable sequence numbers increment regardless of any reply from the server (they're not asking for a reply, they are unreliable by nature), whereas reliable sequence number advances when you get back an ACK packet (8 bytes, specific bits are set in the header only, it has no body).

The header you're seeing is two big-endian int32 numbers, one of which is the size of the packet and the type flags, the second is the sequence of the packet.

So if you send a 1 byte message (like clc_nop), be sure you send 9 as the size of the packet (header + content length).

The maximum size of a reliable message in stock quake is 8192 bytes, which is sent as 1024 byte fragments (that is, 1032 bytes with the header), the ack is sent back when the final fragment is received. I don't recall the details too well at the moment as it's been years since I poked at that code, but you may find netconn.c in darkplaces to be useful because I completely reimplemented the quake network stack there in one file (whereas quake had net_dgrm and a host of other pieces that worked in layers).

In general numbers in the quake protocol are little endian, except the header ints (2x int32) are big endian.

I believe 1024 bytes of content is the longest single packet that stock Quake can send or receive, darkplaces allows up to 65536 to be received, but won't send more than 1400 in one packet (to leave plenty of room for network tunneling headers like VPN stuff), of course that cap is dropped to 1024 in quake protocol for compatibility.

[Spike]

#define NETFLAG_LENGTH_MASK 0x0000ffff
#define NETFLAG_DATA 0x00010000
#define NETFLAG_ACK 0x00020000
#define NETFLAG_NAK 0x00040000
#define NETFLAG_EOM 0x00080000
#define NETFLAG_UNRELIABLE 0x00100000
#define NETFLAG_CTL 0x80000000
this is the first four bytes, in big-endian order (ie: as you see them written in hex).
as lh says, the length is the total length of the udp packet, including this 8-byte header... how he manages to pack 65536 bytes of payload in there I have no idea... surely the max is 65527.
the next four bytes are the sequence number.
the clc data follows that, in little-endian order.
don't bother with naks, and ctl messages are for connectionless things.
if you receive a reliable fragment, you will _NEED_ to ack it. failure to ack a reliable means the server will just keep resending it.
reliables will consume multiple sequences. if you have a 2500 reliable due for you, the first two packets will both have length values of 1032 with 1024 bytes of payload data. the flags for the first two parts will be the same, with just NETFLAG_DATA set. the sequences will be first and first+1, the third packet will have a payload of 328 (total length 336), with NETFLAG_DATA|NETFLAG_EOM, and sequence first+2. Every part of the message needs to be acked separately as they come. You can only safely parse the reliable once you receive the final EOM, because otherwise you'll find yourself wanting to read data that isn't available yet, so buffer+ack reliables until you receive one with EOM, and then ack that too then parse the whole lot as a single lump.
unreliables are never fragmented and don't use EOM at all.

you really do need to read the docs (ie: the engine source). one of:
net_dgram.c (vanilla, Datagram_GetMessage for reading, Datagram_SendUnreliableMessage for sending unreliables, Datagram_SendMessage, SendMessageNext, ReSendMessage Datagram_CanSendMessage, for sending+resending reliables.)
net_chan.c (fte, look for NQPROT ifdefs, the parsing code is 150ish lines net_message contains the entire inbound buffer and contains the result too but with a changed offset, and the transmit functionality (100ish lines) is meant to be spammed once a frame with the new unreliable buffer passed via arguments while the active reliable is in internally stored in chan->reliable_* and the waiting/next reliable is in chan->message.* which is bumped to active when the active reliable becomes clear due to being acked)
netconn.c (dp)

clients generally don't need to care about the exact limits. python probably hides all the memory limits for you anyway. you will need to support defragmenting the server's message.

[Tremor]

LordHavoc and Spike:

Thank you so much for taking your time on this. I appreciate all the input and extra info! I will study these last couple of posts in detail. I appreciate being pointed in the right direction on this. I'll update this thread as I progress - I may have other questions or whatever. Gonna need a day or so to check through the engine source.

I do have one last question for now.

I am indeed responding to each and every reliable. No doubt about it. However, I'm not currently buffering the fragments. So, as I understand it, I simply need to check that 1. the packet is reliable and 2. the length is 1032 - as soon as that happens, start buffering until an EOM appears? Are the quake specs correct when they say that an EOM message will always be a 0x09 reliable type?

[Spike]

DATA|EOM means its the final reliable in the block.
DATA means that its a section that is NOT the final one.
ACK means that its a reliable ack
UNRELIABLE means that its an unreliable message.
any other combination is invalid for an active stream (ignoring length and CCREQ/CCREP messages).

don't bother checking the segment length for validating reliable fragments, by convention they'll be 1024/1032 (depending on how you see it), but they might be smaller for MTU reasons, or larger for efficiency reasons if the engine is modified. The protocol works for anything between 1 and 65535, and remember that its a single segment so the maximum size can be somewhat unbounded... 20mb reliables is fun... yeah... you might want to provide a sanity limit for the max size somewhere (ideally more than 8192), but don't depend on the segment always being 1024.
you might want to ensure that a non-final segment is always at least 256 bytes though, as anything smaller is pointless and can result in wasted resources (read: probably a dos attack), just be sure to allow other sizes too.

[Tremor]

10-4, although I don't see the MTU as being problematic sine that's a layer 2 issue that should be auto-resolved/rebuilt. I suppose though, it wouldn't hurt to account for that.

[Spike]

ipv4 says an mtu must be at least 576. anything bigger is allowed to fragment. and if its allowed to fragment, its allowed to be dropped because of dodgy routers that can't handle it or hosts that think its a DoS attack or hosts that are merely memory limited.
be generous in what you accept, and strict in what you generate, that's the old moto. Which naturally means you don't test the things you're generous with accepting, opening your program up to all sorts of remote exploits, but hey...


ideally a server/client would use a size equal to totalsize/fragmentsrequired. then each fragment has the same sort of duration to it, which can make a difference on slow (56k) links with random reliables blocking the unreliables interspersed within it. keeping the sizes more consistant is just a somewhat sane idea, where possible.
but yeah, implementations ignore all that and just use 1024 for all but the EOM, in a really lame way.