Index | FAQ | Search | Memberlist | Usergroups | Profile | Log in to check your private messages | Log in | Register

Warsow/Qrack Web Map/Model Download (Curl)

| Inside3d Forums Forum Index -> Programming Tutorials

View previous topic :: View next topic
Author | Message
--- | ---
Baker Joined: 14 Mar 2006 Posts: 1538 |

Posted: Tue Aug 11, 2009 12:49 am Post subject: Warsow/Qrack Web Map/Model Download (Curl) |

Rook added some outstanding map/model download to Qrack which I have now added to ProQuake 3.99. The Qrack code was heavily derived from Warsow. I am going to describe the very few changes I had to make to implement this in the Windows version of ProQuake 3.99, which should be nearly the same as doing so with GLQuake or for that matter WinQuake.

Notes wrote:
Off hand I don't see anything operating system specific here either but haven't tried to compile a Mac OS X version. Curl is known to be multi-platform. When I do the OS X version, I'll write up whatever.

End Result
The end result is that connecting to a server, like ThreeWave CTF downloads the maps and model -- but not the sounds -- and does so very rapidly (seems instant even for 5 MB maps on my broadband) and reconnects.
Yes, DarkPlaces does it better but this is still very good but can definitely be improved upon.
But the nicest thing is how simple the modification is.
We begin ...
Requirements for this Tutorial
1. Windows operating system + MSVC6
2. Libcurl.lib (I have no clue which specific version Rook used off the CURL download page).
3. This version of the ProQuake 3.99 source but this should likely work with stock glquake source with Microsoft Visual C++ Express Edition but you'll be on your own but it shouldn't be too hard.
Questions and Miscellaneous Notes
1. The libcurl.lib gets compiled into the binary and the size of the binary is only increased negligibly (a few KB!). Excellent!
2. Does the binary work properly on old Windows 98 or ME? What about Windows 7? Or even Vista? I haven't tried any of those yet with the binary. I'm not going to make the assumption that, say, Windows 98 works with this because I just don't know.
3. The download works super-efficiently and is extremely fast. Far faster than really what you'd expect!
Tutorial
1. Add the following files to your project
You can borrow them from this (download) and be sure to take the libcurl.lib as well.

Quote:
Files to add: curl.h curlver.h easy.h multi.h
webdownload.c
webdownload.h

2. client.h
Add the yellow ....

Quote:
typedef enum { ca_dedicated, // a dedicated server with no ability to start a client ca_disconnected, // full screen console with no connection ca_connected // valid netcon, talking to a server } cactive_t;
typedef struct
{
qboolean web;
char *name;
double percent;
qboolean disconnect; // set when user tries to disconnect, to allow cleaning up webdownload
} download_t;

And still in client.h, add the yellow ...

Quote:
// connection information int signon; // 0 to SIGNONS struct qsocket_s *netcon; sizebuf_t message; // writing buffer to send to server download_t download;

3. Add COM_GetFolder to common.c
Add the yellow.

Quote:
/* ============ COM_GetFolder ============ */
void COM_GetFolder (char *in, char *out)
{
char *last = NULL;
while (*in)
{
if (*in == '/')
last = out;
*out++ = *in++;
}
if (last)
*last = 0;
else
*out = 0;
}
/*
============
COM_FileBase
============
*/
void COM_FileBase (char *in, char *out)
{
char *s, *s2;
s = in + strlen(in) - 1;
while (s != in && *s != '.')
s--;
for (s2 = s ; *s2 && *s2 != '/' ; s2--)
;
if (s-s2 < 2)
strcpy (out,"?model?");
else
{
s--;
strncpy (out,s2+1, s-s2);
out[s-s2] = 0;
}
}

4. In common.h, add this anywhere really like at the end of the file

Quote:
void COM_GetFolder (char *in, char *out);//R00k

5. In cl_parse.c
Add the yellow.

Quote:
/* ===================== CL_WebDownloadProgress Callback function for webdownloads. Since Web_Get only returns once it's done, we have to do various things here:
Update download percent, handle input, redraw UI and send net packets.
=====================
*/
static int CL_WebDownloadProgress( double percent )
{
static double time, oldtime, newtime;
cls.download.percent = percent;
CL_KeepaliveMessage();
newtime = Sys_DoubleTime ();
time = newtime - oldtime;
Host_Frame (time);
oldtime = newtime;
return cls.download.disconnect; // abort if disconnect received
}
/*
==================
CL_ParseServerInfo
==================
*/
void CL_ParseServerInfo (void)
{
char *str, tempname[MAX_QPATH];
int i, nummodels, numsounds;
char model_precache[MAX_MODELS][MAX_QPATH];
char sound_precache[MAX_SOUNDS][MAX_QPATH];
extern cvar_t cl_web_download;
extern cvar_t cl_web_download_url;
extern int Web_Get( const char *url, const char *referer, const char *name, int resume, int max_downloading_time, int timeout, int ( *_progress )(double) );

And further download in cl_parse.c make alike!

Quote:
// now we try to load everything else until a cache allocation fails for (i=1 ; i<nummodels ; i++) { cl.model_precache[i] = Mod_ForName (model_precache[i], false); if (cl.model_precache[i] == NULL)
{
if (cl_web_download.value && cl_web_download_url.string)
{
char url[1024];
qboolean success = false;
char download_tempname[MAX_QPATH],download_finalname[MAX_QPATH];
char folder[MAX_QPATH];
char name[MAX_QPATH];
extern char server_name[MAX_QPATH];
extern int net_hostport;
//Create the FULL path where the file should be written
Q_snprintfz (download_tempname, MAX_OSPATH, "%s/%s.tmp", com_gamedir, model_precache[i]);
//determine the proper folder and create it, the OS will ignore if already exsists
COM_GetFolder(model_precache[i],folder);// "progs/","maps/"
Q_snprintfz (name, sizeof(name), "%s/%s", com_gamedir, folder);
Sys_mkdir (name);
Con_Printf( "Web downloading: %s from %s%s\n", model_precache[i], cl_web_download_url.string, model_precache[i]);
//assign the url + path + file + extension we want
Q_snprintfz( url, sizeof( url ), "%s%s", cl_web_download_url.string, model_precache[i]);
cls.download.web = true;
cls.download.disconnect = false;
cls.download.percent = 0.0;
//let libCURL do it's magic!!
success = Web_Get(url, NULL, download_tempname, false, 600, 30, CL_WebDownloadProgress);
cls.download.web = false;
free(url);
free(name);
free(folder);
if (success)
{
Con_Printf("Web download succesfull: %s\n", download_tempname);
//Rename the .tmp file to the final precache filename
Q_snprintfz (download_finalname, MAX_OSPATH, "%s/%s", com_gamedir, model_precache[i]);
rename (download_tempname, download_finalname);
free(download_tempname);
free(download_finalname);
Cbuf_AddText (va("connect %s:%u\n",server_name,net_hostport));//reconnect after each success
return;
}
else
{
remove (download_tempname);
Con_Printf( "Web download of %s failed\n", download_tempname );
return;
}
free(download_tempname);
if( cls.download.disconnect )//if the user type disconnect in the middle of the download
{
cls.download.disconnect = false;
CL_Disconnect_f();
return;
}
} else
#endif
{
Con_Printf("Model %s not found\n", model_precache[i]);
return;
}
}
CL_KeepaliveMessage ();
}
S_BeginPrecaching ();
for (i=1 ; i<numsounds ; i++)
{
cl.sound_precache[i] = S_PrecacheSound (sound_precache[i]);
CL_KeepaliveMessage ();
}
S_EndPrecaching ();

6. Now in cl_main.c

Quote:
*/ // cl_main.c -- client main loop #include "quakedef.h" #include "curl.h"

Quote:
cvar_t cl_web_download = {"cl_web_download", "1", true}; cvar_t cl_web_download_url = {"cl_web_download_url", "http://www.mywebpage.com/maps/", true}; client_static_t cls; client_state_t cl;

And then ...

Quote:
void CL_Disconnect (void) { // stop sounds (especially looping!) S_StopAllSounds (true); // bring the console down and fade the colors back to normal
// SCR_BringDownConsole ();
// We have to shut down webdownloading first
if( cls.download.web )
{
cls.download.disconnect = true;
return;
}
// if running a local server, shut it down
if (cls.demoplayback)
{
CL_StopPlayback ();
}

And then near end of cl_main.c

Quote:
Cmd_AddCommand ("timedemo", CL_TimeDemo_f); Cvar_RegisterVariable (&cl_web_download, NULL); Cvar_RegisterVariable (&cl_web_download_url, NULL);

7. In your project settings, include libcurl.lib to be compiled into the build. In MSVC6, you do this by clicking Project -> Settings and then I select GL Release with ProQuake 3.99 and click the Link Tab and in "Object/Library Modules" I added libcurl.lib to the end.
You are done.
Here is a before and after of the code more or less so someone with WinMerge or whatever can manually examine the differences.
Source code: Before (download) and after (download).
_________________
Tomorrow Never Dies. I feel this Tomorrow knocking on the door ...

Back to top |

Team Xlink Joined: 25 Jun 2009 Posts: 320 |

Posted: Thu Aug 13, 2009 2:05 am Post subject: |

Very nice Tutorial Baker! This is very nicely written! _________________

Anonymous wrote:
if it works, it works. if it doesn't, HAHAHA!

Back to top |

Downsider Joined: 16 Sep 2008 Posts: 478 |

Posted: Thu Aug 13, 2009 3:33 am Post subject: |

It's cool for people who like to come in and copypaste into their engine, but not understand anything that's going on.

Back to top |

Team Xlink Joined: 25 Jun 2009 Posts: 320 |

Posted: Sun Aug 23, 2009 9:12 pm Post subject: |

Downsider wrote:
It's cool for people who like to come in and copypaste into their engine, but not understand anything that's going on.

Well parts of it are self explanatory so it isn't that big of a deal.
If people want to understand it I think they can and will.
_________________

Anonymous wrote:
if it works, it works. if it doesn't, HAHAHA!

Back to top |

Baker Joined: 14 Mar 2006 Posts: 1538 |

Posted: Wed Jan 06, 2010 5:51 pm Post subject: |

One weakness of this tutorial is apparently a missing model during demo playback and this will cause the client to want to download it. Needs a slight adjustment!

Back to top |

Teiman Joined: 03 Jun 2007 Posts: 309 |

Posted: Fri Jan 08, 2010 4:17 pm Post subject: |

Baker wrote:
One weakness of this tutorial is apparently a missing model during demo playback and this will cause the client to want to download it. Needs a slight adjustment!

downloading could be deactivated while playing a demo.
playing a demo could be detected.

Back to top |

mh Joined: 12 Jan 2008 Posts: 909 |

Posted: Tue Mar 23, 2010 5:20 pm Post subject: |

...and here's a native Windows API version that doesn't need the libcurl DLLs in your Quake folder. This just replaces all of the libcurl stuff, and I've kept the interface the very same, so in theory it's just drop and go. You should be familiar with the above code before even attempting to implement this, as I'm not otherwise documenting in detail what needs to be replaced. In brief however you don't need all of the extra .c files, .h files or .lib files, so you can remove them from your project. As mentioned above, if you're distributing a Windows version you can also remove the libcurl.dll and zlib1.dll files from your distribution, as you won't need those any more either.
Everything else in Baker's original tutorial should stay the same.
I should also mention that you don't need to add any .lib files to your project either, as I'm dynamically linking to DLLs in C:\Windows\System32 (or wherever). (Aside from perhaps winmm.lib, but I think Quake already links with that for it's CD audio.)
Note: the guts of it are in C++, but I've created a C interface so that it can be called from C programs, and also provide a sample implementation (in C).
So here's your cl_webdownload.cpp - this replaces all of the libcurl stuff:

Code:
#include <windows.h> #include <wininet.h> #include <urlmon.h> typedef int (*DOWNLOADPROGRESSPROC) (double); class CDownloader : public IBindStatusCallback
{
public:
CDownloader (DOWNLOADPROGRESSPROC progressproc, DWORD maxtime, DWORD timeout)
{
// progress updater (optional)
this->DownloadProgressProc = progressproc;
// store times in milliseconds for convenience
this->DownloadMaxTime = maxtime * 1000;
this->DownloadTimeOut = timeout * 1000;
// init to sensible values
this->DownloadBeginTime = timeGetTime ();
this->DownloadCurrentTime = timeGetTime ();
this->DownloadLastCurrentTime = timeGetTime ();
}
~CDownloader (void) {}
// progress
STDMETHOD (OnProgress) (ULONG ulProgress, ULONG ulProgressMax, ULONG ulStatusCode, LPCWSTR szStatusText)
{
switch (ulStatusCode)
{
case BINDSTATUS_BEGINDOWNLOADDATA:
// init timers
this->DownloadBeginTime = timeGetTime ();
this->DownloadCurrentTime = timeGetTime ();
this->DownloadLastCurrentTime = timeGetTime ();
return S_OK;
case BINDSTATUS_ENDDOWNLOADDATA:
// amount downloaded was not equal to the amount needed
if (ulProgress != ulProgressMax) return E_ABORT;
// alles gut
return S_OK;
case BINDSTATUS_DOWNLOADINGDATA:
// update current time
this->DownloadCurrentTime = timeGetTime ();
// abort if the max download time is exceeded
if (this->DownloadCurrentTime - this->DownloadBeginTime > this->DownloadMaxTime) return E_ABORT;
// check for a hang and abort if required
if (this->DownloadCurrentTime - this->DownloadLastCurrentTime > this->DownloadTimeOut) return E_ABORT;
// update hang check
this->DownloadLastCurrentTime = this->DownloadCurrentTime;
// send through the standard proc
if (ulProgressMax && this->DownloadProgressProc)
return (this->DownloadProgressProc ((int) ((((double) ulProgress / (double) ulProgressMax) * 100) + 0.5)) ? S_OK : E_ABORT);
else return S_OK;
default:
break;
}
return S_OK;
}
// unimplemented methods
STDMETHOD (GetBindInfo) (DWORD *grfBINDF, BINDINFO *pbindinfo) {return E_NOTIMPL;}
STDMETHOD (GetPriority) (LONG *pnPriority) {return E_NOTIMPL;}
STDMETHOD (OnDataAvailable) (DWORD grfBSCF, DWORD dwSize, FORMATETC *pformatetc, STGMEDIUM *pstgmed ) {return E_NOTIMPL;}
STDMETHOD (OnObjectAvailable) (REFIID riid, IUnknown *punk) {return E_NOTIMPL;}
STDMETHOD (OnStartBinding) (DWORD dwReserved, IBinding *pib) {return E_NOTIMPL;}
STDMETHOD (OnStopBinding) (HRESULT hresult, LPCWSTR szError){return E_NOTIMPL;}
STDMETHOD (OnLowResource) (DWORD blah) {return E_NOTIMPL;}
// IUnknown methods - URLDownloadToFile never calls these
STDMETHOD_ (ULONG, AddRef) () {return 0;}
STDMETHOD_ (ULONG, Release) () {return 0;}
STDMETHOD (QueryInterface) (REFIID riid, void __RPC_FAR *__RPC_FAR *ppvObject) {return E_NOTIMPL;}
private:
DOWNLOADPROGRESSPROC DownloadProgressProc;
DWORD DownloadBeginTime;
DWORD DownloadCurrentTime;
DWORD DownloadLastCurrentTime;
DWORD DownloadMaxTime;
DWORD DownloadTimeOut;
};
typedef HRESULT (__stdcall *URLDOWNLOADTOFILEPROC) (LPUNKNOWN, LPCSTR, LPCSTR, DWORD, LPBINDSTATUSCALLBACK);
typedef BOOL (__stdcall *DELETEURLCACHEENTRYPROC) (__in LPCSTR);
extern "C" int Web_Get (const char *url, const char *referer, const char *name, int resume, int max_downloading_time, int timeout, int (*_progress) (double))
{
// assume it's failed until we know it's succeeded
int DownloadResult = 0;
// this is to deal with kiddies who still think it's cool to remove all IE components from their computers,
// and also to resolve potential issues when we might be linking to a different version than what's on the
// target machine. in theory we could just use standard URLDownloadToFile and DeleteUrlCacheEntry though.
HINSTANCE hInstURLMON = LoadLibrary ("urlmon.dll");
HINSTANCE hInstWININET = LoadLibrary ("wininet.dll");
if (hInstURLMON && hInstWININET)
{
// grab the non-unicode versions of the functions
URLDOWNLOADTOFILEPROC QURLDownloadToFile = (URLDOWNLOADTOFILEPROC) GetProcAddress (hInstURLMON, "URLDownloadToFileA");
DELETEURLCACHEENTRYPROC QDeleteUrlCacheEntry = (DELETEURLCACHEENTRYPROC) GetProcAddress (hInstWININET, "DeleteUrlCacheEntryA");
if (QURLDownloadToFile && QDeleteUrlCacheEntry)
{
// always take a fresh copy (fail silently if this fails)
QDeleteUrlCacheEntry (url);
// create the COM interface used for downloading
CDownloader *DownloadClass = new CDownloader (_progress, max_downloading_time, timeout);
// and download it
HRESULT hrDownload = QURLDownloadToFile (NULL, url, name, 0, DownloadClass);
// clean up
delete DownloadClass;
// check result
if (FAILED (hrDownload))
DownloadResult = 0;
else DownloadResult = 1;
}
else DownloadResult = 0;
}
else DownloadResult = 0;
if (hInstURLMON) FreeLibrary (hInstURLMON);
if (hInstWININET) FreeLibrary (hInstWININET);
return DownloadResult;
}

The only missing functionality from this is that it doesn't support resuming a download.
I wasn't entirely certain if the download callback provided by Baker uses a 0 to 1 or a 0 to 100 scale for it's percent complete; my code scales 0 to 100 but you can change that if it bothers you (it's in the "return this->DownloadProgressProc" line.)
And here's the sample implementation in C:

Code:
#include <windows.h> #include <stdio.h> #include <conio.h> #pragma comment (lib, "winmm.lib") int Web_Get (const char *url, const char *referer, const char *name, int resume, int max_downloading_time, int timeout, int (*_progress) (double));
char *TargetURL = "http://download.microsoft.com/download/win2000platform/sp/sp2/nt5/en-us/w2ksp2.exe";
char *TargetFile = "C:\\Win2KSP2.exe";
int CL_WebDownloadProgress (double percent)
{
static int oldpct = -1;
if ((int) percent != oldpct)
{
printf ("...Downloading %i%%\n", (int) percent);
oldpct = (int) percent;
}
return 1;
}
void main (void)
{
timeBeginPeriod (1);
printf ("Downloading %s\nto %s\n", TargetURL, TargetFile);
if (Web_Get (TargetURL, NULL, TargetFile, 0, 600, 30, CL_WebDownloadProgress))
printf ("Download succeeded\n");
else printf ("Download failed\n");
printf ("Press any key... ");
while (1)
{
if (_kbhit ()) break;
Sleep (5);
}
}

For testing purposes I've just set it to download Windows 2000 SP2 - I'm on a gigabit internet connection here so I need something large enough to make download times and notification meaningful!
If you want you can compile the two of these into a console application and run the sample implementation to confirm that it works before you go hacking at your engine - I probably recommend doing that.
_________________
DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines
Last edited by mh on Wed Mar 24, 2010 5:13 pm; edited 1 time in total

Back to top |

Sajt Joined: 16 Oct 2004 Posts: 1026 |

Posted: Tue Mar 23, 2010 7:13 pm Post subject: |

Shouldn't you delete DownloadClass afterwards? Anyway, it's cool to have code like this lying around as reference. Sometimes it is nice to have an engine that can operate as a solitary exe instead of having to come with all sorts of dlls (on Windows).
_________________
F. A. Špork, an enlightened nobleman and a great patron of art, had a stately Baroque spa complex built on the banks of the River Labe.

Back to top |

mh Joined: 12 Jan 2008 Posts: 909 |

Posted: Tue Mar 23, 2010 7:43 pm Post subject: |

Well spotted, you win the prize! I also seem to have mixed up some return types while transitioning my original test code to a compatible interface. I'll clean these up tomorrow. Update: done. _________________ DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines

Back to top |

Display posts from previous:

| Inside3d Forums Forum Index -> Programming Tutorials | All times are GMT
Page 1 of 1

Jump to:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

Powered by phpBB © 2004 phpBB Group