Inside3D!
     

Warsow/Qrack Web Map/Model Download (Curl)

 
Post new topic   Reply to topic    Inside3d Forums Forum Index -> Programming Tutorials
View previous topic :: View next topic  
Author Message
Baker



Joined: 14 Mar 2006
Posts: 1538

PostPosted: Tue Aug 11, 2009 12:49 am    Post subject: Warsow/Qrack Web Map/Model Download (Curl) Reply with quote

Rook added some outstanding map/model download to Qrack which I have now added to ProQuake 3.99. The Qrack code was heavily derived from Warsow.

I am going to describe the very few changes I had to make to implement this in the Windows version of ProQuake 3.99, which should be nearly the same as doing so with GLQuake or for that matter WinQuake.

Notes wrote:
Off hand I don't see anything operating system specific here either but haven't tried to compile a Mac OS X version. Curl is known to be multi-platform. When I do the OS X version, I'll write up whatever.


End Result

The end result is that connecting to a server, like ThreeWave CTF downloads the maps and model -- but not the sounds -- and does so very rapidly (seems instant even for 5 MB maps on my broadband) and reconnects.

Yes, DarkPlaces does it better but this is still very good but can definitely be improved upon.

But the nicest thing is how simple the modification is.

We begin ...

Requirements for this Tutorial

1. Windows operating system + MSVC6
2. Libcurl.lib (I have no clue which specific version Rook used off the CURL download page).
3. This version of the ProQuake 3.99 source but this should likely work with stock glquake source with Microsoft Visual C++ Express Edition but you'll be on your own but it shouldn't be too hard.

Questions and Miscellaneous Notes

1. The libcurl.lib gets compiled into the binary and the size of the binary is only increased negligibly (a few KB!). Excellent!
2. Does the binary work properly on old Windows 98 or ME? What about Windows 7? Or even Vista? I haven't tried any of those yet with the binary. I'm not going to make the assumption that, say, Windows 98 works with this because I just don't know.
3. The download works super-efficiently and is extremely fast. Far faster than really what you'd expect!

Tutorial

1. Add the following files to your project

You can borrow them from this (download) and be sure to take the libcurl.lib as well.

Quote:
Files to add:

curl.h
curlver.h
easy.h
multi.h
webdownload.c
webdownload.h


2. client.h

Add the yellow ....

Quote:
typedef enum {
ca_dedicated, // a dedicated server with no ability to start a client
ca_disconnected, // full screen console with no connection
ca_connected // valid netcon, talking to a server
} cactive_t;

typedef struct
{
qboolean web;
char *name;
double percent;
qboolean disconnect; // set when user tries to disconnect, to allow cleaning up webdownload
} download_t;


And still in client.h, add the yellow ...

Quote:
// connection information
int signon; // 0 to SIGNONS
struct qsocket_s *netcon;
sizebuf_t message; // writing buffer to send to server
download_t download;


3. Add COM_GetFolder to common.c

Add the yellow.

Quote:
/*
============
COM_GetFolder
============
*/
void COM_GetFolder (char *in, char *out)
{
char *last = NULL;

while (*in)
{
if (*in == '/')
last = out;
*out++ = *in++;
}
if (last)
*last = 0;
else
*out = 0;
}



/*
============
COM_FileBase
============
*/
void COM_FileBase (char *in, char *out)
{
char *s, *s2;

s = in + strlen(in) - 1;

while (s != in && *s != '.')
s--;

for (s2 = s ; *s2 && *s2 != '/' ; s2--)
;

if (s-s2 < 2)
strcpy (out,"?model?");
else
{
s--;
strncpy (out,s2+1, s-s2);
out[s-s2] = 0;
}
}


4. In common.h, add this anywhere really like at the end of the file

Quote:
void COM_GetFolder (char *in, char *out);//R00k


5. In cl_parse.c

Add the yellow.

Quote:

/*
=====================
CL_WebDownloadProgress
Callback function for webdownloads.
Since Web_Get only returns once it's done, we have to do various things here:
Update download percent, handle input, redraw UI and send net packets.
=====================
*/
static int CL_WebDownloadProgress( double percent )
{
static double time, oldtime, newtime;

cls.download.percent = percent;
CL_KeepaliveMessage();

newtime = Sys_DoubleTime ();
time = newtime - oldtime;

Host_Frame (time);

oldtime = newtime;

return cls.download.disconnect; // abort if disconnect received
}



/*
==================
CL_ParseServerInfo
==================
*/
void CL_ParseServerInfo (void)
{
char *str, tempname[MAX_QPATH];
int i, nummodels, numsounds;
char model_precache[MAX_MODELS][MAX_QPATH];
char sound_precache[MAX_SOUNDS][MAX_QPATH];

extern cvar_t cl_web_download;
extern cvar_t cl_web_download_url;
extern int Web_Get( const char *url, const char *referer, const char *name, int resume, int max_downloading_time, int timeout, int ( *_progress )(double) );



And further download in cl_parse.c make alike!

Quote:
// now we try to load everything else until a cache allocation fails
for (i=1 ; i<nummodels ; i++)
{
cl.model_precache[i] = Mod_ForName (model_precache[i], false);
if (cl.model_precache[i] == NULL)
{

if (cl_web_download.value && cl_web_download_url.string)
{
char url[1024];
qboolean success = false;
char download_tempname[MAX_QPATH],download_finalname[MAX_QPATH];
char folder[MAX_QPATH];
char name[MAX_QPATH];
extern char server_name[MAX_QPATH];
extern int net_hostport;

//Create the FULL path where the file should be written
Q_snprintfz (download_tempname, MAX_OSPATH, "%s/%s.tmp", com_gamedir, model_precache[i]);

//determine the proper folder and create it, the OS will ignore if already exsists
COM_GetFolder(model_precache[i],folder);// "progs/","maps/"
Q_snprintfz (name, sizeof(name), "%s/%s", com_gamedir, folder);
Sys_mkdir (name);

Con_Printf( "Web downloading: %s from %s%s\n", model_precache[i], cl_web_download_url.string, model_precache[i]);

//assign the url + path + file + extension we want
Q_snprintfz( url, sizeof( url ), "%s%s", cl_web_download_url.string, model_precache[i]);

cls.download.web = true;
cls.download.disconnect = false;
cls.download.percent = 0.0;

//let libCURL do it's magic!!
success = Web_Get(url, NULL, download_tempname, false, 600, 30, CL_WebDownloadProgress);

cls.download.web = false;

free(url);
free(name);
free(folder);

if (success)
{
Con_Printf("Web download succesfull: %s\n", download_tempname);
//Rename the .tmp file to the final precache filename
Q_snprintfz (download_finalname, MAX_OSPATH, "%s/%s", com_gamedir, model_precache[i]);
rename (download_tempname, download_finalname);

free(download_tempname);
free(download_finalname);

Cbuf_AddText (va("connect %s:%u\n",server_name,net_hostport));//reconnect after each success
return;
}
else
{
remove (download_tempname);
Con_Printf( "Web download of %s failed\n", download_tempname );
return;
}

free(download_tempname);

if( cls.download.disconnect )//if the user type disconnect in the middle of the download
{
cls.download.disconnect = false;
CL_Disconnect_f();
return;
}
} else
#endif
{
Con_Printf("Model %s not found\n", model_precache[i]);
return;
}
}
CL_KeepaliveMessage ();
}


S_BeginPrecaching ();
for (i=1 ; i<numsounds ; i++)
{
cl.sound_precache[i] = S_PrecacheSound (sound_precache[i]);
CL_KeepaliveMessage ();
}
S_EndPrecaching ();


6. Now in cl_main.c

Quote:
*/
// cl_main.c -- client main loop

#include "quakedef.h"
#include "curl.h"


Quote:

cvar_t cl_web_download = {"cl_web_download", "1", true};
cvar_t cl_web_download_url = {"cl_web_download_url", "http://www.mywebpage.com/maps/", true};


client_static_t cls;
client_state_t cl;


And then ...

Quote:
void CL_Disconnect (void)
{
// stop sounds (especially looping!)
S_StopAllSounds (true);

// bring the console down and fade the colors back to normal
// SCR_BringDownConsole ();


// We have to shut down webdownloading first
if( cls.download.web )
{
cls.download.disconnect = true;
return;
}



// if running a local server, shut it down
if (cls.demoplayback)
{
CL_StopPlayback ();
}


And then near end of cl_main.c

Quote:
Cmd_AddCommand ("timedemo", CL_TimeDemo_f);

Cvar_RegisterVariable (&cl_web_download, NULL);
Cvar_RegisterVariable (&cl_web_download_url, NULL);



7. In your project settings, include libcurl.lib to be compiled into the build. In MSVC6, you do this by clicking Project -> Settings and then I select GL Release with ProQuake 3.99 and click the Link Tab and in "Object/Library Modules" I added libcurl.lib to the end.

You are done.

Here is a before and after of the code more or less so someone with WinMerge or whatever can manually examine the differences.

Source code: Before (download) and after (download).
_________________
Tomorrow Never Dies. I feel this Tomorrow knocking on the door ...
Back to top
View user's profile Send private message
Team Xlink



Joined: 25 Jun 2009
Posts: 320

PostPosted: Thu Aug 13, 2009 2:05 am    Post subject: Reply with quote

Very nice Tutorial Baker!


This is very nicely written!
_________________
Anonymous wrote:
if it works, it works. if it doesn't, HAHAHA!
Back to top
View user's profile Send private message
Downsider



Joined: 16 Sep 2008
Posts: 478

PostPosted: Thu Aug 13, 2009 3:33 am    Post subject: Reply with quote

It's cool for people who like to come in and copypaste into their engine, but not understand anything that's going on. Crying or Very sad
Back to top
View user's profile Send private message
Team Xlink



Joined: 25 Jun 2009
Posts: 320

PostPosted: Sun Aug 23, 2009 9:12 pm    Post subject: Reply with quote

Downsider wrote:
It's cool for people who like to come in and copypaste into their engine, but not understand anything that's going on. Crying or Very sad



Well parts of it are self explanatory so it isn't that big of a deal.
If people want to understand it I think they can and will.
_________________
Anonymous wrote:
if it works, it works. if it doesn't, HAHAHA!
Back to top
View user's profile Send private message
Baker



Joined: 14 Mar 2006
Posts: 1538

PostPosted: Wed Jan 06, 2010 5:51 pm    Post subject: Reply with quote

One weakness of this tutorial is apparently a missing model during demo playback and this will cause the client to want to download it.

Needs a slight adjustment!
Back to top
View user's profile Send private message
Teiman



Joined: 03 Jun 2007
Posts: 309

PostPosted: Fri Jan 08, 2010 4:17 pm    Post subject: Reply with quote

Baker wrote:
One weakness of this tutorial is apparently a missing model during demo playback and this will cause the client to want to download it.

Needs a slight adjustment!


downloading could be deactivated while playing a demo.
playing a demo could be detected.
Back to top
View user's profile Send private message
mh



Joined: 12 Jan 2008
Posts: 909

PostPosted: Tue Mar 23, 2010 5:20 pm    Post subject: Reply with quote

...and here's a native Windows API version that doesn't need the libcurl DLLs in your Quake folder. This just replaces all of the libcurl stuff, and I've kept the interface the very same, so in theory it's just drop and go.

You should be familiar with the above code before even attempting to implement this, as I'm not otherwise documenting in detail what needs to be replaced. In brief however you don't need all of the extra .c files, .h files or .lib files, so you can remove them from your project. As mentioned above, if you're distributing a Windows version you can also remove the libcurl.dll and zlib1.dll files from your distribution, as you won't need those any more either. Very Happy

Everything else in Baker's original tutorial should stay the same.

I should also mention that you don't need to add any .lib files to your project either, as I'm dynamically linking to DLLs in C:\Windows\System32 (or wherever). (Aside from perhaps winmm.lib, but I think Quake already links with that for it's CD audio.)

Note: the guts of it are in C++, but I've created a C interface so that it can be called from C programs, and also provide a sample implementation (in C).

So here's your cl_webdownload.cpp - this replaces all of the libcurl stuff:
Code:
#include <windows.h>
#include <wininet.h>
#include <urlmon.h>


typedef int (*DOWNLOADPROGRESSPROC) (double);

class CDownloader : public IBindStatusCallback
{
public:
   CDownloader (DOWNLOADPROGRESSPROC progressproc, DWORD maxtime, DWORD timeout)
   {
      // progress updater (optional)
      this->DownloadProgressProc = progressproc;

      // store times in milliseconds for convenience
      this->DownloadMaxTime = maxtime * 1000;
      this->DownloadTimeOut = timeout * 1000;

      // init to sensible values
      this->DownloadBeginTime = timeGetTime ();
      this->DownloadCurrentTime = timeGetTime ();
      this->DownloadLastCurrentTime = timeGetTime ();
   }

   ~CDownloader (void) {}

   // progress
   STDMETHOD (OnProgress) (ULONG ulProgress, ULONG ulProgressMax, ULONG ulStatusCode, LPCWSTR szStatusText)
   {
      switch (ulStatusCode)
      {
      case BINDSTATUS_BEGINDOWNLOADDATA:
         // init timers
         this->DownloadBeginTime = timeGetTime ();
         this->DownloadCurrentTime = timeGetTime ();
         this->DownloadLastCurrentTime = timeGetTime ();
         return S_OK;

      case BINDSTATUS_ENDDOWNLOADDATA:
         // amount downloaded was not equal to the amount needed
         if (ulProgress != ulProgressMax) return E_ABORT;

         // alles gut
         return S_OK;

      case BINDSTATUS_DOWNLOADINGDATA:
         // update current time
         this->DownloadCurrentTime = timeGetTime ();

         // abort if the max download time is exceeded
         if (this->DownloadCurrentTime - this->DownloadBeginTime > this->DownloadMaxTime) return E_ABORT;

         // check for a hang and abort if required
         if (this->DownloadCurrentTime - this->DownloadLastCurrentTime > this->DownloadTimeOut) return E_ABORT;

         // update hang check
         this->DownloadLastCurrentTime = this->DownloadCurrentTime;

         // send through the standard proc
         if (ulProgressMax && this->DownloadProgressProc)
            return (this->DownloadProgressProc ((int) ((((double) ulProgress / (double) ulProgressMax) * 100) + 0.5)) ? S_OK : E_ABORT);
         else return S_OK;

      default:
         break;
      }

      return S_OK;
   }

   // unimplemented methods
   STDMETHOD (GetBindInfo) (DWORD *grfBINDF, BINDINFO *pbindinfo) {return E_NOTIMPL;}
   STDMETHOD (GetPriority) (LONG *pnPriority) {return E_NOTIMPL;}
   STDMETHOD (OnDataAvailable) (DWORD grfBSCF, DWORD dwSize, FORMATETC *pformatetc, STGMEDIUM *pstgmed ) {return E_NOTIMPL;}
   STDMETHOD (OnObjectAvailable) (REFIID riid, IUnknown *punk) {return E_NOTIMPL;}
   STDMETHOD (OnStartBinding) (DWORD dwReserved, IBinding *pib) {return E_NOTIMPL;}
   STDMETHOD (OnStopBinding) (HRESULT hresult, LPCWSTR szError){return E_NOTIMPL;}
   STDMETHOD (OnLowResource) (DWORD blah) {return E_NOTIMPL;}

   // IUnknown methods - URLDownloadToFile never calls these
   STDMETHOD_ (ULONG, AddRef) () {return 0;}
   STDMETHOD_ (ULONG, Release) () {return 0;}
   STDMETHOD (QueryInterface) (REFIID riid, void __RPC_FAR *__RPC_FAR *ppvObject) {return E_NOTIMPL;}

private:
   DOWNLOADPROGRESSPROC DownloadProgressProc;
   DWORD DownloadBeginTime;
   DWORD DownloadCurrentTime;
   DWORD DownloadLastCurrentTime;
   DWORD DownloadMaxTime;
   DWORD DownloadTimeOut;
};


typedef HRESULT (__stdcall *URLDOWNLOADTOFILEPROC) (LPUNKNOWN, LPCSTR, LPCSTR, DWORD, LPBINDSTATUSCALLBACK);
typedef BOOL (__stdcall *DELETEURLCACHEENTRYPROC) (__in LPCSTR);

extern "C" int Web_Get (const char *url, const char *referer, const char *name, int resume, int max_downloading_time, int timeout, int (*_progress) (double))
{
   // assume it's failed until we know it's succeeded
   int DownloadResult = 0;

   // this is to deal with kiddies who still think it's cool to remove all IE components from their computers,
   // and also to resolve potential issues when we might be linking to a different version than what's on the
   // target machine.  in theory we could just use standard URLDownloadToFile and DeleteUrlCacheEntry though.
   HINSTANCE hInstURLMON = LoadLibrary ("urlmon.dll");
   HINSTANCE hInstWININET = LoadLibrary ("wininet.dll");

   if (hInstURLMON && hInstWININET)
   {
      // grab the non-unicode versions of the functions
      URLDOWNLOADTOFILEPROC QURLDownloadToFile = (URLDOWNLOADTOFILEPROC) GetProcAddress (hInstURLMON, "URLDownloadToFileA");
      DELETEURLCACHEENTRYPROC QDeleteUrlCacheEntry = (DELETEURLCACHEENTRYPROC) GetProcAddress (hInstWININET, "DeleteUrlCacheEntryA");

      if (QURLDownloadToFile && QDeleteUrlCacheEntry)
      {
         // always take a fresh copy (fail silently if this fails)
         QDeleteUrlCacheEntry (url);

         // create the COM interface used for downloading
         CDownloader *DownloadClass = new CDownloader (_progress, max_downloading_time, timeout);

         // and download it
         HRESULT hrDownload = QURLDownloadToFile (NULL, url, name, 0, DownloadClass);

         // clean up
         delete DownloadClass;

         // check result
         if (FAILED (hrDownload))
            DownloadResult = 0;
         else DownloadResult = 1;
      }
      else DownloadResult = 0;
   }
   else DownloadResult = 0;

   if (hInstURLMON) FreeLibrary (hInstURLMON);
   if (hInstWININET) FreeLibrary (hInstWININET);

   return DownloadResult;
}


The only missing functionality from this is that it doesn't support resuming a download.

I wasn't entirely certain if the download callback provided by Baker uses a 0 to 1 or a 0 to 100 scale for it's percent complete; my code scales 0 to 100 but you can change that if it bothers you (it's in the "return this->DownloadProgressProc" line.)

And here's the sample implementation in C:
Code:
#include <windows.h>
#include <stdio.h>
#include <conio.h>

#pragma comment (lib, "winmm.lib")

int Web_Get (const char *url, const char *referer, const char *name, int resume, int max_downloading_time, int timeout, int (*_progress) (double));
char *TargetURL = "http://download.microsoft.com/download/win2000platform/sp/sp2/nt5/en-us/w2ksp2.exe";
char *TargetFile = "C:\\Win2KSP2.exe";

int CL_WebDownloadProgress (double percent)
{
   static int oldpct = -1;

   if ((int) percent != oldpct)
   {
      printf ("...Downloading %i%%\n", (int) percent);
      oldpct = (int) percent;
   }

   return 1;
}


void main (void)
{
   timeBeginPeriod (1);

   printf ("Downloading %s\nto %s\n", TargetURL, TargetFile);

   if (Web_Get (TargetURL, NULL, TargetFile, 0, 600, 30, CL_WebDownloadProgress))
      printf ("Download succeeded\n");
   else printf ("Download failed\n");

   printf ("Press any key... ");

   while (1)
   {
      if (_kbhit ()) break;
      Sleep (5);
   }
}


For testing purposes I've just set it to download Windows 2000 SP2 - I'm on a gigabit internet connection here so I need something large enough to make download times and notification meaningful!

If you want you can compile the two of these into a console application and run the sample implementation to confirm that it works before you go hacking at your engine - I probably recommend doing that. Wink
_________________
DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines


Last edited by mh on Wed Mar 24, 2010 5:13 pm; edited 1 time in total
Back to top
View user's profile Send private message Visit poster's website
Sajt



Joined: 16 Oct 2004
Posts: 1026

PostPosted: Tue Mar 23, 2010 7:13 pm    Post subject: Reply with quote

Shouldn't you delete DownloadClass afterwards?

Anyway, it's cool to have code like this lying around as reference. Sometimes it is nice to have an engine that can operate as a solitary exe instead of having to come with all sorts of dlls (on Windows).
_________________
F. A. Špork, an enlightened nobleman and a great patron of art, had a stately Baroque spa complex built on the banks of the River Labe.
Back to top
View user's profile Send private message
mh



Joined: 12 Jan 2008
Posts: 909

PostPosted: Tue Mar 23, 2010 7:43 pm    Post subject: Reply with quote

Well spotted, you win the prize! Laughing

I also seem to have mixed up some return types while transitioning my original test code to a compatible interface. I'll clean these up tomorrow. Embarassed

Update: done.
_________________
DirectQ Engine - New release 1.8.666a, 9th August 2010
MHQuake Blog (General)
Direct3D 8 Quake Engines
Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:   
Post new topic   Reply to topic    Inside3d Forums Forum Index -> Programming Tutorials All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2004 phpBB Group