If you get a video mode, which your monitor can't display upon the first start of Quake, you should do the following:
Open "TextEdit" which comes with the MacOS X installation.
Locate the file "config.cfg" inside your "id1" folder and open it with "TextEdit".
Edit the line "_vid_default_mode x" within the "id1/config.cfg" file and change the value of "x" until you get a working display mode. For example: "_vid_default_mode 0". Mode "0", "1" and "2" are windowed modes.