You are not logged in.

Post a reply

Write your message and submit
Are you human or robot? If you have trouble, mail to spirit åt quaddicted døt c

Checking if this is requested by a real person and not an automated program.

Go back

Topic review (newest first)

Spirit
2019-12-23 20:44:53

We have a dedicated channel on dumptruck's discord now, you are welcome to join in there.

Here is what I posted as mission:

So here is the gist:
This is about https://www.quaddicted.com/reviews/ and the database behind it

The problem(s)
- The site is in a state where updates are requiring too much work and the code is terrible
- Uploading files relies on me and only few people can then add them to the database
- Because people release different versions of files with the same filename, the whole system of using the filename as idenficator is not working
- Because people like to release maps not as zip but also as plain bsp files or rar or 7z, the limitation to zip files does not work
- Because files are often packaged weirdly and file paths inside of releases can clash with others (think "readme.txt") automated installations with the Quake Injector often fail or require manual repackaging of files (which goes against the whole idea of an archive). All we have in the database at the moment is an instruction where to extract a zip file which does not make a conflict-free and working installation possible in some cases


The idea(s)
- Instead of having the database editor interface available to just a few people (:heart:negke:heart:), we will have a public Git repository filled with one JSON file per released file
- This JSON file will describe the file itself and have a dictionary of tags describing the contents
- This includes neutral information like author names, the title, tags about the size or texture set, the releasedate etc.
- This does not include the description (maybe it will? let's discuss this), the editor's rating, the user ratings or comments, those will stay in the Quaddicted database
- Things like dependencies, external links, alternative download URLs could all be added as tags
- Things like installation instructions and commandline parameters as well
- The files included in the file itself (e. g. things inside a zip archive) will be listed in a dictionary with rudimentary information, this could also be extended for files included in these (e. g. zipped sources, contents of pak files, etc.). This will be used e. g. to determine if something is a mod or just a plain bsp release.
- Anyone will be able to send a Pull Request for proposing changes to these files! Spot a mistake? You can fix it! Installation was broken? You can fix it! Lots of people will get merge rights on the repository.
- The JSON files will very regularly get ingested into a database at Quaddicted that serves a new and improved https://www.quaddicted.com/reviews/ and links the data to its Quaddicted data (description, ratings, comments)


Now, for some bad news. URLs will not be nice and readable like https://www.quaddicted.com/reviews/100b2.html anymore. As I said above, filenames can not serve as identificators unfortunately. Instead we will use a sha256 hash of the file. That's something ugly like 51f6502bd842a0a9d0e439e0f59000f13333141be0f15d1185c323f64bf0612... https://www.quaddicted.com/reviews/51f6 … 64bf0612d/ for a "review" page, https://www.quaddicted.com/files/by-sha … lename.zip for a download URL. A side benefit is that this is content-addressable, an engine or tool could be pointed at a random file, calculate its hash and get the appropriate JSON file for installation and launching instructions.
Here is an example JSON generated from the current DB: https://github.com/SpiritQuaddicted/Qua … cf181.json

How you can help
- The Quake Injector and other tools/engines relying on the current system will stop getting updates and probably break. If you know Java, updating the Quake Injector to this would be awesome. If you are an engine coder, I know that FTEQW and Mark V have some QI-based support.
- I would like to base the new backend and frontend on Python. Backend I can do, but web frontend I suck at. If you know Flask or Django, you could write the new public face of the Quake files at Quaddicted :)
- The JSON files need a schema and a validator, fancy that? Yay!
- Discuss my idea (please also read through the messy https://www.quaddicted.com/forum/viewtopic.php?id=622 ), propose better alternatives (on a small scale, not the general idea)
- Come up with a way to describe complex installation instructions in JSON, think not only extracting to a certain relative path in the Quake directory, but renaming files or placing them in different locations
- Come up with a way to describe launching instructions and engine requirements. Maybe we should just have a commandline:fteqw=-this and that sub-namespace?
- Pull Requests should go through validation and linting before they could be accepted, if you know how to set up that kind of stuff at GitHub, you would help so much!

So far I wrote some Python to play with the existing database, exported it to JSON files (https://github.com/SpiritQuaddicted/Qua … layground/) and imported those back into a new database with just two tables: files and tags.

Next step will be a prototyp of a webpage like https://www.quaddicted.com/reviews/ and the details pages linked from it.

Spirit
2019-12-22 13:52:00

I am on dumptruck's discord and would love some collaboration :)

Spirit
2019-12-22 08:55:02

Maybe a conversion of the current database to that JSON format first, then the reverse of parsing that into the database?

I will be working on that today.

Spirit
2019-12-10 21:32:03

I might find some time in the end of the year. Any suggestions what to focus on? I would prefer to iteratively replace bits of the current system step by step so it stays fun and managable. Maybe a conversion of the current database to that JSON format first, then the reverse of parsing that into the database? Then we could put the JSON files into Git, let everyone contribute and have the database take updates to the live site.

efess
2019-11-10 01:30:38

I'm down. Lets do this!

Spirit
2019-11-06 21:39:27

How about this year? Small, global, hackathon style?

Spirit
2019-01-16 19:41:52

The chat approach turned out not to be a good idea so above is what we discussed. Input and more discussion would be HIGHLY appreciated!

Spirit
2019-01-16 19:41:12
Topher wrote:

the json file can get large with a sha512 for every file.
so we need to send that json only when the user download a package
there has to be a "master" list with the file's sha512 stripped
and then a json for every package, only with the hashes and the identifier.

it's a good idea to add a screenshots array. the metadata will be there. Later the multiple screenshots can be added to the main web site or quake injector. , i don't know the bandwith numbers; that's important with screenshots, even if they are only thumbnails.

Spirit wrote:

for a quake injector thing i would imagine the initial db download being big (everything included) then subsequent updates only being diffs. placing everything into the same file (in git) would make it easiest to maintain. client facing software could decide what it wants to send out. those file lists are meant to be optional! i don't even know what they would be used for in the end

screenshots would be part of the "community stuff" to me, so not part of the file jsons. instead quaddicted or whatever other sites might spawn, could organise screenshots in any way they like for a package :)

ericw wrote:

I like the direction this is taking!
re: screenshots, often mappers/modders have chosen official screenshots or made cover images (screenshot with a logo overlaid, etc.) - it might be nice to have official media like this, that could be considered auxiliary files related to the release, in the json, and community screenshots in the other SQL db?

Spirit wrote:

thanks! :)

official screenshots could just become single items, i would not really like that (waste of space) but it should work. a json itself should only bother with one file and that specific file only, the one it describes

Maxi wrote:

If I understand correctly hash here is mainly for identifying what file you've already downloaded. But should it be main id? What is wrong about good old incremental number? I mean, nothing wrong with hashes too, only disadvantage I see is longer url adresses, but it's not a big deal I guess.

Spirit wrote:

numeric ids would be easy but then there would be some ordering no one needs and the only benefit would be shorter urls i think. for quaddicted pages (not the dowlonads) i'd probably truncate the hashes to something shorter anyways

ericw wrote:

I'd like to toss up 2 ideas:

using the hash as part of the file name for uploads is a great idea, but the JSON file names should not include the hash, but rather should be negotiated by hand to be globally unique. I'm envisioning the json file name being the "quaddicted archive entry name" = "package name" = "release name". I'd have all of the .json files living in a single directory of a git repository. The JSON filename would be identical to the URL for the quaddicted review page, so you'd have quoth2.json and quaddicted.com/reviews/quoth2. Why not use the hash in the json file name? It's bad for usability because it punishes releases that already have unique names (most releases). As an analogy, if you had a Linux package manager like that required doing: sudo apt install firefox-0bd38f just because there are two packages called, say,xutils, and they de-duplicated them by putting a bit of hash in every single package name - it would seem silly. I'd rather instead manually deduplicate the few name collisions, like 3 different maps called castle.bsp or whatever, it's fine if we add a numeric suffix to depulicate them like castle, castle2, castle3, etc.
I would change from a 1:1 mapping from json to the linked file, and allow multiple files to be linked from the .json metadata. Specifically, having the ability to upload multiple files (patches, updates) without creating a new archive entry and have an ordered list of versions. Whether or not to package something as a separate archive entry (separate .json) or bundle it as a newer version of an existing entry would be a community decision, depending on whether it's desired for the new release to have a separate page with its own reviews, tags, etc. But with the current system, we're forced to have a separate archive entry for every upload, and I'd like a new system to be flexible enough to allow not doing that. (suppose there's a map jam released that's uploaded to Quaddicted on the day of release, and the devs release an update a week later to fix some crashes etc., this would probably be a clear cut case where you'd want it to be just a new version inside the same .json metadata.) This would be a powerful feature because it gives you the ability to have a stable URL say for Quoth 2 quaddicted.com/reviews/quoth2, that will still work and give you the latest version of quoth, supposing quoth 2.3 comes out. At the same time you can share a URL to one of the specific file versions like "https://www.quaddicted.com/files/by-sha512/5/78.../quoth2.2.zip if you want to do that.
How I'd imagine this would be something like, suppose this is mapjam10.json:

{"versions" :
  [
  { "name" : "mapjam10update1",
     "date" : "2019-01-07",
     "size": 54792367,
     "sha512": "7814d73af2041c13a8d0700caccf4516cae7d16f62bfd9a196b19794dae5bbfe5ae533d79761a9bd72904cea11dc3bb997368d2d3c603873281b5396ba6915f5",
   "urls": ["https://www.quaddicted.com/files/by-sha512/5/7814d73af2041c13a8d0700caccf4516cae7d16f62bfd9a196b19794dae5bbfe5ae533d79761a9bd72904cea11dc3bb997368d2d3c603873281b5396ba6915f5/mapjam10update1.zip"]
  },
  { "name" : "mapjam10final",
     "date" : "2019-01-01",
     "size": 2343367,
     "sha512": "a814d73af2041c13a8d0700caccf4516cae7d16f62bfd9a196b19794dae5bbfe5ae533d79761a9bd72904cea11dc3bb997368d2d3c603873281b5396ba6915f5",
   "urls": ["https://www.quaddicted.com/files/by-sha512/5/a814d73af2041c13a8d0700caccf4516cae7d16f62bfd9a196b19794dae5bbfe5ae533d79761a9bd72904cea11dc3bb997368d2d3c603873281b5396ba6915f5/mapjam10final.zip"]
  },
  ],
 "tags": ["title:Map jam 10",
  "author:xyz",
  "game:quake",
  "mod:quoth",
  "releasedate:2019-01-01",
  "dependency:id1"]
}

The review page would be at quaddicted.com/reviews/mapjam10. mapjam10update1.zip would be the suggested download but you would have a list of previous versions and could download the mapjam10final.zip if you wanted to check out the original upload.

Spirit wrote:

thanks!!

about the filenames for the json:

quaddicted review pages are currently mapped to filenames and that was a massive failure (showcased perfectly by preach releasing a different quoth2pt2full.zip, baker using the same filename for count Mark V releases and others). linux package repositories are vetted by lots of maintainers and gatekeepers, and they are super centralised. in gaming we have people releasing files everywhere with whatever names they choose. file immutability is important to me and something i want to make a cornerstone of this new setup. i did not know how bad the filename situation would be when i started quaddicted. filenames are something people choose and i would not want to alter them.

what you are suggesting is actually something different and something i really like! in a linux package repo there is no firefox.deb for example, but a immutable, uniquely named firefox-123-abc-etc.deb. but there is a "basename" for the thing 'firefox' and by default it references the latest version of that thing. so i think we should have a basename tag or similar, plus something for the version and the kind of file (patch? full version? addon?)

(would quoth2 be just another version of quoth? who would decide?)

I'd have all of the .json files living in a single directory of a git repository.
idgames2 alone is ~15000 files, that would not work well.

i see website and metadata as two different things here btw. quaddicted or some other site could very well generate their own unique identifiers. but people are use to useless urls nowadays so i see no issues with long ugly hashes in them (although i personally hate that myself...)

quaddicted would not be based 1:1 on this dump of json files. i hope to be able to generate e.g. a list for sp maps, qw maps, tf maps, engines, cd rips, videos, etc. in other words this system should be able to be suit any kind of file imaginable and allow others to then create and present specific selections.

the idea of using the hash for the filename of the json is having a trivial mapping between "what file do i have here" to "this is what this is and how you can use/install it"

maybe 'group' would be a better tag than 'basename'. hm, no idea... maybe there is some wisdom about this to be found in other archival communities (music?)?

and about putting multiple files into one json, hm, that's quite orthogonal to what i had in mind when i thought of their structure... i would want something as unopinionated as possible at a base with tags as option to add opinionated/subjective aspects. some tags of course would need extra attention but things like style and size would be hard to measure.

patched releases can be quite different from their original, or even consider quoth 2 vs 2.2.

tags would also be a place where the possible "startmaps" would be defined, this would break apart if there were different releases inside

what if someone releases unofficial patches or addons, that is too touchy

linux repos have "recommends" and "optional" etc. maybe we could use tags like that to hint at patches and addons

cue flamewar about highres skins being recommends or optional ;)

spike just shared fte's manifest docs with me, i need to look at that to see if we could use ideas from it for installation/unpacking instructions https://sourceforge.net/p/fteqw/code/HE … ifests.txt

Spirit
2019-01-16 19:37:09

examples of potential metadata (with bugs and discrepancies as I wrote them over time):

for gmsp3:

{"filename": "gmsp3.zip",
 "files": {"gmsp3.txt": {"size": 4950, "timestamp": "2003-10-19 16:22:52"},
           "gmsp3v2.bsp": {"size": 5617208,
                           "timestamp": "2003-10-12 19:44:58"}},
 "install": {"extract": "{quake}/id1/maps/"},
 "sha512": "20cb367d7ade48ff98288e588d9fc34e127ee51a84baab92105e1d109822b34a5238bfc04acac56a5865a442d3c0b64694dd191289394bb36d3fdcbad381983d",
 "size": 2319710,
 "tags": ["title:Day of the Lords",
          "author:Howard \"GlassMan\" Shaw",
          "game:quake",
          "mod:id1",
          "bsp:gmsp3v2",
          "style:castle",
          "style:medieval",
          "size:big",
          "todowhat:classic",
          "size:large",
          "textures:unreal",
          "textures:rtgnosis",
          "award:planetquake-map-of-the-week",
          "releasedate:2003-10-19",
          "dependency:id1"],
 "urls": ["https://www.quaddicted.com/files/by-sha512/d/20cb367d7ade48ff98288e588d9fc34e127ee51a84baab92105e1d109822b34a5238bfc04acac56a5865a442d3c0b64694dd191289394bb36d3fdcbad381983d/gmsp3.zip"]}

quoth2.2

{"sha512": "7814d73af2041c13a8d0700caccf4516cae7d16f62bfd9a196b19794dae5bbfe5ae533d79761a9bd72904cea11dc3bb997368d2d3c603873281b5396ba6915f5",
 "filename": "quoth2pt2full.zip",
 "size": 54792367,
 "tags": ["title:Quoth 2.2",
  "author:Preach",
  "author:necros",
  "author:Kell",
  "game:quake",
  "mod:quoth",
  "releasedate:2014-05-04",
  "dependency:id1",
  "version:2.2",
  "basename:quoth"],
 "urls": ["https://www.quaddicted.com/files/by-sha512/5/7814d73af2041c13a8d0700caccf4516cae7d16f62bfd9a196b19794dae5bbfe5ae533d79761a9bd72904cea11dc3bb997368d2d3c603873281b5396ba6915f5/quoth2pt2full.zip"],
 "install": {"extract": "{quake}/quoth/"}}

quoth2.2 with included files from the zip and included files from one of its paks, those things of course would be optional:

{
  "sha512": "7814d73af2041c13a8d0700caccf4516cae7d16f62bfd9a196b19794dae5bbfe5ae533d79761a9bd72904cea11dc3bb997368d2d3c603873281b5396ba6915f5",
  "filename": "quoth2pt2full.zip",
  "size": 54792367,
  "tags": [
    "title:Quoth 2.2",
    "author:Preach",
    "author:necros",
    "author:Kell",
    "game:quake",
    "mod:quoth",
    "releasedate:2014-05-04",
    "dependency:id1",
    "version:2.2",
    "basename:quoth"
  ],
  "urls": [
    "https://www.quaddicted.com/files/by-sha512/5/7814d73af2041c13a8d0700caccf4516cae7d16f62bfd9a196b19794dae5bbfe5ae533d79761a9bd72904cea11dc3bb997368d2d3c603873281b5396ba6915f5/quoth2pt2full.zip"
  ],
  "install": {
    "extract": "{quake}/quoth/"
  },
  "files": {
    "launch.bat": {
      "size": 6927,
      "timestamp": "2014-05-03T11:23:28.000Z",
      "sha512": "c29493f67183b46cb1758e4d9680e2a6e7a675cb996741f8103098d1d3d8292a0fb2e2067fed5723aed7fdfdf387583a7ee34426094d3b24f290952342fc8220"
    },
    "pak0.pak": {
      "size": 48226771,
      "timestamp": "2008-02-06T20:38:00.000Z",
      "sha512": "c0d6f7553368e459522f63d6bd4cfc767a0f7275bd6826b434b3fac378bbeb5bd1935fa32d28959a1d6f4bfed19625f6a45048b695b29f31372c466dec59858e",
      "files": {
        "gfx.wad": {
          "size": 128932,
          "sha512": "fbb73295dc57facb7fb120e9f6d5e56c675cd9cfe5f574c1ab2be8a12776e8ca997a70fe5a10b0f0ce8361d73bf89f9b3644babeb7c494fd5b66266d70d97ac0"
        },
        "gfx/env/bseaup.tga": {
          "size": 786450,
          "sha512": "54bc7c070a2f935b0ffb4f2bccd6d5cafd34c53d5073bdc459dfb01f1590efed1c7218723c258ee572ee2a3fe2c7c61700e43c67e952417b31fa462350154bdb"
        },
        "gfx/env/bseabk.tga": {
          "size": 786450,
          "sha512": "d76af137a8484d092f8852a4920bc5f7e72c7feeaf5b7bb9b5f3c8eb25b18ff67f51b495ca8150d603482c75302f55577488af9e6a4d42681c8dc04b0148cacf"
        },
        "gfx/env/bseadn.tga": {
          "size": 786450,
          "sha512": "781141d24979ea3e652ac22e1a5bac5609b8a4464d5379a11cfbef104127a2fcbd44b5f5a0fe1dbb08206749e3da27d3ffc105845a132399babab41d0aefedb4"
        },
        "gfx/env/bseaft.tga": {
          "size": 786450,
          "sha512": "4c881507ef0a00f7b93fa6c349ed99b9a6469f24f5082789128d58cbeb93284aef04045a2a9520fce5cabe1c78ac6845fbc37366a92f8ff20607ef90de29acc5"
        },
        "gfx/env/bsealf.tga": {
          "size": 786450,
          "sha512": "3c4b36e1916c994e67f34c221476b718d0e1709504ff13da9523a2701f9a82cacc5fb2485dc4bb0e1efe11ef8c11e9cbca69103e6c5d6a377b740c0ea9648ab1"
        },
        "gfx/env/bseart.tga": {
          "size": 786450,
          "sha512": "79e8788b9a1e018f37de69594b1745f8f719603ad0653f6cf29d65b4c67d3d0b7a227a564de2c90eeb2104a7f1653c95ab76c04b84000de1f86d2cd7d3690884"
        },
        "gfx/env/fodrianlf.tga": {
          "size": 502698,
          "sha512": "436f6f951cd5c09a7cdfac878af56bd90fa531186e72a59a4a9bd04a3478a285df50dde7db51785b1ab0ed093ddda4a0d07bfa5c4c9895ce3515cf1960128f11"
        },
        "gfx/env/fodrianrt.tga": {
          "size": 468382,
          "sha512": "990ef367a39e485182fcaf5e14f84efce2a1cc8b764994fce332219cf3187721cf8099294a70ca047aaf59180834073900eec5d1947dbf930b50ffffdbe28520"
        },
        "gfx/env/fodrianup.tga": {
          "size": 538367,
          "sha512": "2f16b501421f116b3f2bd0f1ad094f168033adb59a2d0ef5cd84ffa21a3439fd9e137c66d5c07436e41fac02ccbf364199b9243bf16438f58a725738fc6e985f"
        },
        "gfx/env/starsbk.tga": {
          "size": 51294,
          "sha512": "2fc454733b177ee21ba5dfc74fe40fe0c17aa05dd405d7004cb1dbfa8779dada4432498a8460f4a595d7c768bbe3a8c43fae1506410496275073cb41bbbd990e"
        },
        "gfx/env/starsdn.tga": {
          "size": 51322,
          "sha512": "4abe8d9c39eb75d6496ed55c99b7d4c41c7110c7e0a07f15575e5ab625eaa1ccbafcfe0d543f9a240953e3c6f2d282bd14e08aaa3d0a9e6c1ab0d3e1ea864651"
        },
        "gfx/env/starsft.tga": {
          "size": 51685,
          "sha512": "745039e042cde1e0e82bb96b6c49e8ba168bc1cc9a1a6ce172bf760daad3a67ea3f5df342d01dfe73311b335f04d774e3427664301d331d77bf06b2299ce5117"
        },
[stripped] 

pak1.pak because why not, it's not just zip files:

{"filename": "pak1.pak",
 "install": {"basedir": "{quake}/id1/"},
 "releasedate": "123-10-19",
 "sha512": "123",
 "size": 123,
 "tags": ["title:Quake",
          "author:id Software",
          "game:quake",
          "mod:id1",
          "bsp:e2m1",
          "bsp:e2m2",
          "bsp:e2m3",
          "bsp:e2m4",
          "bsp:e2m5",
          "bsp:e2m6",
          "bsp:...",
          "version:1.06",
          "dependency:id1?!"]
}
ericw
2018-12-18 06:36:56

Thanks - got in!

Spirit
2018-12-17 17:43:04

Oh god how complicated did they make. It was trivial a while back. :o)

The mails should have gone out now.

ericw
2018-12-17 06:37:41

Hi Spirit, I wasn't able to sign up at https://chat.quaddicted.com/
I clicked sign up, entered my email address and got:

Configuration error


It appears there are problems with the email configuration.

See /var/log/zulip/errors.log for more details on the error.

You may also want to test your email configuration, as described in the Production installation docs.

After making your changes, remember to restart the Zulip server.

Spirit
2018-12-16 21:40:44

Maxi, welcome! I am sure you can contribute in many ways, even if they are just comments here and there. Hell, I learned PHP and SQL when building this site originally and it still works (debatebly well ;) ).


topher wrote:

seems solid. i see that currently you have 128bit hashes in the download metadata, between parens.
i would go directly with two chars, that will give 255 folders, that's what git repositories use.

Great idea to follow git there, thanks! The 128bit hashes are md5, those are only good for validation of downloads, not as identifiers and someone could just hack a matching one to be funny.

topher wrote:

about the json file, most implementations take only the last entry of duplicated keys. Luckily the json format is not complex.

I use Python and thus would treat these as dictionaries, duplicate keys definitely should not be there. Instead we will need to use lists for those things that might be multiplicities(?).

topher wrote:

also we will need validators and other things

the good thing about the json is that anyone that can map can write a json and understand the rules pretty quickly.

about the metadata and filecontents i'm thinking two things:
a) guidelines
b) code to handle the most commons cases

yeeee, human-editable is a major benefit. Validator (and some tool to help authors to create these files themselves) would come once we fixed a first version of the format.

topher wrote:

i'm guessing that everything that is already in the archives are well formated and with valid metadata, so the guidelines/code are for new releases

It depends :} But I would use the existing db to generate such json files as a start. Generally I would want them to exist for any Quake file imaginable, retroactively created by the community. All the mods, engines, whatnot. The tagging system should be able to handle this (I like https://wiki.openstreetmap.org/wiki/Map_Features a lot).

topher wrote:

for example, some people put a map file and gfx in the root of the zip file, and in the instructions explain where they go.

They can be community managed, if they're in a github repo. for all purposes, well formed zip files are preferable to instructions to rename and move files.

People will release files in any way they like and there is no central instance (like idgames) to force anything. People release badly packaged maps to this day. Thus this idea of being able to supplement their files with metadata. One important key would be unpacking/installation instructions.

Thanks for your awesome input!

Spirit
2018-12-16 20:13:25
topher wrote:

ok i don't know how to login in that zulip webapp

Sorry, I accidentally left it invite-only (the default). Total fail on my end... It's open now. There is also a desktop client: https://github.com/zulip/zulip-electron

Board footer