#26 2019-11-06 21:39:27

Spirit
Administrator

Re: A weekend of Quaddicted future?

How about this year? Small, global, hackathon style?

#27 2019-11-10 01:30:38

efess
Member

Re: A weekend of Quaddicted future?

I'm down. Lets do this!

#28 2019-12-10 21:32:03

Spirit
Administrator

Re: A weekend of Quaddicted future?

I might find some time in the end of the year. Any suggestions what to focus on? I would prefer to iteratively replace bits of the current system step by step so it stays fun and managable. Maybe a conversion of the current database to that JSON format first, then the reverse of parsing that into the database? Then we could put the JSON files into Git, let everyone contribute and have the database take updates to the live site.

#29 2019-12-22 08:55:02

Spirit
Administrator

Re: A weekend of Quaddicted future?

Maybe a conversion of the current database to that JSON format first, then the reverse of parsing that into the database?

I will be working on that today.

#30 2019-12-22 13:52:00

Spirit
Administrator

Re: A weekend of Quaddicted future?

I am on dumptruck's discord and would love some collaboration :)

#31 2019-12-22 14:52:10

Spirit
Administrator

Re: A weekend of Quaddicted future?

#32 2019-12-23 20:44:53

Spirit
Administrator

Re: A weekend of Quaddicted future?

We have a dedicated channel on dumptruck's discord now, you are welcome to join in there.

Here is what I posted as mission:

So here is the gist:
This is about https://www.quaddicted.com/reviews/ and the database behind it

The problem(s)
- The site is in a state where updates are requiring too much work and the code is terrible
- Uploading files relies on me and only few people can then add them to the database
- Because people release different versions of files with the same filename, the whole system of using the filename as idenficator is not working
- Because people like to release maps not as zip but also as plain bsp files or rar or 7z, the limitation to zip files does not work
- Because files are often packaged weirdly and file paths inside of releases can clash with others (think "readme.txt") automated installations with the Quake Injector often fail or require manual repackaging of files (which goes against the whole idea of an archive). All we have in the database at the moment is an instruction where to extract a zip file which does not make a conflict-free and working installation possible in some cases


The idea(s)
- Instead of having the database editor interface available to just a few people (:heart:negke:heart:), we will have a public Git repository filled with one JSON file per released file
- This JSON file will describe the file itself and have a dictionary of tags describing the contents
- This includes neutral information like author names, the title, tags about the size or texture set, the releasedate etc.
- This does not include the description (maybe it will? let's discuss this), the editor's rating, the user ratings or comments, those will stay in the Quaddicted database
- Things like dependencies, external links, alternative download URLs could all be added as tags
- Things like installation instructions and commandline parameters as well
- The files included in the file itself (e. g. things inside a zip archive) will be listed in a dictionary with rudimentary information, this could also be extended for files included in these (e. g. zipped sources, contents of pak files, etc.). This will be used e. g. to determine if something is a mod or just a plain bsp release.
- Anyone will be able to send a Pull Request for proposing changes to these files! Spot a mistake? You can fix it! Installation was broken? You can fix it! Lots of people will get merge rights on the repository.
- The JSON files will very regularly get ingested into a database at Quaddicted that serves a new and improved https://www.quaddicted.com/reviews/ and links the data to its Quaddicted data (description, ratings, comments)


Now, for some bad news. URLs will not be nice and readable like https://www.quaddicted.com/reviews/100b2.html anymore. As I said above, filenames can not serve as identificators unfortunately. Instead we will use a sha256 hash of the file. That's something ugly like 51f6502bd842a0a9d0e439e0f59000f13333141be0f15d1185c323f64bf0612... https://www.quaddicted.com/reviews/51f6 … 64bf0612d/ for a "review" page, https://www.quaddicted.com/files/by-sha … lename.zip for a download URL. A side benefit is that this is content-addressable, an engine or tool could be pointed at a random file, calculate its hash and get the appropriate JSON file for installation and launching instructions.
Here is an example JSON generated from the current DB: https://github.com/SpiritQuaddicted/Qua … cf181.json

How you can help
- The Quake Injector and other tools/engines relying on the current system will stop getting updates and probably break. If you know Java, updating the Quake Injector to this would be awesome. If you are an engine coder, I know that FTEQW and Mark V have some QI-based support.
- I would like to base the new backend and frontend on Python. Backend I can do, but web frontend I suck at. If you know Flask or Django, you could write the new public face of the Quake files at Quaddicted :)
- The JSON files need a schema and a validator, fancy that? Yay!
- Discuss my idea (please also read through the messy https://www.quaddicted.com/forum/viewtopic.php?id=622 ), propose better alternatives (on a small scale, not the general idea)
- Come up with a way to describe complex installation instructions in JSON, think not only extracting to a certain relative path in the Quake directory, but renaming files or placing them in different locations
- Come up with a way to describe launching instructions and engine requirements. Maybe we should just have a commandline:fteqw=-this and that sub-namespace?
- Pull Requests should go through validation and linting before they could be accepted, if you know how to set up that kind of stuff at GitHub, you would help so much!

So far I wrote some Python to play with the existing database, exported it to JSON files (https://github.com/SpiritQuaddicted/Qua … layground/) and imported those back into a new database with just two tables: files and tags.

Next step will be a prototyp of a webpage like https://www.quaddicted.com/reviews/ and the details pages linked from it.

#33 2020-12-22 10:05:27

hemebond
Member

Re: A weekend of Quaddicted future?

Spirit wrote:

- Instead of having the database editor interface available to just a few people (:heart:negke:heart:), we will have a public Git repository filled with one JSON file per released file

This will not make it easier for regular mappers. It would require a Github account and knowledge of how source control works. Or knowing someone who knows these things.

Spirit wrote:

- This JSON file will describe the file itself and have a dictionary of tags describing the contents
- This includes neutral information like author names, the title, tags about the size or texture set, the releasedate etc.

This would just be a mostly unstructured document of strings that the "client" (website compiler or Quake engine) would have to manually parse as a different syntax.

Spirit wrote:

- Things like dependencies, external links, alternative download URLs could all be added as tags
- Things like installation instructions and commandline parameters as well

This is where fixing the packages would make the data more useful and more likely to be used.

Spirit wrote:

- The files included in the file itself (e. g. things inside a zip archive) will be listed in a dictionary with rudimentary information, this could also be extended for files included in these (e. g. zipped sources, contents of pak files, etc.). This will be used e. g. to determine if something is a mod or just a plain bsp release.

This is apparently not particularly useful and would require more functionality in the clients to work around the bad "data". A list of "start" maps is at least what FTEQW wants.

Spirit wrote:

- Anyone will be able to send a Pull Request for proposing changes to these files! Spot a mistake? You can fix it! Installation was broken? You can fix it! Lots of people will get merge rights on the repository.

I think a simple web interface created specifically for uploading maps and mods would be better.

Spirit wrote:

- The JSON files will very regularly get ingested into a database at Quaddicted that serves a new and improved https://www.quaddicted.com/reviews/ and links the data to its Quaddicted data (description, ratings, comments)

I think Quaddicted would be far more useful as a dedicated map/mod repository with an interface made for simple clients. It's also in a position to enforce some standardisation on uploads; or at least recommendations.

Spirit wrote:

- Come up with a way to describe launching instructions and engine requirements. Maybe we should just have a commandline:fteqw=-this and that sub-namespace?

Having to describe how a particular client should launch or use the data is, I think, a losing battle. Far better, I think, to make the data clean, and then describe it well.

Spirit wrote:

- Pull Requests should go through validation and linting before they could be accepted, if you know how to set up that kind of stuff at GitHub, you would help so much!

Likewise, uploads via the website could/should be unpublished until approved by a moderator.

This is just how I see Quaddicted; like a lvlworld for Quake 1 maps. I think making it a general "bucket" of files would be to make it less useful; and also make it a less-used clone of moddb.com.

#34 2020-12-23 20:42:51

Spirit
Administrator

Re: A weekend of Quaddicted future?

Thanks! I hope this was not new to you, but that you had seen it last year when this all started?

I think there are two important things I must stress:
- There will never ever be editing of files. The site is an archive and must support any kind of format or structure.
- A focus on just SP maps (and whatever constitutes SP mods) would be a missed opportunity as all other Quake files are also in dire need of a proper catalog.

hemebond wrote:
Spirit wrote:

- Instead of having the database editor interface available to just a few people (:heart:negke:heart:), we will have a public Git repository filled with one JSON file per released file

This will not make it easier for regular mappers. It would require a Github account and knowledge of how source control works. Or knowing someone who knows these things.

That was still a huge step from being three people with editor access as it is nowadays, plus it would have been public, with free history and tons of tooling and APIs around it.

hemebond wrote:
Spirit wrote:

- This JSON file will describe the file itself and have a dictionary of tags describing the contents
- This includes neutral information like author names, the title, tags about the size or texture set, the releasedate etc.

This would just be a mostly unstructured document of strings that the "client" (website compiler or Quake engine) would have to manually parse as a different syntax.

It would be simply key=value strings, nothing hard to parse and easily supporting multiple values for a key (e. g. multiple authors). Dunno though, the alternative would be having something like:

"tags": {"author":["single author"], "dependency": ["packageA", "packageB"]}

Same data but you are right, this would save a step of additional parsing.

hemebond wrote:
Spirit wrote:

- Things like dependencies, external links, alternative download URLs could all be added as tags
- Things like installation instructions and commandline parameters as well

This is where fixing the packages would make the data more useful and more likely to be used.

Repackaging goes against the spirit of an archive and it was a big mistake that we did this in the past instead of fixing the issues at the core of the system (metadata and Quake Injector support). Even if it was different, this would be a horribly time-intense labour.

hemebond wrote:
Spirit wrote:

- The files included in the file itself (e. g. things inside a zip archive) will be listed in a dictionary with rudimentary information, this could also be extended for files included in these (e. g. zipped sources, contents of pak files, etc.). This will be used e. g. to determine if something is a mod or just a plain bsp release.

This is apparently not particularly useful and would require more functionality in the clients to work around the bad "data". A list of "start" maps is at least what FTEQW wants.

This would be hugely useful. For example one could easily see if a particular file is also bundled in other releases. Or clients could make guesses about features of a package based on its metadata (without having to download and inspect it themselves).

hemebond wrote:
Spirit wrote:

- Anyone will be able to send a Pull Request for proposing changes to these files! Spot a mistake? You can fix it! Installation was broken? You can fix it! Lots of people will get merge rights on the repository.

I think a simple web interface created specifically for uploading maps and mods would be better.

Sure! This was a thought on the process if a Git-based approach was used.

hemebond wrote:
Spirit wrote:

- The JSON files will very regularly get ingested into a database at Quaddicted that serves a new and improved https://www.quaddicted.com/reviews/ and links the data to its Quaddicted data (description, ratings, comments)

I think Quaddicted would be far more useful as a dedicated map/mod repository with an interface made for simple clients. It's also in a position to enforce some standardisation on uploads; or at least recommendations.

Quaddicted is nowhere near that position nor should it ever be. Recommendations have been there for years and are constantly ignored even by experienced people. People will always release things as they like and the system must be able to handle that. It is/should-be an archive, not a stream-lined, repackaged bunch of selected releases.

hemebond wrote:
Spirit wrote:

- Come up with a way to describe launching instructions and engine requirements. Maybe we should just have a commandline:fteqw=-this and that sub-namespace?

Having to describe how a particular client should launch or use the data is, I think, a losing battle. Far better, I think, to make the data clean, and then describe it well.

It would cost pretty much nothing to have optional namespaces for certain clients. If someone cares for one, they can add what is needed (think of all the DP gameplayfixes/-bugs for example), other clients can just ignore it.

hemebond wrote:
Spirit wrote:

- Pull Requests should go through validation and linting before they could be accepted, if you know how to set up that kind of stuff at GitHub, you would help so much!

Likewise, uploads via the website could/should be unpublished until approved by a moderator.

Sure! This was a thought on the process if a Git-based approach was used.

hemebond wrote:

This is just how I see Quaddicted; like a lvlworld for Quake 1 maps. I think making it a general "bucket" of files would be to make it less useful; and also make it a less-used clone of moddb.com.

We have a long history of trouble on the border between maps and mods, this is something that has done a lot of damage in the past. The site already is a bunch of other files, take a look through https://www.quaddicted.com/files/. The origin was the "SP maps" and thus only that got an interface at a certain point. This is not the vision I have nor has it been robust for the past 5 years or so.

#35 2020-12-24 01:51:32

fairweather
Member

Re: A weekend of Quaddicted future?

There's a bit of a problem between what people want and expect, and what currently happens for files. Dwell Episode 1 being split between version 1.0 and 1.2 is nonsensical, as is Copper being split between like 7 different versions.
As a middle-ground, some kind of in-listing versioning would do wonders. Not every listing can be AD (where there is a mass of comments and ratings with each release) and having to re-link your listing across sites is extremely annoying.
It could function the same as it does now, where versions have different ratings and comments, but a tab/dropdown could switch you back to older versions, since downloading them is most likely an extremely niche thing 99.99% of people would never do. You could also possibly keep the ratings and comments, but show what version they were commented on (older ratings would be compiled into an average, and newer ones would be split into their own thing.)
The old Dwell / Copper / SMEJ listings do nothing but cause confusion, and I feel it's only this way because you're trying to keep Injector compatibility at the detriment of the usability of the site. Injector should be updated as the site needs it, not the other way around imo.

(Edit: It also doesn't help that this is inconsistent at best, with SMEJ having it's patch relegated to an optional download... patches should be opt-out, nobody wants to download a broken version of a map pack just because the archivist thinks the patch is less worthy of it, or something.)

Last edited by fairweather (2020-12-24 01:54:59)

#36 2020-12-24 08:05:09

hemebond
Member

Re: A weekend of Quaddicted future?

Spirit wrote:

Thanks! I hope this was not new to you, but that you had seen it last year when this all started?

I had seen it but couldn't find it again because I didn't realise it was in the news section :-D

Spirit wrote:

I think there are two important things I must stress:
- There will never ever be editing of files. The site is an archive and must support any kind of format or structure.
- A focus on just SP maps (and whatever constitutes SP mods) would be a missed opportunity as all other Quake files are also in dire need of a proper catalog.

Spirit wrote:

The origin was the "SP maps" and thus only that got an interface at a certain point. This is not the vision I have nor has it been robust for the past 5 years or so.

Hmm, I'm going to message you on Discord so we can have a chat. I think I've been a bit unfair with where I've been taking the rebuild, pushing it in the direction I wanted, and I don't think I share your goal for it.

#37 2020-12-24 08:13:52

Spirit
Administrator

Re: A weekend of Quaddicted future?

No need to be so snarky. Seriously, that's not nice.

Functionality on the frontend is very important and something that could have been implemented by anyone via https://github.com/SpiritQuaddicted/Quaddicted-reviews (best don't touch that old spaghetti...). It would be a major effort to integrate your ideas in that. Same for the Injector.

I like your ideas but they are not directly related to the core issues I put the focus on first and which are supposed to be the foundation of a new frontend. Does the structure of metadata discussed in this thread not provide the basis for your ideas? If not, please suggest how it could be modelled instead.

#38 2020-12-24 08:14:51

Spirit
Administrator

Re: A weekend of Quaddicted future?

Hemebond: Yeah, let's do that! I am sure we can find a good way. :)

#39 2020-12-24 10:25:56

onetruepurple
Banned

Re: A weekend of Quaddicted future?

hemebond wrote:

I think I've been a bit unfair with where I've been taking the rebuild, pushing it in the direction I wanted, and I don't think I share your goal for it.

See, this may be a problem... I think the mapping community's goals for Quaddicted might be a bit incompatible with whatever Spirit considers it to be.

Last edited by onetruepurple (2020-12-24 14:56:37)

#40 2020-12-24 23:13:56

Spirit
Administrator

Re: A weekend of Quaddicted future?

No one needs vague statements of controversy. Please don't post such negativity.

I see nothing incompatible here, if you do, be constructive if you can.

#41 2023-01-10 21:56:21

Spirit
Administrator

Re: A weekend of Quaddicted future?

I had some time between the years and so I continued old work and built a small Flask app that reads JSONB from Postgres and renders websites with jinja2. Comments are currently implemented using embedded Discourse threads. Discourse is also used as main base for user accounts and authentication. Files can be hosted anywhere.

https://github.com/SpiritQuaddicted/Qua … /by-sha256 got updated too. Those JSONs are created from the current database plus some extra data on tag classifications. I am not 100% sure how editing will look like, it will probably just be something like https://jsoneditoronline.org/ or https://www.jeremydorn.com/json-editor to create or edit the JSON data and then actually just a little Git repository for a start because that gives us easy ways for automated checks and community moderation in pull/merge requests.

There should be a public PostgREST endpoint for querying the database in smart ways some day, I would like to use that to power the website too. Right now it's simply psycopg2.

SHA256 hashes are used as identifiers for things so we won't have issues with filename collisions anymore. Unfortunately SHA256 hashes are long and unwieldy so I think some truncation for less huge URLs would be nice. Not sure how much can be stripped away before collisions become a reasonable issue though. https://www.quaddicted.com/file/62681e9 … de4cf4e8e5 is a bit crazy... I guess appending filenames or titles at the end would be nice too.

If *you* want to help, these topics would be awesome for someone to tackle: JSON Schema, some nice jinja2 templates for map things, mods, etc., a compatible re-implementation of the Quake Injector in PyQt, a reasonably way to have image uploads including metadata etc., a good classification system with tags (I am inspired by https://wiki.openstreetmap.org/wiki/Map_features on that front)

And I am not sure if using Discourse is really the best way forward. I like it as a user and admin but when embedding, the users can not add replies on the same page, they always have to click through to the actual full forum thread page. I could not find a better alternative though. If you know something, please shout. The issue it solves is user management, comment moderation and authentication.

#42 2023-01-11 01:42:30

dumptruck_ds
Member

Re: A weekend of Quaddicted future?

Great news that this work is moving forward!

#43 2023-05-24 20:21:11

Spirit
Administrator

Re: A weekend of Quaddicted future?

I did some work backend- and frontend-wise for grouping packages/releases together. It is hard to find tags that nicely model this though, any thoughts on this? https://github.com/SpiritQuaddicted/Qua … dac185f371

It's kind of a messy mixture of https://wiki.archlinux.org/title/PKGBUILD#Dependencies and https://packaging.python.org/en/latest/ … g-metadata with a manually set main_group_package=yes which we would need to keep updated for the group. All heavily inspired by the great free tagging system of OSM: https://wiki.openstreetmap.org/wiki/Map_features

And I am not sure if having a readable name for the group (quoth here) is even a good idea or if it would lead to issues and fighting later. A UUID would be enough for making a nice frontend but it would be horrible for human editors of the data.

https://www.quaddicted.com/files/misc/2 … llapse.mp4 shows what this enables.

#44 2023-06-24 16:57:39

Spirit
Administrator

Re: A weekend of Quaddicted future?

Oops, the diff above was missing "release_group=quoth" on all of them. I am not super happy with that structure, any thoughts?

Work this week included:
- An automated process that updates the database if JSONs are edited on GitHub \o/
- First test migration of the forum from the current one to Discourse which was mostly successful (some markup didn't convert, e.g. inline images :(
- And some minor stuff like fixing problems with user accounts, polishing the template of map pages, testing some Discourse plugin to rate topics (since release comments are basically just forum topics this might just be enough for us!)

TODOs include:
- Converting the map comments to posts, including their ratings
- Provide a sweet API against the DB (PostgREST is the plan) for tools and whatever people want to build
- Figure out how we can easily get aggregate ratings from that Discourse plugin without writing a custom script
- Decide how to handle release images/screenshots

Where *you* can help:
- See the JSON stuff above and try to find problems with it
- Building a nice looking Discourse theme (I'd prefer to go more brown again instead of this weird greenish tint, but anything Quakey is fine!)
- Build a new Quake Injector (PyQt please!) D:
- Any thoughts on images for releases? Currently we have 1 which is not enough for many things, e.g. map packs. There might be official images as well as "editor's impressions" and even users' images. How to handle this? Where to host which? Should official images get JSON descriptions as well? How to reference them? I'd prefer a clean distinction between official imagery and editor/user imagery. The latter could just be hosted by the forum because it allows file uploads.
- If you're a programmer and know a bit of Ruby, help with converting the comments to fit https://github.com/discourse/discourse/ … generic.rb would be greatly appreciated. The first post would need to be created in a special way so that it is referenced correct when embedding. Just shoot me a mail if you wanna try to tackle this.

#45 2023-07-23 17:01:57

Spirit
Administrator

Re: A weekend of Quaddicted future?

Migrating the comments to Discourse is harder than expected because back then I wrote the commenting system I thought it would be soo cooool to have few limits on characters in usernames. Something that led to usernames like the awesome "Ðꧧï©å‡êР©ø®þ§ê". Discourse has reasonable rules about usernames and now I have to check and write some mappings manually. Not hard but not fun so that takes a while...

Board footer