#26 2019-11-06 21:39:27

Spirit
Administrator

Re: A weekend of Quaddicted future?

How about this year? Small, global, hackathon style?

#27 2019-11-10 01:30:38

efess
Member

Re: A weekend of Quaddicted future?

I'm down. Lets do this!

#28 2019-12-10 21:32:03

Spirit
Administrator

Re: A weekend of Quaddicted future?

I might find some time in the end of the year. Any suggestions what to focus on? I would prefer to iteratively replace bits of the current system step by step so it stays fun and managable. Maybe a conversion of the current database to that JSON format first, then the reverse of parsing that into the database? Then we could put the JSON files into Git, let everyone contribute and have the database take updates to the live site.

#29 2019-12-22 08:55:02

Spirit
Administrator

Re: A weekend of Quaddicted future?

Maybe a conversion of the current database to that JSON format first, then the reverse of parsing that into the database?

I will be working on that today.

#30 2019-12-22 13:52:00

Spirit
Administrator

Re: A weekend of Quaddicted future?

I am on dumptruck's discord and would love some collaboration :)

#31 2019-12-22 14:52:10

Spirit
Administrator

Re: A weekend of Quaddicted future?

#32 2019-12-23 20:44:53

Spirit
Administrator

Re: A weekend of Quaddicted future?

We have a dedicated channel on dumptruck's discord now, you are welcome to join in there.

Here is what I posted as mission:

So here is the gist:
This is about https://www.quaddicted.com/reviews/ and the database behind it

The problem(s)
- The site is in a state where updates are requiring too much work and the code is terrible
- Uploading files relies on me and only few people can then add them to the database
- Because people release different versions of files with the same filename, the whole system of using the filename as idenficator is not working
- Because people like to release maps not as zip but also as plain bsp files or rar or 7z, the limitation to zip files does not work
- Because files are often packaged weirdly and file paths inside of releases can clash with others (think "readme.txt") automated installations with the Quake Injector often fail or require manual repackaging of files (which goes against the whole idea of an archive). All we have in the database at the moment is an instruction where to extract a zip file which does not make a conflict-free and working installation possible in some cases


The idea(s)
- Instead of having the database editor interface available to just a few people (:heart:negke:heart:), we will have a public Git repository filled with one JSON file per released file
- This JSON file will describe the file itself and have a dictionary of tags describing the contents
- This includes neutral information like author names, the title, tags about the size or texture set, the releasedate etc.
- This does not include the description (maybe it will? let's discuss this), the editor's rating, the user ratings or comments, those will stay in the Quaddicted database
- Things like dependencies, external links, alternative download URLs could all be added as tags
- Things like installation instructions and commandline parameters as well
- The files included in the file itself (e. g. things inside a zip archive) will be listed in a dictionary with rudimentary information, this could also be extended for files included in these (e. g. zipped sources, contents of pak files, etc.). This will be used e. g. to determine if something is a mod or just a plain bsp release.
- Anyone will be able to send a Pull Request for proposing changes to these files! Spot a mistake? You can fix it! Installation was broken? You can fix it! Lots of people will get merge rights on the repository.
- The JSON files will very regularly get ingested into a database at Quaddicted that serves a new and improved https://www.quaddicted.com/reviews/ and links the data to its Quaddicted data (description, ratings, comments)


Now, for some bad news. URLs will not be nice and readable like https://www.quaddicted.com/reviews/100b2.html anymore. As I said above, filenames can not serve as identificators unfortunately. Instead we will use a sha256 hash of the file. That's something ugly like 51f6502bd842a0a9d0e439e0f59000f13333141be0f15d1185c323f64bf0612... https://www.quaddicted.com/reviews/51f6 … 64bf0612d/ for a "review" page, https://www.quaddicted.com/files/by-sha … lename.zip for a download URL. A side benefit is that this is content-addressable, an engine or tool could be pointed at a random file, calculate its hash and get the appropriate JSON file for installation and launching instructions.
Here is an example JSON generated from the current DB: https://github.com/SpiritQuaddicted/Qua … cf181.json

How you can help
- The Quake Injector and other tools/engines relying on the current system will stop getting updates and probably break. If you know Java, updating the Quake Injector to this would be awesome. If you are an engine coder, I know that FTEQW and Mark V have some QI-based support.
- I would like to base the new backend and frontend on Python. Backend I can do, but web frontend I suck at. If you know Flask or Django, you could write the new public face of the Quake files at Quaddicted :)
- The JSON files need a schema and a validator, fancy that? Yay!
- Discuss my idea (please also read through the messy https://www.quaddicted.com/forum/viewtopic.php?id=622 ), propose better alternatives (on a small scale, not the general idea)
- Come up with a way to describe complex installation instructions in JSON, think not only extracting to a certain relative path in the Quake directory, but renaming files or placing them in different locations
- Come up with a way to describe launching instructions and engine requirements. Maybe we should just have a commandline:fteqw=-this and that sub-namespace?
- Pull Requests should go through validation and linting before they could be accepted, if you know how to set up that kind of stuff at GitHub, you would help so much!

So far I wrote some Python to play with the existing database, exported it to JSON files (https://github.com/SpiritQuaddicted/Qua … layground/) and imported those back into a new database with just two tables: files and tags.

Next step will be a prototyp of a webpage like https://www.quaddicted.com/reviews/ and the details pages linked from it.

#33 2020-12-22 10:05:27

hemebond
Member

Re: A weekend of Quaddicted future?

Spirit wrote:

- Instead of having the database editor interface available to just a few people (:heart:negke:heart:), we will have a public Git repository filled with one JSON file per released file

This will not make it easier for regular mappers. It would require a Github account and knowledge of how source control works. Or knowing someone who knows these things.

Spirit wrote:

- This JSON file will describe the file itself and have a dictionary of tags describing the contents
- This includes neutral information like author names, the title, tags about the size or texture set, the releasedate etc.

This would just be a mostly unstructured document of strings that the "client" (website compiler or Quake engine) would have to manually parse as a different syntax.

Spirit wrote:

- Things like dependencies, external links, alternative download URLs could all be added as tags
- Things like installation instructions and commandline parameters as well

This is where fixing the packages would make the data more useful and more likely to be used.

Spirit wrote:

- The files included in the file itself (e. g. things inside a zip archive) will be listed in a dictionary with rudimentary information, this could also be extended for files included in these (e. g. zipped sources, contents of pak files, etc.). This will be used e. g. to determine if something is a mod or just a plain bsp release.

This is apparently not particularly useful and would require more functionality in the clients to work around the bad "data". A list of "start" maps is at least what FTEQW wants.

Spirit wrote:

- Anyone will be able to send a Pull Request for proposing changes to these files! Spot a mistake? You can fix it! Installation was broken? You can fix it! Lots of people will get merge rights on the repository.

I think a simple web interface created specifically for uploading maps and mods would be better.

Spirit wrote:

- The JSON files will very regularly get ingested into a database at Quaddicted that serves a new and improved https://www.quaddicted.com/reviews/ and links the data to its Quaddicted data (description, ratings, comments)

I think Quaddicted would be far more useful as a dedicated map/mod repository with an interface made for simple clients. It's also in a position to enforce some standardisation on uploads; or at least recommendations.

Spirit wrote:

- Come up with a way to describe launching instructions and engine requirements. Maybe we should just have a commandline:fteqw=-this and that sub-namespace?

Having to describe how a particular client should launch or use the data is, I think, a losing battle. Far better, I think, to make the data clean, and then describe it well.

Spirit wrote:

- Pull Requests should go through validation and linting before they could be accepted, if you know how to set up that kind of stuff at GitHub, you would help so much!

Likewise, uploads via the website could/should be unpublished until approved by a moderator.

This is just how I see Quaddicted; like a lvlworld for Quake 1 maps. I think making it a general "bucket" of files would be to make it less useful; and also make it a less-used clone of moddb.com.

#34 2020-12-23 20:42:51

Spirit
Administrator

Re: A weekend of Quaddicted future?

Thanks! I hope this was not new to you, but that you had seen it last year when this all started?

I think there are two important things I must stress:
- There will never ever be editing of files. The site is an archive and must support any kind of format or structure.
- A focus on just SP maps (and whatever constitutes SP mods) would be a missed opportunity as all other Quake files are also in dire need of a proper catalog.

hemebond wrote:
Spirit wrote:

- Instead of having the database editor interface available to just a few people (:heart:negke:heart:), we will have a public Git repository filled with one JSON file per released file

This will not make it easier for regular mappers. It would require a Github account and knowledge of how source control works. Or knowing someone who knows these things.

That was still a huge step from being three people with editor access as it is nowadays, plus it would have been public, with free history and tons of tooling and APIs around it.

hemebond wrote:
Spirit wrote:

- This JSON file will describe the file itself and have a dictionary of tags describing the contents
- This includes neutral information like author names, the title, tags about the size or texture set, the releasedate etc.

This would just be a mostly unstructured document of strings that the "client" (website compiler or Quake engine) would have to manually parse as a different syntax.

It would be simply key=value strings, nothing hard to parse and easily supporting multiple values for a key (e. g. multiple authors). Dunno though, the alternative would be having something like:

"tags": {"author":["single author"], "dependency": ["packageA", "packageB"]}

Same data but you are right, this would save a step of additional parsing.

hemebond wrote:
Spirit wrote:

- Things like dependencies, external links, alternative download URLs could all be added as tags
- Things like installation instructions and commandline parameters as well

This is where fixing the packages would make the data more useful and more likely to be used.

Repackaging goes against the spirit of an archive and it was a big mistake that we did this in the past instead of fixing the issues at the core of the system (metadata and Quake Injector support). Even if it was different, this would be a horribly time-intense labour.

hemebond wrote:
Spirit wrote:

- The files included in the file itself (e. g. things inside a zip archive) will be listed in a dictionary with rudimentary information, this could also be extended for files included in these (e. g. zipped sources, contents of pak files, etc.). This will be used e. g. to determine if something is a mod or just a plain bsp release.

This is apparently not particularly useful and would require more functionality in the clients to work around the bad "data". A list of "start" maps is at least what FTEQW wants.

This would be hugely useful. For example one could easily see if a particular file is also bundled in other releases. Or clients could make guesses about features of a package based on its metadata (without having to download and inspect it themselves).

hemebond wrote:
Spirit wrote:

- Anyone will be able to send a Pull Request for proposing changes to these files! Spot a mistake? You can fix it! Installation was broken? You can fix it! Lots of people will get merge rights on the repository.

I think a simple web interface created specifically for uploading maps and mods would be better.

Sure! This was a thought on the process if a Git-based approach was used.

hemebond wrote:
Spirit wrote:

- The JSON files will very regularly get ingested into a database at Quaddicted that serves a new and improved https://www.quaddicted.com/reviews/ and links the data to its Quaddicted data (description, ratings, comments)

I think Quaddicted would be far more useful as a dedicated map/mod repository with an interface made for simple clients. It's also in a position to enforce some standardisation on uploads; or at least recommendations.

Quaddicted is nowhere near that position nor should it ever be. Recommendations have been there for years and are constantly ignored even by experienced people. People will always release things as they like and the system must be able to handle that. It is/should-be an archive, not a stream-lined, repackaged bunch of selected releases.

hemebond wrote:
Spirit wrote:

- Come up with a way to describe launching instructions and engine requirements. Maybe we should just have a commandline:fteqw=-this and that sub-namespace?

Having to describe how a particular client should launch or use the data is, I think, a losing battle. Far better, I think, to make the data clean, and then describe it well.

It would cost pretty much nothing to have optional namespaces for certain clients. If someone cares for one, they can add what is needed (think of all the DP gameplayfixes/-bugs for example), other clients can just ignore it.

hemebond wrote:
Spirit wrote:

- Pull Requests should go through validation and linting before they could be accepted, if you know how to set up that kind of stuff at GitHub, you would help so much!

Likewise, uploads via the website could/should be unpublished until approved by a moderator.

Sure! This was a thought on the process if a Git-based approach was used.

hemebond wrote:

This is just how I see Quaddicted; like a lvlworld for Quake 1 maps. I think making it a general "bucket" of files would be to make it less useful; and also make it a less-used clone of moddb.com.

We have a long history of trouble on the border between maps and mods, this is something that has done a lot of damage in the past. The site already is a bunch of other files, take a look through https://www.quaddicted.com/files/. The origin was the "SP maps" and thus only that got an interface at a certain point. This is not the vision I have nor has it been robust for the past 5 years or so.

#35 2020-12-24 01:51:32

fairweather
Member

Re: A weekend of Quaddicted future?

There's a bit of a problem between what people want and expect, and what currently happens for files. Dwell Episode 1 being split between version 1.0 and 1.2 is nonsensical, as is Copper being split between like 7 different versions.
As a middle-ground, some kind of in-listing versioning would do wonders. Not every listing can be AD (where there is a mass of comments and ratings with each release) and having to re-link your listing across sites is extremely annoying.
It could function the same as it does now, where versions have different ratings and comments, but a tab/dropdown could switch you back to older versions, since downloading them is most likely an extremely niche thing 99.99% of people would never do. You could also possibly keep the ratings and comments, but show what version they were commented on (older ratings would be compiled into an average, and newer ones would be split into their own thing.)
The old Dwell / Copper / SMEJ listings do nothing but cause confusion, and I feel it's only this way because you're trying to keep Injector compatibility at the detriment of the usability of the site. Injector should be updated as the site needs it, not the other way around imo.

(Edit: It also doesn't help that this is inconsistent at best, with SMEJ having it's patch relegated to an optional download... patches should be opt-out, nobody wants to download a broken version of a map pack just because the archivist thinks the patch is less worthy of it, or something.)

Last edited by fairweather (2020-12-24 01:54:59)

#36 2020-12-24 08:05:09

hemebond
Member

Re: A weekend of Quaddicted future?

Spirit wrote:

Thanks! I hope this was not new to you, but that you had seen it last year when this all started?

I had seen it but couldn't find it again because I didn't realise it was in the news section :-D

Spirit wrote:

I think there are two important things I must stress:
- There will never ever be editing of files. The site is an archive and must support any kind of format or structure.
- A focus on just SP maps (and whatever constitutes SP mods) would be a missed opportunity as all other Quake files are also in dire need of a proper catalog.

Spirit wrote:

The origin was the "SP maps" and thus only that got an interface at a certain point. This is not the vision I have nor has it been robust for the past 5 years or so.

Hmm, I'm going to message you on Discord so we can have a chat. I think I've been a bit unfair with where I've been taking the rebuild, pushing it in the direction I wanted, and I don't think I share your goal for it.

#37 2020-12-24 08:13:52

Spirit
Administrator

Re: A weekend of Quaddicted future?

No need to be so snarky. Seriously, that's not nice.

Functionality on the frontend is very important and something that could have been implemented by anyone via https://github.com/SpiritQuaddicted/Quaddicted-reviews (best don't touch that old spaghetti...). It would be a major effort to integrate your ideas in that. Same for the Injector.

I like your ideas but they are not directly related to the core issues I put the focus on first and which are supposed to be the foundation of a new frontend. Does the structure of metadata discussed in this thread not provide the basis for your ideas? If not, please suggest how it could be modelled instead.

#38 2020-12-24 08:14:51

Spirit
Administrator

Re: A weekend of Quaddicted future?

Hemebond: Yeah, let's do that! I am sure we can find a good way. :)

#39 2020-12-24 10:25:56

onetruepurple
Banned

Re: A weekend of Quaddicted future?

hemebond wrote:

I think I've been a bit unfair with where I've been taking the rebuild, pushing it in the direction I wanted, and I don't think I share your goal for it.

See, this may be a problem... I think the mapping community's goals for Quaddicted might be a bit incompatible with whatever Spirit considers it to be.

Last edited by onetruepurple (2020-12-24 14:56:37)

#40 2020-12-24 23:13:56

Spirit
Administrator

Re: A weekend of Quaddicted future?

No one needs vague statements of controversy. Please don't post such negativity.

I see nothing incompatible here, if you do, be constructive if you can.

Quick reply

Write your message and submit
Are you human or robot? If you have trouble, mail to spirit åt quaddicted døt c

Checking if this is requested by a real person and not an automated program.

Board footer