Community Content Pack 2.0 Guidelines

[Johannes]

Note: for those just stopping by, scroll all the way down for the download, and something else


Community Content Pack 2.0 Guidelines

This Guide Discusses a format of organizing user-created content in Overgrowth.
It has been designed with the content-browser, editor, and the Server that will house it in mind.
It will be able to handle content as small as a custom texture, and as large as a full blown replacement mod.





Index

1. - Why should I use this stupid format anyways?
2. - The good news and the bad news.
3. The Folder Format
4. - Custom/Username/Projectname/
5. - /Version/
6. - /Content
7. The 'About' File
8. - JSON
9. - Content
10. Community Content Pack 2.0
11. Suggestions



So, you've made some custom models, or maybe even some full Levels, and want to release your masterpieces for the whole Community to see?

Well this is a guide is for you! It will should you how you should format your stuff together so we can enjoy it, and it can be included in the next Community Content Pack.





Why should I use this stupid format anyways?

You might wonder why these rules are put in place. Couldn't we just place our stuff into the Data folders as each of us pleases?

This is true, that would work for a time, but after a while it would become incredibly difficult to find what you need, and without a standard format, everyone would have have their own custom instructions for installing their content. It would be a complete and total mess, especially if you start mixing the files into those of the original engine, and boy will it be hard to find stuff then, and when something messes up and you want to find where the problem is good luck.

To solve these problems, creating a standard location and format for user-created content is crucial.

Another reason for keeping strict rules on the format of the content pack is because ultimately it will be housed on a server and accessed via the in-game content browser. For the browser to be able to easily manage all this information things need to be consistent.

Lastly, having it all in one location allows you to just drag and drop it from one weekly alpha to the next, without having to go file-hunting.



The good news and the bad news

The good news is that because this will all be managed by the server, mod browser and engine, ultimately you won't have to worry about it much at all.

The bad news is that until the mod browser and server are set up, you need to manually format your stuff, and with the next release some of the format might have changed. Such is the way things go during alpha development. That or, you could just wait until the final release, but you wouldn't be on the SPF if you just wanted to wait, now would you

Regardless this guide should help with understanding and following this format.









=Disclaimer: All of this is still subject to change until the server is up and the mod browser are fully implemented.=





The Folder Format

First and foremost the user created content should NEVER mix with the original game content, apart maybe from very quick and momentary tests.
Where and how you place your content is as follows:

Code: Select all

/Data/Custom/USERNAME/PROJECTNAME/VERSION(optional)/CONTENT

Here is an example of what this looks like, featuring Nimai's Kohlibriania Village:

Note: this is obviously a screenshot of the mac version, but the file structure inside Data will be identical for the windows, and linux versions.

The following sections will describe each part of this file-path, and why it is the way it is:



Custom/Username/Projectname/

Any user-created content needs to be inside the Custom folder at the very least. This will make transferring your work and content from one alpha to the next very easy and pain-free: Just drag the custom folder from the old alpha's Data folder to that in the new alpha.
Ideally you should keep all your personal stuff inside Custom/your-forum-name/


As of now file-paths are absolute, and moving folders will break them. Because of that, it's simply best to have files that objects and levels will link to in their final locations.

A possible alternative to absolute filepaths as they are now will be suggested later.



/Version/

Sometimes there will be multiple versions of a Project released over time.

One might argue 'why not just replace the previous version?'
Well keep in mind that this folder system will be the same as that on the server, which will house all the versions. Multiple versions are necessary because sometimes people just like previous versions more, or (as some of us know all to well with some of the alphas) sometimes newer versions of something don't work as well as they should.

To support multiple versions an optional version folder can be added into to the file-path as in the image at the beginning. This folder will always be formated as either "v#.#" or "v#.#.#". This allows the user to keep Seperate versions of a project in their Custom folder while allowing the versions to all be saved under one name.

Once implemented, the content browser should by default load the most recent version of a project, but you'll be able to choose an older version of you want to.

Again as it stands now, because file-paths are absolute, you need to change all the paths linking to your content from levels and objects to include the "v#.#" folder so that they won't break when you decide your piece should have different versions.

A possible alternative to absolute filepaths as they are now will be suggested later.



/Content

Once inside the Projects folder (or version folder if there is one) the folder structure should mirror that of the actual Data folder.

Not only will this be consistent, but it will enable selective, non-destructive replacement of existing files.

For example:
Say you want to change a texture of the original Campaign, and just for the sake of this example let's say that that texture is located at /Data/Textures/image1.png
You could place your image at /Data/Custom/YOURNAME/PROJECTNAME/Textures/image1.png
and tell the engine to load your project as a replacement.

You could use this method to replace any aspect of the game, from sounds to animations to shaders, and so it would allow for extensive customization that is easily redistributed and doesn't actually change anything in the original Files.

This illustrates a possible solution for the problem of absolute Filepaths: If a project is loaded, have the engine treat the contents inside each Project folder as if it was actually in the normal file structure in Data. See the "Suggestions" section for a full proposal.



The 'About' File

If the Folder structure illustrated above is the 'body' of the Project, then the About file is its 'soul'.

It is a JSON file that stores all the metadata that will be used to organize and manage the Projects within the Server and Content Browser.

It Lives in the Project folder (or version folder if one exists) besides the content folders:

Picture 10.png (32.22 KiB) Viewed 5358 times

Because the engine should be able to find this file quickly and easily it is important that it is -always- in this location, and will always be titled "About.json"



About.json files should -not- be modified after their release. This means that they will -not- contain meta-data such as user-rating, etc.



JSON

Originally the About file was going to be XML based, but was recently modified to be JSON based.

JSON is a Javascript based Data 'interchange' format that is easily readable for humans and because it is Javascript based, will play nice with the engine, including the future Content browser, which uses Google's V8 Javascript engine.

The Content pack includes a Template file that will make editing very easy, and so learning JSON is not a prerequisite, but if you are curious hit the 'JSON' Link above for the wikipedia article.

Because exactly what the content browser will require from this file is not 100% decided yet, the exact syntax is still subject to change, but overall its structure shouldn't change much anymore.



Content

An example of what is contained in an About.json file:

TextEditScreenSnapz004.png (11.42 KiB) Viewed 5361 times

It is largely self explanatory, and there is a fully commented version included in the content pack which explains each section in detail.

Here are a select few sections explained:

• - "Content": "tags": ["Tag1","Tag2"]
  These tags will be used by the content browser to help find content. the title of the project or the author(s) don't need to be included, and should also be considered by the browser's search engine. The search engine should ignore the letter 's' at the end of tags, so 'Canyon' and 'Canyons' would be considered the same tag.
  
  - "Content":"dependencies":["/Username1/Project1/","/Username2/Project2/v1.0/"] Filepaths to other Projects who are used in this Project. If listed Projects are not present or not loaded, this Project won't load either. This allows users to use content from other Projects, without having to copy the content and so saving space. It's likely that this will ultimately be replaced by a some kind of ID the server uses for each project, or a code or hash-code that takes the version of the Project into account.
  
  - "Team":"Authors":"forum_id":"#####"
  The Forum ID of the author's account on the wolfire forums (found by clicking the user's name at the top of a post, and looking at the end of the url). Once the server is set up and users have usernames there that userID could be used instead, or "server_id" might replace it, and the forum ID could just be included in the user's account on the server. So far at least all of the OG modding has been on the forums so it is a good temporary way of identifying the author.

Community Content Pack 2.0

For those of you that weren't scared off by all that mumbo jumbo and just want to try out the current version of the content pack, which includes a template About.JSON and a fully commented version, as well as most of the current user-created stuff here is the file.

TO DOWNLOAD THE NEW CONTENT PACK PLEASE CLICK HERE! (about 50 MB)




Suggestions

This section is mainly stuff for the devs to consider when they start implementing the Content browser


About and relative file-paths:

Absolute file-paths, meaning filepaths that are based on the location of a file relative to the root folder, so the parent of /Data/, are impractical for custom user content and can become really long and difficult to manage:

Code: Select all

Data/Custom/User/Project/v1.0.1/Textures/Terrain/Terrain1/Terrain1_c.png

This is also further complicated by the fact that level files don't store objects, but the contents of objects, giving you thousands of long filepaths in each level, that makes moving stuff around very annoying.

A possible solution for this is to have an option to make the file-paths relative to the project folder, or version folder; basically the folder containing About.json
This would make the file read like this: Data/Textures/Terrain/Terrain1/Terrain1_c.png where the engine would deal with 'Data' as the filepath of About.json.

This solution ties in well with the previously suggested non-destructive file replacement. Wherever the engine keeps track of which files are currently installed there needs to be a boolean that defines whether the given project is enabled or not.

Basically when the engine loads it Iterates through files in /Custom/ and Finds the About.json files at /Custom/username/projectname/About.json or /Custom/username/projectname/v#.#/About.json or Custom/username/projectname/v#.#.#/About.json
and then uses their location as the root of the contents of each project.

The contents browser uses displays the static metadata from the about.json file (remember, About.json is never modified after download)



I'm probably not making much sense anymore, it's quite late, and I had to retype much of this after my computer decided to crash and I hadn't backed it up in a good while. Sorry.
I know I had some more suggestions on my mind earlier, but can't think of any more at the moment...

Since I don't want to leave you hanging, and so you have an idea of how I envision the end result, here is a mock-up of the Content browser I threw together:

(Please forgive the lame description I gave Hale's awesome map)

Each 'Box' with a thumbnail, title, version number and type is a Project. Currently the browser's sorting mode is set to 'Authors' (equivalent to the 'username' folder') which shows an alphabetized list of the Authors, each of which, when expanded, shows their projects.
Grayed out Authors and Projects show that they are not currently set to load.

I will post another mockup of what browsing the community server will look like in the future.



So what do you think?

-Jo

[Nimai]

Wicked layout for the Content Browser Jo!!
i will work my texture pack in with the content format sometime tomorrow.
Awesome work!

[Sajber]

Very nice indeed! Good job!

***

No offense...but the mock up content browser...is über sexy!

[Odin]

Really pretty one. I am just wondering why there is a rating system if you are not able to (seems so) sort or filter objects by their rating.

[Johannes]

Really pretty one. I am just wondering why there is a rating system if you are not able to (seems so) sort or filter objects by their rating.

Ah heh, good eye
This view is supposed to be of what the user currently has downloaded on their computer. I figured if the user downloaded something, the rating souldn't matter once it's on their computer, since ratings are there (in part) to help you decide/find what you should download.
You -would- be able to sort pieces by rating if you clicked 'Community' which (for lack of better name) is the button that connects you to the online database of user-made content, where the rating is more meaningful.
However you are right, a button for rating-based organization probably would have been good anyways

[Mykei]

The content browser is obviously inspired by Gmod

[Johannes]

The content browser is obviously inspired by Gmod

if you look back at some of my older posts concerning the content pack you'll see that I often reffer to gmod

I'd be willing to argue that in some ways my design is better. Gmod tries to throw all the tools in there as well and it can be a pain in the ass to find what you want. Nonetheless many things worked for them. My file design was particularly inspired by gmod's addon format as well as using a meta file (though for them it's LUA based) in that package. however the annoying thing with gmod is that people often don't conform to the standards set, and browsing/installing content cannot be done inside the engine itself.

[Jendraz]

Nice, jo This would make user-mods much easier to manage.

[matto1990]

Just so you are aware, there are some naming errors with some of the About.json files. Some are called:

About.Json

With a capital J in the file extention, which caused my praser to miss some of them. I got round it by making sure it was case insensitive when checking however you might want to change it so they are totally consistant.

Cheers for the packs!

[Johannes]

Just so you are aware, there are some naming errors with some of the About.json files. Some are called:

About.Json

With a capital J in the file extention, which caused my praser to miss some of them. I got round it by making sure it was case insensitive when checking however you might want to change it so they are totally consistant.

Cheers for the packs!

Ah, thank you
Honestly I've been writing all of these completely manually based on specifications I learned off Wikipedia a week ago. I would truly be surprised if the whole thing actually loaded properly the first time they plugged it into a JSON loader.
I'll make sure the extensions are all consistent next time, thanks again

[matto1990]

Ah, thank you
Honestly I've been writing all of these completely manually based on specifications I learned off Wikipedia a week ago. I would truly be surprised if the whole thing actually loaded properly the first time they plugged it into a JSON loader.
I'll make sure the extensions are all consistent next time, thanks again

Funny you mention that. They don't work What I can do for you is go through each one and tidy them up so they do work and then send them all to you. That going to be helpful?

Edit: I did it anyway You can download it. It's in a zip file with exactly the same folder structure as the second pack you sent out, just with all the files that aren't json ones stripped out. You should be able to merge them into your folders with no problems at all.

At that needed doing was making the team element valid (removing the "Author" bit) and getting rid of some whitespace that was confusing all the validators. Oh and the empty dependancies bit was wrong (just [] and not [""] works).

To be honest I'm not a json expert either but they seem to work and php can prase them perfectly which is good enoguh for me

[CameO73]

After my my initial post I gave it some more thought. As mentioned above, the "Author:" identifiers are not necessary (since these objects will be accessed through the array -- see example below).

If you really want to remove the quotes around the field identifiers (making it more readable, but not JSON valid; it will load fine in any JS engine, though), you have to make sure it's not one of the reserved words. As you can see, this can be a pain (since common ids like "name" and "date" are reserved). If you want to travel that road, it could look something like this:

Code: Select all

{
   Project:{
      Content: {
         title: "Placeholder Textures, Models and Terrain",
         version: "1.0",
         type: "Compilation",
         created: "03/22/2009",
         url: "http://forums.wolfire.com/viewtopic.php?f=13&t=4779",
         state: "Finished",
         tags: ["Placeholder", "Whaleman", "Debug", "Objects", "Textures",  "Models", "Terrains"],   
         note: "A compilation of textures, models and a terrain for the engine for it to use if it cannot find the appropriate kind of file, instead of crashing. it also includes Objects in a level to demonstrate their use.",
         dependencies: [""]   
      },
      Team: [
         {
            author:"Jo-Shadow",
            role:"Project Leader",
            email:"[email protected]",
            website:"http://blog.jo-shadow.com",
            forum_id:"13570"
         },
         {      
            author:"Sajber",      
            role:"Artist",
            email:"",
            website:"http://www.moddb.com/members/sajber",
            forum_id:"14605"
         }
      ]
   }
}

(note: apart from the reserved words I also changed the Team.title to Team.role to distinguish them better from the Project.title)

If the JSON files are generated (and you really can't be bothered to check for reserved keywords everytime these properties are changed) you might just want to keep the quotes.

One tip: the best way to check these JSON structures is to use Firefox + Firebug. Firebug has a console that makes debugging JS easy. Just create an object with the above JSON and use console.debug() to check 'em. Use the large command line in Firebug for this purpose (Console tab -> Options -> Larger Command Line).

Code: Select all

TEST = { ... JSON ... }

console.debug(TEST.Project.Team[0].author)

--> result = Jo-Shadow

Edit: I noticed the reserved keywords reference I previously used was a bit strict. I've changed it to a better reference. It turns out that identifiers like "name", "date" and "status" shouldn't give any problems (tested this with Firebug). As you can see removing the quotes might not be worth the trouble

[matto1990]

The trouble with removing quotes is that you make it invalid json, which might not be a problem for browsers which are used to dealing with developers imperfections, however practically all other json decoders are much more dtrict. I've built a php system which scans the zip package for any About.json files and prases them to be put into a MySQL database. The end goal of this is to develop either a website or AIR application which uses that database to act as an auto installer for Overgrowth.

We need to make sure that the json files have the highest compatability possible so that it doesn't limit peoples potential to use them. I agree that it's much easier to read, however human readability is only part of the equation.

[Johannes]

The trouble with removing quotes is that you make it invalid json, which might not be a problem for browsers which are used to dealing with developers imperfections, however practically all other json decoders are much more dtrict. I've built a php system which scans the zip package for any About.json files and prases them to be put into a MySQL database. The end goal of this is to develop either a website or AIR application which uses that database to act as an auto installer for Overgrowth.

We need to make sure that the json files have the highest compatability possible so that it doesn't limit peoples potential to use them. I agree that it's much easier to read, however human readability is only part of the equation.

As you correctly pointed out the JSON files will never be used outside of two scenarios:
1. The Server-side part of the Content browser, for not-yet-downloaded content.
2. The Client-side part of the Content browser, for already-downloaded content.

Once it works with those two, it doesn't matter at all how compatible the files are with anything else. Because of that I will take both of your suggestions about the quotes into account, but I will wait until The Devs start testing JSON files with a server and client-side interpreter. If it becomes apparent then that it just won't do with the quotes, we'll leave them in, otherwise we'll leave them out.

Frankly I feel because most users will just copy and paste their information into the basic template file, and never touch the original JSON, removing the quotes will have minimal effect, and later all of this information will be input via the engine anyways, and at that point it really doesn't matter what the readability of the file is like for humans. The only Argument I can see for removing quotes that would still matter then is Filesize, which would be minimal at a few dozen characters, but worth considering nonetheless.

I will make the author changes you suggested CameO73, and matto1990, I will download the 'fixed' version you made and check out the changes to see if there's anything else I missed. Thanks alot to both of you for helping here =)


Also not to be nitpicky, (rather, I'm really just bit confused), but matto1990, you keep saying "praser" and "prases"... Do you mean the word "parser" and "parses"?
I'm just confused because you say it so consistently that maybe it's a word I don't know, but I can't find it anywhere.

[matto1990]

Also not to be nitpicky, (rather, I'm really just bit confused), but matto1990, you keep saying "praser" and "prases"... Do you mean the word "parser" and "parses"?
I'm just confused because you say it so consistently that maybe it's a word I don't know, but I can't find it anywhere.

Sorry, that's just me spelling badly. Just imagine it's being spelt right and you'll get the idea.

I was actually saying the exact opposite about the quotes. I think they should be kept in. There's no point making it invalid json just so it's slightly easier to read. I've built a program that uses the json files and it wont work if the quotes are removed. If we can keep them in it would be greatly appreciated