Keen:Definitions

This is a highly technical document written by programmers (for other programmers to read). In time, the language will be revised but we felt it was important to have the information available even in its full technical jargon form.

Many of these definition standards were established in version 0.5. Of special interest in 0.6 is the addition of #Definition_Merging

We have a data driven engine, where the conditions and parameters of the world are highly customizable and can be composed in many ways. The system that provides and exposes all of this customization is the definition system.

A good number of objects in the engine can be parameterized or specified through definitions, this ranges from entities, to entity components, inventory items, session components and more. As such definitions are the primary format through which game data is stored and made available.

Note This article is meant to be read in the order the topics are laid out. We also assume basic understanding of xml files and good understanding of the basics of the VRage engine.

Overview

Definitions are, to the user, a set of xml files with the .sbc extension inside the game's /Content/Data folder. These files represent C# objects which are called Object Builders. They are essentially a user friendly intermediate format we use to interface the easy to write xml definitions, with the game ready definition objects used by the engine and game logic.

These intermediate files will always inherit from a base class called MyObjectBuilder_DefinitionBase, this class contains a user provided ID, and a few utility members that we will discuss in detail later in this article.

Object builders are de-serialized using C# xml serialization and collected by a class called the Definition Loader (MyDefinitionLoader). This class will classify and merge definitions from game and mods based on their types. After that they will be converted into definition objects and post-processed with the help of the Definition Factory (MyDefinitionFactory).

These definition objects all inherit from MyDefinitionBase and are liked with their object builders via the MyDefinitionType attribute. Every definition has a void Init(MyObjectBuilder_DefinitionBase) and MyObjectBuilder_DefinitionBase GetObjectBuilder() methods which allow conversion to and from object builder.

Once definitions are initialized they are passed on to a class called MyDefinitionManager. This class simply holds all of the definitions and allows the user to query them by their id and or type.

Example

For the rest of this article we will refer to an example definition and how to modify several aspects of it to achieve a more user friendly result. Note that this example is artificially constructed to suit this document, it should not be taken as an example of the type of data a definition should contain.

The example is a definition for a projectile (for a throwing weapon, or ballista):

[MyDefinitionType(typeof(MyObjectBuilder_ProjectileDefinition))]
public class MyProjectileDefinition : MyDefinitionBase {
	public float DeviationRadians;

	public float Speed;

	public string Model;

	public long TimeoutTicks;

	public Color ModelTint;

	public int HitParticleEffectId;

	public Dictionary<MyStringHash, float> DamagePerMaterial;

	protected override void Init(MyObjectBuilder_DefinitionBase){...}
	public override MyObjectBuilder_DefinitionBase GetObjectBuilder(){...}
}

Initially we will just have our object builder contain the same data, but we will soon have to modify it:

public class MyObjectBuilder_ProjectileDefinition : MyObjectBuilder_DefinitionBase {
	public float DeviationRadians;

	public float Speed;

	public string Model;

	public long TimeoutTicks;

	public Color ModelTint;

	public int HitParticleEffectId;

	public Dictionary<MyStringHash, float> DamagePerMaterial;
}

Definition Format

The format of an .sbc file should be as follows:

<?xml version="1.0" encoding="UTF-8"?>
<Definitions xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
	...
	<Definition xsi:type="{DefinitionType}">
	</Definition>
	...
</Definitions>

Each file can contain any number of definitions, each of a any type. The type of a definition is given by the xsi:type attribute. This tells our serialization system what is the type to be used for de-serializing the definition.

For the most part definitions will be identified by the type of their object builder. When creating a new type just beware of collisions between names since they must be unique.

Some definitions will use custom type names, they will still work with their object builder types. That was an attempt to push for definitions to have more user friendly type names and to use some form of namespaces to make collisions less likely. That approach has mostly died out by now.

The format above is what we would call the format for new definitions.

Object Builder Structure

Object builders are usually a POD type, the focus is in storing information in a way that produces xml which is friendly to users. There are several ways in which we can structure the data to make the xml more readable or even function altogether (depending on the case). We will look onto those in detail.

Note that object builders are not just used for definitions, they are a general serialization intermediate used across the engine. Before you proceed to write your own you should also have a read at Xml Serialization and Multiplayer Serialization not to get caught off guard.

Dictionaries

Dictionaries are a very common data structure, they make sense to use in a lot of definitions, and even definitions themselves can be seen as part of a key-value relation with respect to their id and their content.

Still, dictionaries are not supported natively by .net's xml serialization (the serializer will simply fail to generate and thrown an exception). Additionally they have no direct equivalent in xml, instead you have to use a list and construct the dictionary at runtime (unlike other markups like json and yaml).

Looking only at the dictionary in our example definition, we have to produce something like:

class MyObjectBuilder_ProjectileDefinition {
	struct DamageEntry {
		public MyStringHash Material;
		public float Amount;
	};

	public DamageEntry[] DamagePerMaterial;
}

Then later when initializing the definition we must convert this list into a dictionary:

class MyProjectileDefinition {
	protected override void Init(MyObjectBuilder_DefinitionBase builder) {
		var ob = (MyObjectBuilder_ProjectileDefinition)builder;
		...

		foreach(var entry in ob.DamagePerMaterial)
			DamagePerMaterial[entry.Material] = entry.Amount;
	}
}

User and Xml Friendly Types

When it comes to having non primitive types in your object builder it is always very important to consider three aspects:

• Readability: Make sure your types are easy to specify and fit the document elegantly;
• Consistency: If the data the type specifies is used often you should consider using the same type in all definitions.
• Serializability: Not all types will work with the xml serializer, you need to make sure to use the proper intermediate types if that is the case.

This also means choosing the right type of input format, the data is to be specified by designers and users that, while hopefully in power of sufficient knowledge about how thew game functions, might not have a technical background and or good understanding of maths and software engineering. Therefore the input data must usually be expressed in terms of better know units (such as metres, seconds, degrees) rather than the specific unit chosen by the programmer.

This is particularly relevant when inputting more complex data such as 3d orientations, quaternions and matrices may be closest to the actual end result, but forward-up pairs and explicit positions are simpler to write and clearer to the user. Whenever possible any data that requires high input precision should come from some dummy on a model or other source that can be queried at runtime, and positioned by the user with a more intuitive tool than the definition xml.

Case Study

Lets take a look at the problematic fields in our definition:

• public float DeviationRadians: This field is not readable because radians are not intuitive to most users. We should replace it with degrees.
• public long TimeoutTicks: This field is not readable because ticks are system dependent and users are not aware of how ticks translate to a more natural unit of time. One solution is to replace it with seconds or some other unit. But better yet is to be consistent and use a TimeDefinition, which allows the user to specify time as a the unit that is most relevant to the situation.
• public Color ModelTint: Color is not a serializable type, we need to replace it with an alternative format. For consistency we should use ColorDefinitionRGBA which is already used in many other places.
• public int HitParticleEffectId: In this case the type is serializable but particle effect indices's are not user friendly, the better solution is to allow the user to input the name of the particle effect.
• public DamageEntry[] DamagePerMaterial: In the previous section we already dealt with one serializability aspect of this member which was the incompatibility of dictionaries with serialization. Another type that is not serializable is the MyStringHash type. To fix this we should use a string in the object builder instead.

With all of these considerations this is our new object builder (which now for the first time is usable):

public class MyObjectBuilder_ProjectileDefinition : MyObjectBuilder_DefinitionBase {
	public float Deviation;

	public float Speed;

	public string Model;

	public TimeDefinition Timeout;

	public ColorDefinitionRGBA ModelTint;

	public string HitParticleEffect;

	struct DamageEntry {
		public string Material;
		public float Amount;
	};

	public DamageEntry[] DamagePerMaterial;
}

And the final Init() method that accounts for all of these changes:

class MyProjectileDefinition {
	protected override void Init(MyObjectBuilder_DefinitionBase builder) {
		var ob = (MyObjectBuilder_ProjectileDefinition)builder;
		...

		DeviationRadians = MathHelper.ToRadians(ob.Deviation);
		Speed = ob.Speed;
		Model = ob.Model;
		TimeoutTicks = ((TimeSpan)ob.Timeout).Ticks;
		ModelTint = ob.ModelTint;
		HitParticleEffectId = EffeMyDefinitionManager.Get<MyEffectDefinition>(entry.EffectId).ParticleId;

		foreach(var entry in ob.DamagePerMaterial)
			DamagePerMaterial[MyStringHash.GetOrCompute(entry.Material)] = entry.Amount;
	}
}

This is the first version of the object builder that will actually serialize correctly. So let's take a look at an example of the generated xml:

<Definition xsi:type="MyObjectBuilder_ProjectileDefinition">
	<Id Type="ProjectileDefinition" Subtype="Arrow"/>

	<Deviation>5</Deviation>
	<Speed>53</Speed>
	<Model>Models/Projectiles/Arrow.mwm</Model>
	<Timeout Seconds="5"/>
	<ModelTint Hex="#FFCC00"/>
	<HitParticleEffect>FeathersPoof</HitParticleEffect>

	<DamagePerMaterial>
		<DamageEntry>
			<Material>Stone</Material>
			<Amount>3</Amount>
		</DamageEntry>
		<DamageEntry>
			<Material>Wood</Material>
			<Amount>5</Amount>
		</DamageEntry>
		<DamageEntry>
			<Material>Flesh</Material>
			<Amount>15</Amount>
		</DamageEntry>
	</DamagePerMaterial>
</Definition>

Xml Attributes

Another way to make object builders produce better and more readable xml is through the use of attributes. Attributes allow more information to be compressed to single tags, that way we can produce results that are more semantic and compact.

In our example the best usage of attributes would be in the DamageEntry struct:

struct DamageEntry {
	[XmlAttribute]
	public string Material;
	[XmlAttribute]
	public float Amount;
};

The parsed xml gets simplified to:

<Definition xsi:type="MyObjectBuilder_ProjectileDefinition">
	...
	<DamagePerMaterial>
		<DamageEntry Material="Stone"	Amount="3" 	/>
		<DamageEntry Material="Wood"	Amount="5" 	/>
		<DamageEntry Material="Flesh"	Amount="15" />
	</DamagePerMaterial>
</Definition>

List as Elements

The last thing we could do to make our definition even more compact is to get rid of the <DamagePerMaterial> tags. This can be accomplished by means of the XmlElement attribute:

class MyObjectBuilder_ProjectileDefinition {
	...
	[XmlElement]
	public DamageEntry[] DamagePerMaterial;
}

This attribute tells the xml serializer to not create a separate block where to place the list elements, instead they are placed at the same level as the rest of the fields. The final result is as follows:

<Definition xsi:type="MyObjectBuilder_ProjectileDefinition">
	<Id Type="ProjectileDefinition" Subtype="Arrow"/>

	<Deviation>5</Deviation>
	<Speed>53</Speed>
	<Model>Models/Projectiles/Arrow.mwm</Model>
	<Timeout Seconds="5"/>
	<ModelTint Hex="#FFCC00"/>
	<HitParticleEffect>FeathersPoof</HitParticleEffect>

	<DamageEntry Material="Stone"	Amount="3" 	/>
	<DamageEntry Material="Wood"	Amount="5" 	/>
	<DamageEntry Material="Flesh"	Amount="15" />
</Definition>

Definition Class

Once all of the details of the object builder are set you are ready to spend more time on your definition. Ideally the object builder is in an ideal format for data input, whereas the definition is ready to be used by the game.

This means that in many situations you will cache post-processed results in the definition itself, such as: angles already in radians or sin(), cos(), tan() values; lengths already squared; referenced definitions can be cached; and more.

Additionally you'll want to sanitize the data from the object builder and make sure the generated definition has valid data.

Loading

Once the definition object builder is de-serialized it goes through a series of processes that lead up to initialization of the definition object. These in-between processes are called loading. It is comprised of three steps:

• Collection: During collection all object builders from base game and all mods are separated by their type and mod context;
• Merging: After being collected the object builders are queued based on their load order and indexed by id. Object builders from mods are loaded according to the mod's load order. If a mod specified an object builder with the same id as a base game definition the two are merged. Merging is described in further detail bellow.
• Copying: After definitions are merged they are indexed for the CopyFrom mechanism. This allows definitions to be created from existing ones, reducing duplication.

Finally each definition object is initialized from it's object builder. The whole process is done in turns, one object builder type at a time.

Initializing

Upon initialization the object builder will be filled with the data provided by the user, the definition should then validate and then load this data. Whenever values are out of range or in incorrect format the definition may output warnings or errors to the Log. If the definition is in a state that it is not valid at all, and therefore cannot be used, it may throw an instance of MyDefinitionException. In this case the definition loader will write to the log the provided message and additional information about the source of the error.

If any other exception is thrown the loader will record the event, but will also re-throw the exception. The save loading process will catch this exception and then report the issue with loading the world.

Dependencies

Definitions can specify dependencies, this is useful when one definition wishes to cache a reference to another definition of a different type. To do that one simply needs to annotate the definition class with the [MyDependency(typeof(MyOtherDefinition))] Attribute.

This will affect the order in which definitions are loaded, which is why definitions get initialized in turns, all of one type at a time. Then it is possible to request any definition depended upon by simply asking the definition manager like usual.

A definition automatically has it's parent types as dependencies, which is a requirement for copying from parent definitions. Any other dependencies are then considered when establishing the topological ordering of definition types. Additionally some definitions depend not only on another definition type but all of it's children, it is possible to establish this relation by setting the Recursive property of the MyDependency attribute.

Here is an example where the recursive tag is important:

[MyDefinitionType(typeof(MyObjectBuilder_PlanetGeneratorDefinition))]
[MyDependency(typeof(MyVoxelMaterialDefinition), Recursive = true)]
[MyDependency(typeof(MyWorldEnvironmentDefinition), Recursive = true)]
public class MyPlanetGeneratorDefinition : MyDefinitionBase
{
	...
}

In this definition both MyVoxelMaterialDefinition and MyWorldEnvironmentDefinition can have children. This results in a need for depending on them as well via the Recursive attribute. It is extremely important to always consider weather or not a dependency should be recursive when creating a definition, especially because the problem may not be apparent at first try.

Of course it is possible that cycles be formed when specifying dependencies, if that is the case then the programmer must choose which dependency should have priority and the other should be left to runtime or post-processing. A dependency on itself will also be considered a cycle.

Definition Management

Once definitions are loaded and initialized they are passed on to a class called the MyDefinitionManager which provides a suite of methods used to query definitions. This class allows definitions to be queried by id, subtype and or type. For the most part definitions are not added after the game has already loaded.

Definition Groups

Definitions in general cannot ever share the same Id. That is quite true for definitions of the same type. But not quite if their types differ. More specifically definitions are organized in so called Definition Groups, these are nothing but separations in the top level definition dictionary, what they achieve is that definitions in different groups may share the same Id.

All definitions in a group must share the same base class, that class which was annotated with MyDefinitionGroupAttribute. The group is then inferred at query time by checking the type of the definition being asked for (all queries in MyDefinitionManager take a definition type as a generic parameter). It is very important then that whenever a definition is asked for, the user be sure to use the right type so as to query the correct group.

Definition Id

All definitions are identified by a unique id. This id is a pair, composed of type and subtype. Together these two identify the definition in the game, and are used to locate it when necessary.

The id is expressed by means of the SerializableDefinitionId object (and in game as the MyDefinitionId struct). You will find id's expressed in two ways usually:

• A more compact form, which is recommended:

<Id Type="MyObjectBuilder_Character" Subtype="Medieval_female" />

• And a legacy format that we are trying to phase out:

<Id>
	<TypeId>MyObjectBuilder_Character</TypeId>
	<SubtypeId>Medieval_female</SubtypeId>
</Id>

Definition Type

The type is the name of the object builder for the object you want to create. This touches into a slight nuance of definitions, there are essentially two cases: either they represent a piece of data, used by some system; or they represent an object, other than the definition, that is ultimately constructed from it.

This is usually follows a pattern, when a definition only defines an object, the object will have the same name minus the definition suffix. E.g.: a definition for the inventory component will have an object builder named MyObjectBuilder_ModelComponentDefinition but when used this definition will specify the type MyObjectBuilder_ModelComponent or ModelComponent for short (object builder types can optionally drop the MyObjectBuilder_ prefix in data). Whereas a mining definition for instance will specify VoxelMiningDefinition because the end result is the definition alone (which is then shared by several objects).

Subtype

Within the same type of definition, it is the subtype that identifies it, this is what separates distinct objects of the same type. In general the subtype can be any string, and can even be absent (a behavior sometimes used to signify that this definition is the default).

Definition Handlers

Every definition type gets an associated definition handler instance. These classes provide hooks for handling aspects that last the lifetime of the definition type. There are currently three events that can be modified:

• BeforeLoad: Fired right before a definition type starts to load, all dependent definitions are already initialized.
• AfterLoad: Fired after all definitions have been loaded, the list of the loaded definitions is also provided.
• Unload: Fired when definitions are being unloaded, if your definition caches any resources that need disposing, this is the right moment.

These methods allow better management of any static data associated with the definition as well. For instance some definitions (such as voxel material definitions) will have a secondary index which is static and handled by the definition class itself.

Handlers also allow the user to fine tune the definition merging process, if that is required, by overriding the HandleOverride method.

Definition Merging

When a mod defines a definition with the same id as one in the base game or previously loaded mod, this definition is not copied over the previous, but merged. This process is fundamental because it allows modders to partially modify game definitions without risks.

When merging a modder can specify how the process proceeds, this is done by setting the Merge attribute in the definition itself. There are three merge modes: Override, the default mode which completely replaces the original definition; Merge, which will preserve untouched fields and override lists (if specified); and Append, that is similar to merge, except declared collections get appended to, instead of overridden.

Implementation and Usage

Merging is implemented using a special property of XML Serialization, that it works through assignment of public members of a class. This means that is a member is a nullable type—either System.Nullable<T> or inheriting from System.Object—then if the serialized xml does not provide that element it will be null, and otherwise it will be assigned a value.

This allows us to determine if a member was set in the definition xml or not. Which is the core of the merging system. Essentially we recursively traverse every object in the definition and where the base def specifies a value and the delta (the new definition) does not, we copy that over.

It is critical to note that this is done for every field in the object builder (be it public or private). This means that properties will not be ignored, but not invokes since the backing field will be modified directly. If an object builder needs more careful merging you can override the HandleOverride method of the definition handler.

In addition you can control this using two mechanisms:

• Attributes: There are to attribute you can use: NoMerge which causes a member to be ignored when merging; and FieldMerger that let's you specify the type of a custom merger class for the annotate field.
• Implementing IMyCustomObjectBuilderMerge: Implementing this interface on a type to be merged gives that type full control over how it is merged. Note that this can be used either on the definition itself or for any field therein.

Example

Lets take our example and see how we can make it merge friendly. Essentially what we have to do is take any fields that we want to be mergeable and make sure they are nullable:

public class MyObjectBuilder_ProjectileDefinition : MyObjectBuilder_DefinitionBase {
	public float? Deviation; // Here

	public float? Speed; // Here

	public string Model; // Already Nullable

	public TimeDefinition Timeout; // Here

	public ColorDefinitionRGBA ModelTint; // Here

	public string HitParticleEffect; // Already Nullable

	struct DamageEntry {
		[XmlAttribute]
		public string Material;
		[XmlAttribute]
		public float Amount;
	};

	[XmlElement]
	public DamageEntry[] DamagePerMaterial;
}

Of course any default values must be moved to Init(), and we now need to have them for every field in the definition since they all could be null.

Note that the DamageEntry struct was not modified, that is because it is only used as a member of a list, and as such we will never override members field by field, but only at the level of the list.

Since this list represents a dictionary we can leverage a custom list that handles merging more intelligently, the MyMergingList. This class relies on the list element equals method to make sure that when merging in append mode (and only then), any duplicates in the delta will replace the base definition, instead of being in the list twice. This is very handy especially for dictionaries, where we can craft the Equals and GetHashCode methods so that they only look at the key.

This is how it looks like:

struct DamageEntry {
	[XmlAttribute]
	public string Material;
	[XmlAttribute]
	public float Amount;

	public override bool Equals(object other) {
		if(other == null || other.GetType() != GetType())
			return false;
		return ((DamageEntry)other).Material == Material;
	}

	public override int GetHashCode() {
		return Material.GetHashCode();
	}
};

[XmlElement]
public MyMergingList<DamageEntry> DamagePerMaterial = new MyMergingList<DamageEntry>();

And so, given our example xml above a modder could override the damage of a specific material as follows:

<!-- Merge mode to to append! -->
<Definition xsi:type="MyObjectBuilder_ProjectileDefinition" Merge="Append">
	<!-- Definition to append to -->
	<Id Type="ProjectileDefinition" Subtype="Arrow" />

	<!-- Override flesh damage to 30 -->
	<DamageEntry Material="Flesh"	Amount="30" />
</Definition>

CopyFrom

Merging is also used for another handy mechanism which is the CopyFrom property of definition object builders, this allows the definition to specify that they are based on the data from another definition.

Copying can be controlled using the Copy attribute on the definition. This attribute supports the same values as Merge, but it defaults to Merge instead, and it does support merging, unlike the Merge attribute which is always taken from the delta definition.

Copy from can reduce substantially the amount of code required by a definition and avoid mistakes cause by forgetting to update similar definitions. As an example we have the character container definitions after CopyFrom was applied:

<Definition xsi:type="MyObjectBuilder_ContainerDefinition">
	<Id Type="Character" Subtype="Character"/>

	<Component Type="InventorySpawnComponent"/>
  </Definition>

  <Definition xsi:type="MyObjectBuilder_ContainerDefinition" Copy="Append">
	<Id Type="Character" Subtype="Animal" />

	<CopyFrom Type="Character" Subtype="Character"/>

	<Component Type="Inventory" Subtype="Animal"/>
	<Component Type="CharacterStatComponent" Subtype="Peasant_male" />
	<Component Type="CharacterSoundComponent" Subtype="Deer" />
  </Definition>

  <Definition xsi:type="MyObjectBuilder_ContainerDefinition" Copy="Append">
	<Id Type="Character" Subtype="Humanoid" />

	<CopyFrom Type="Character" Subtype="Character"/>

	<Component Type="CharacterRagdollComponent"/>
	<Component Type="MedievalCharacterUseComponent" Subtype="CharacterUse"/>
	<Component Type="Inventory" Subtype="Internal"/>

	<Component Type="EntityEquipmentComponent" Subtype="Humanoid"/>
	<Component Type="EntityStanceComponent" />
	<Component Type="CharacterHandItemsComponent" Subtype="Humanoid" />
	<Component Type="CombatComponent" Subtype="Humanoid" />
  </Definition>

  <Definition xsi:type="ContainerDefinition" Copy="Append">
	<Id Type="Character" Subtype="PlayableCharacter"/>

	<CopyFrom Type="Character" Subtype="Humanoid"/>

	<Component Type="CharacterControllerComponent" />

	<Component Type="CraftingComponent" Subtype="Humanoid"/>

	<Component Type="CharacterStatComponent" Subtype="HumanoidStats"/>

	<Component Type="EntityQuestComponent" Subtype="Humanoid" />

	<Component Type="AreaInventory" Subtype="Ground"/>
	<Component Type="AreaInventoryAggregate" Subtype="NearbyInventories"/>

	<Component Type="EntityStateComponent" Subtype="CharacterStances"  />

	<Component Type="QuickEquipComponent" Subtype="Humanoid"  />

	<Component Type="CharacterShapecastDetectorComponent" Subtype="Default" />
  </Definition>

<Definition xsi:type="MyObjectBuilder_ContainerDefinition" Copy="Append">
	<Id Type="MyObjectBuilder_Character" Subtype="Medieval_female" />

	<CopyFrom Type="Character" Subtype="PlayableCharacter" />

	<Component Type="CharacterSoundComponent" Subtype="MedievalFemale"/>
</Definition>

<Definition xsi:type="MyObjectBuilder_ContainerDefinition" Copy="Append">
	<Id Type="MyObjectBuilder_Character" Subtype="Medieval_male" />

	<CopyFrom Type="Character" Subtype="PlayableCharacter" />

	<Component Type="CharacterSoundComponent" Subtype="MedievalMale"/>
</Definition>