M2TW Modeler's Toolbox - Version 0.6 Beta Released On 06/06/2015

[Zarathos]

Ladies and gentlemen, I'm proud to announce you the release of my M2TW Modeler's Toolbox.



INTRODUCTION & OVERVIEW

1) What is it?

It's a brand new tool, developed from scratch by me, that allows modders to work easily on both models (it can convert M2TW meshes to MS3D models and vice-versa or merge MS3D models) and textures (it can convert M2TW texture to DDS format and vice-versa or write UV mappings to DDS files).

2) Nice, but why shouldn't we just keep on using GOAM, GOAT and other existing tools?

M2TWMT has many, many advantages:

• Application: it has been developed under .NET Framework, so it doesn't require Python to be installed anymore and it's much, much faster than any similar program based on Python's Framework.
• Interface: unlike other tools, it offers a modern-looking and very user-friendly interface, simple and intuitive even for the newbies (no command-line operations and consoles).
• Basic Functions: it can process both single files by target and many files at once by selecting a folder. It's the only existing tool out there (as far as I know) that correctly handles:
  
  • bones and bone assignments allowing modders to obtain the real skeleton of any model without the original skeleton files;
  • bounding spheres of M2TW meshes;
  • non-standard models that can be found in M2TW Kingdoms' campaigns.
  
  
• Advanced Functions: it implements an Extraction Tool that can extract models and textures directly from a mod's EDU file and a Packing/Unpacking Tool that allows the manipulation of animations and skeletons pack archives taking care of duplicates.
• Validation & Errors: it implements very complex algorithms capable of fixing seam normals and unassigned/underweighted vertices during conversion and merging operations and a validation algorithm for validating MS3D files before converting them to MESH format in order to avoid bad surprises. On the top of that, it implements errors handling and reporting functions that allow modders to easily and quickly understand what's going wrong with their files during any operation.

TUTORIALS

Spoiler Alert, click show to read: 

1) Does this application offer any keyboard shortcut?

Sure! Here is a small list of the implemented keyboard shortcuts:

• CTRL + Left: shows the previous tool.
• CTRL + Right: shows the next tool.
• CTRL + X: closes the application.

2) What are the various options for?

Here is a small overview of that each of them does:

• Rebuild Skeletons: if enabled, every time you convert a MESH model to MS3D format, the application will ask you if you wish to rebuild its skeleton selecting a skeleton file.
• Show Warnings: if enabled, the application will show every warning message. Although this feature it really imporant new users, I recommend you to disable it after a few usages.
• Topmost: if enabled, the application will always stay on the top of your desktop. I recommend you to use this feature in combination with the next one if you really want to prevent the application's form to be overlapped by other applications' windows.
• Transparency: if enabled, the application's window will become semitransparent.
• Use File Suffixes: if enabled, this option will append the ".converted" suffix to every file resulting from a convertion operation providing an additional security layer over your work against files overwriting and helping you identify the application's output quicker and easier. Some suffixes are always appended to output files regardless of the value of this option.

3) How to work with multiple files?

To switch from single target mode to multiple targets mode (and vice-versa, of course), all you have to do is to press the SHIFT button. After doing that, the suffix "(All)" will appear on the buttons' text meaning that you can execute operations on every file contained in the selected directory. The application will process every file found in the top-level directory only, so subdirectories will be ignored. For what concerns the Merge MS3D function, the first file you select will be the primary one to which every other MS3D model found in the top-level directory will be merged.
It's very important to remember that, in multiple targets mode, there will be no checks and confirmation requests about files overwriting so... be careful because this can make you lose all of your previously reworked files!
It's also important to remember that, at present, processing multiple MESH to MS3D conversions will not allow you to rebuild original models' skeletons. Anyway, rebuilding skeletons is only useful if you need to work with animations or to work with bone assignments; skeletons are, in fact, totally unnecessary for what concerns the simple process of models editing.



4) How to rebuild original skeletons?

The original models' "base pose" skeletons can be found into the extensionless files that can be found inside /data/animations/skeletons.dat. So, to rebuild original skeletons you first have to unpack that file using this tool (update its executable files with those ones which are more recent and contain a lot of bug fixes). In order to do this you have to unzip the downloaded archive in your /data/animations/ directory and then run extract_skeletons.bat​:





Now, let's say you want to convert Wardog's MESH to MS3D having its skeleton correctly rebuilt. All you have to do is to look at its /data/unit_models/battle_models.modeldb entry to see what is the primary animation that it's using and then, when asked by the application if you want to rebuild its skeleton, just answer "Yes" and select the corresponding extensionless skeleton file:







Here is the difference between the same MESH converted to MS3D using this tool with rebuilt skeleton (it looks exacly how it was made by its author) and GOAT/GOM (they give to the model a default premade skeleton based on what they think it should be and then they re-rig every vertex to the current existing bones, sometimes even leaving some of them unassigned):



If you convert any MESH with my tool and without rebuilding its skeleton, you will notice that in the corresponding MS3D there will be no bones hierarchy defined and all the bones will be placed at position (0, 0, 0) with a rotation of (0, 0, 0). Once again (let me repeat it because it's really important), in 90% of cases, having a correct skeleton for editing a MESH is almost irrelevant and will not affect the unit's behavior on battle map. It's only suitable to have it when you are working with animations, massive joints rigging (using Milkshape's "Joints Tool", for example) and resizing/scaling (it's better to see how the model will work with its skeleton after modifying its size or to modify its skeleton according to its new size).
Here is the result of such operation on the Wardog model took as example:



If you convert non-conventional M2TW models (creatures, humans, mounts or siege engines different from vanilla ones because of their shape or the presence of additional bones) to MS3D with GOAT or GOM in order to edit them, once they are put back into the game all you can obtain is bad results (bugged or malformed units, broken or weird-looking animations or, in the worst case scenario, crashes).
I'll make a clearer example: if you edit an existing archer's model with this bowstring animation using GOAT or GOM as conversion tool, after putting it back into the game you will immediately notice that its bowstring is not moving anymore even if you didn't touched its bow. Why? Because neither GOAT nor GOM can take into account the existance of two additional bones needed to the bow animation (one makes the bow bend and the other one moves the bowstring): they just assign to your model a default human bone re-rigging all of the bow vertices to the nearest bone or leaving them unassigned. This is what my tool can avoid!

TECHNICAL DETAILS

Spoiler Alert, click show to read: 

1) About Non-Standard Models

Their format wasn't really that hard to crack. I don't know what's the reason for it, but in fact their binary format is a mix between siege engines (type 0) and regular models (type 1). All you have to do to extract them properly is to read them like if they were siege engine models until you reach the end of their vertices' section and then, upon verifying that their variant value is 23, you read their vertices data sections using the regular models' approach (pseudocode):

/* Here you have the common approach for reading model's vertices... */

Int32 verticesCount = binaryReader.ReadInt32();

for (Int32 i = 0; i < verticesCount; ++i)
     vertices[i] = binaryReader.ReadVertexM2TW();

binaryReader.SkipBytes(18);

// Here is where you can discover that they are non-standard models and change their type to regular in order to properly read the remaining bytes... */

Byte variant = binaryReader.ReadByte();

if (!isRegular && (variant != 23))
     isRegular = true;

/* And here you keep on reading remaining vertices data using a different approach for type 0 and type 1 models... */

if (isRegular)
{
     binaryReader.SkipBytes(1);
     tangentSpaceFlags = binaryReader.ReadInt32();
     binaryReader.SkipBytes(2);
}

2) About Skeletons' File Format

How did I discovered that models' "base pose" data was still laying hidden somewhere in M2TW files? Well, I started suspecting it the day I messed up by mistake a model I was working on (I forgot to create a group's comment) and, after putting it back into the game, I stumbled upon a so called "Jesus Syndrome": the model was standing motionless on the battle map with a position that looked to be exactly the default human models' position.
So I started looking for this data inside CAS files that handle models' animations (in fact, position and rotation of bones during a given time interval). The problem is that, in M2TW, there are many different types of CAS files and every CAS file contains data for a different animation, except for the default pose, apparently. I would also have been a real nightmare for me to find a way to standardize their analysis and to make my application select the right one for skeleton rebuilding process.
That's why I started analyzing skeleton files and, with some luck, time and a good hex editor, I found what I was looking for:

Byte[4] -> Header Chunk 1
UInt16 -> BonesCount
Byte[3] -> Header Chunk 2

Then, the following structure for BonesCount times:

Byte[5] -> Header
Float -> Position X
Float -> Position Y
Float -> Position Z
Byte -> Parent ID
Byte[59] -> Unknown Data
Null-Terminated String -> Name

And now a little bit of pseudo code:

binaryReader.SkipBytes(4);

UInt32 bonesCount = binaryReader.ReadUInt16();
Bone[] bones = new Bone[bonesCount];

binaryReader.SkipBytes(3);

for (Int32 i = 0; i < bonesCount; ++i)
{
     binaryReader.SkipBytes(5);

     Bone bone = new Bone();
     bone.X = binaryReader.ReadFloat();
     bone.Y = binaryReader.ReadFloat();
     bone.Z = binaryReader.ReadFloat();
     bone.ParentID = binaryReader.ReadByte();

     binaryReader.SkipBytes(59);

     String boneName = "";

     while (binaryReader.PeekChar() != 0)
          boneName += binaryReader.ReadChar();

     bone.Name = boneName;

     bones[i] = bone;
}

3) About Bounding Spheres

First of all, if you want to know what a bounding sphere is, read this (a short definition, for what concerns 3d modeling, could be the following: "given a non-empty set of points of finite extension in a 3-dimensional space, the bounding sphere for that set is the smallest 3-dimensional solid sphere containing all its points").
Here is an example of a MESH bounding sphere data (after the conversion to MS3D format, you can read them in Milkshape from Model Tab -> Comment):

-0.01930268 ; X Offset
0.2401515 ; Y Offset
0.4286239 ; Z Offset
1.331218 ; Radius
/

In M2TW engine, they have many different usages:

• they are used to interrupt charges;
• they make soldiers aware of their position on the battlefield and into the unit formation itself;
• they make soldiers aware of the proximity of enemy soldiers so that they know when one of them is close enough to be reached and hit by their weapons;
• they works as helpers for collision calculations especially for what concerns siege engines' projectiles.

The most common symptoms of wrong bounding spheres are:

• charges are interrupted too close or too far from enemy units;
• CTDs under certain circumstances (especially during siege battles);
• reinforcement units are stuck along the battlefield map's border and can't be moved anymore;
• soldiers perform hitting animations but the edge of their weapons is not touching enemies' body (this is particularly noticeable when they use lances, pikes and spears).
• the spacing of unit formations is incorrect and soldiers move in a silly way while repositioning.

After discovering how bounding spheres are calculated for M2TW meshes (a variant of the Ritter's algorithm), I can now say for sure that 99,9% of the custom models created to this day have wrong enclosures that can potentially lead to one or more of the above listed problems. Since creating custom models for M2TW became possible, modelers started using ready-made values for their new models taken from existing Vanilla M2TW units (from a similar archer's model for a new archer unit, from a similar pikeman's model for a new pikeman unit and so on)... an approach that was probably enough to obtain a working bounding sphere, but that was also producing bounding spheres completely unrelated to the real models themselves.

REQUIREMENTS

The application requires .NET Framework 4.0 to work under Windows operative system. Otherwise, it requires the latest version of Mono to run properly (you can find it here).



DISCLAIMERS

• The application is currently in beta stage of development and this means it could crash or produce unexpected results. Use it at your own risk, properly test its output before including it into your projects and report to me any bug or problem you find during its usage.
• Under no circumstances is this tool permitted to be used for the taking of other mods' material for public release unless you have permission from the creator of the content. Use it responsibly and have fun modding M2TW!

CHANGELOG

Spoiler Alert, click show to read: 

From 0.5 Beta to 0.6 Beta

• Added an experimental Cas Conversion function that allows to convert stratmap CAS files to MS3D and vice-versa.
• Added the Fix Bounding Spheres function that allows to quickly fix the bounding spheres of all the models contained in the targeted folder recursively.
• Cleared, optimized and refactored/reorganized the source code in order to improve the overall performance of the application.
• Fixed major and minor bugs.
• Improved the reporting system to provide maximum detail information concerning errors.
• Redesigned the whole interface with function grouped by type on different panels.

From 0.4 Beta to 0.5 Beta

• Added a few keyboard shortcuts.
• Added even more detailed error descriptions in order to allow a better understanding of the occurring problems and their causes.
• Fixed major bugs concerning the conversion of models without file suffixes.
• Fixed minor bugs.
• Improved the MS3D To MESH function so that MS3D models are converted to M2TW format with a correct bounding sphere.
• Redesigned the interface to produce a much more compact application.

From 0.3 Beta to 0.4 Beta

• Added the Packing/Unpacking function that allows modders to easily work with pack files containing animations and skeletons avoiding file overwritings.
• Cleared, micro-optimized and reorganized the source code in order to improve the overall performance of the application.
• Downgraded the application to Framework 4.0 for compatibility requirements towards Windows XP users.
• Fixed major bugs concerning the conversion of models.
• Fixed minor bugs.

From 0.2 Beta to 0.3 Beta

• Added various Options and Settings that allow users to customize the application's behaviors and interface for a better usage experience.
• Applied cosmetic changes to the user interface.
• Cleared, micro-optimized and reorganized the source code in order to improve the overall performance of the application.
• Fixed minor bugs.
• Improved the Extraction function.
• Improved the parsing functions of M2TW data files.
• Improved the integrity check of the conversion operation from MS3D to MESH.
• Improved the management of selected files and folders.

From 0.1 Beta to 0.2 Beta

• Added an integrity check to the conversion operation from MS3D to MESH in order to avoid the creation of M2TW models with invalid groups or unassigned/underweighted vertices.
• Added more detailed error descriptions in order to allow a better understanding of the occurring problems and their causes.
• Added the Extraction function that allows to quickly export models and textures directly from mod folders.
• Added the support for multiple files to the Merge MS3D function.
• Added the support for Photoshop PSD texture files to the Merge MS3D function.
• Fixed minor bugs.
• Improved the management of selected files and folders.
• Increased the performance of files reading and writing operations.
• Increased the safety of every operation in order to avoid unhandler exceptions.
• Modified the writing operation of UV mappings in order to output them coloured by group with a beautiful rainbow effect.

TERMS & CONDITIONS

This tool is free to use and permission to publish files converted/modified with it is not required, but is denied if the author is not willing to adopt a free-use policy on his own creations.

Like a very wise person said:

I am of the belief that all modding should be open and free to use by anybody in any way they choose. After all, we're all building on someone else's work or shared knowledge to some degree, directly or indirectly. If your attitude is one of "anything I make is not for anybody else to reuse" then fine, that's your right, but then you should not be reusing anybody else's work either, in my opinion. If you're absorbing other people's work to improve your mod then let others absorb your work to improve theirs.

I am sharing my knowledge and my skills with this community, and so should you. Since in this community the only way to react to narrow-mindedness is to set up even more narrow-minded policies, here is the deal: accept this rule or keep on using GOAT/GOM. I'm placing control signatures in the unused bytes of every single file format that can be edited with my tool so, no matter what you do, I can always know whether your files have been converted/modified with my tool or not... and if you are restricting/denying permissions to someone I will instantly report the abuse.

On the top of that, forget the permission to use this tool if this guy is actively collaborating with you and/or your team or is still a major contributor of your mod/submod/compilation.