Reverse Engineering Lugaru assets

[rudel_ic]

Disregard this thread, it's all a waste of time anyway

---------------------------------------------
PROGRESS
.solid:
- Basic format of non-animated .solids is done
- First working converter is done: .x to .solid (works for specific meshes)
## RELEASE 0.2 ##
- Texture mapping works. Read this post for detailed instructions: http://wolfire.com/forum/viewtopic.php?p=54293#54293
- Weapons work perfectly fine! To find out how to make one, read this post
Dialogues:
Done, see here, search the forums
Campaigns:
Done, see here, search the forums

TODO
.solid:
- Format of animated .solids
All other assets (animations, skeletons, maps)



We'll start with the .solid format, simply because it is the simplest and most requested format.
A .solid file contains the description of a 3D model through its coordinates in 3D space, as one would expect.

I've analyzed the Box.solid, which contains the description for, what do you know, hold your breath, the box.

Turns out somebody from these forums actually got the format right:

Viking is still here, as am I :p
The format is:

Code: Select all

short vertexNum; 

short TriangleNum; 

float vertex[i].x, vertex[i].y, vertex[i].z; 
//where i is 0 to vertexNum-1 

int Triangles[i].vertex[0], Triangles[i].vertex[1], Triangles[i].vertex[2]; 
//where i is 0 to TriangleNum-1, and vertex[0] is the first vertex in
the triangle, vertex[1] the second, and so on

float Triangles[i].gx[0], Triangles[i].gx[1], Triangles[i].gx[2]; 
//where i is 0 to TriangleNum-1, and gx is the horizontal texture uv 

float Triangles[i].gy[0], Triangles[i].gy[1], Triangles[i].gy[2]; 
//where i is 0 to TriangleNum-1, and gy is the vertical texture uv

I can show you that it is the correct format by posting a simple screenshot:

That is the hexadecimal representation of the bytes in the file.
The datatypes zip mentioned are spot-on, but whether or not it is the correct interpretation remains to be seen.

So writing a .solid converter is NOT magic, it's just tedious! I am in the process of doing this, but if somebody is faster than me (I have a life), go ahead and use this information. It should be enough data to write a simple parser / converter.

To get to a format that a lot of 3D apps understand and that is still similar, it's of course important to choose wisely.

My choice is Microsoft's X format (for Direct3D models). It is very similar and extremely well-known; practically every 3D modeling suite supports it.[/size]

[Makrond]

PYTHON!!!

I feel random. Good job - sad that this is a little redudant though.

[rudel_ic]

Sadly, it turns out that at least some of these data types are not obvious representations of common C data types - an integer with the HEX value 0x3F800000 for example is either really big or below zero, depending on the interpretation. That doesn't work out for indices that are between 0 and 11.
Even if the order is swapped (0x0000803F), it's still too big. Something doesn't quite work out here.

[rudel_ic]

With something like

Code: Select all

file.read()*256*256*256+file.read()*256*256+file.read()*256+file.read();

to retrieve an integer the old-fashioned way, I end up with the following for the vertex indices:

Code: Select all

Vertex indices for triangle 0:	131073 65536 0
Vertex indices for triangle 1:	1605107712 855638016 1605107712
Vertex indices for triangle 2:	1605107712 1605107712 0
Vertex indices for triangle 3:	1 65539 196608
Vertex indices for triangle 4:	1605107712 855638016 855638016
Vertex indices for triangle 5:	0  1605107712  0
Vertex indices for triangle 6:	262144 3 196608
Vertex indices for triangle 7:	1605107712 1605107712 0
Vertex indices for triangle 8:	1605107712 1403781120 1403781120
Vertex indices for triangle 9:	5 327682 131072
Vertex indices for triangle 10:	0  1605107712  0
Vertex indices for triangle 11:	0 1605107712 1605107712

Apart from the fact that these numbers are definitely not indices, let's cue them up:

Code: Select all

1.  0
2.  1
3.  5
4.  65536
5.  65539
6.  131072
7.  131073
8.  196608
9.  327682
10. 855638016
11. 1403781120
12. 1605107712

So there ARE as many distinct indices as there are triangles, which is a good sign; the question still is how to interpret these suckers. The method of retrieval mentioned above doesn't cut it.

For further analyzing, here are the HEX representations for these numbers, cued up like above, with their distinct interpretations as described:

Code: Select all

1.  0x00_00_00_00 -> 0
2.  0x00_00_00_01 -> 1
3.  0x00_00_00_05 -> 5
4.  0x00_01_00_00 -> 65536
5.  0x00_01_00_03 -> 65539
6.  0x00_02_00_00 -> 131072
7.  0x00_02_00_01 -> 131073
8.  0x00_03_00_00 -> 196608
9.  0x00_05_00_02 -> 327682
10. 0x33_00_00_00 -> 855638016
11. 0x33_80_00_00 -> 1403781120
12. 0x3F_80_00_00 -> 1605107712

I will build this cube, assuming CCW vertex order, front-to-back, top-to-bottom, left-to-right and see whether or not the indices actually form a cube.

The coordinates as-retrieved:

Code: Select all

Coords for vertex 0:
			x:	-25.0
			y:	23.000006
			z:	-22.999994
Coords for vertex 1:
			x:	-25.0
			y:	-26.000006
			z:	25.999992
Coords for vertex 2:
			x:	-25.0
			y:	-25.999992
			z:	-23.000006
Coords for vertex 3:
			x:	-25.0
			y:	22.999992
			z:	26.000006
Coords for vertex 4:
			x:	25.0
			y:	23.000006
			z:	-22.999994
Coords for vertex 5:
			x:	25.0
			y:	-25.999992
			z:	-23.000006
Coords for vertex 6:
			x:	25.0
			y:	-26.000006
			z:	25.999992
Coords for vertex 7:
			x:	25.0
			y:	22.999992
			z:	26.000006

To make it simpler, I'll "normalize" the cube, meaning breaking it down to smaller numbers around zero (note that normalization is actually something else).
Here are the reinterpreted vertices:

Code: Select all

{
(-2,1,-1),
(-2,-3,3),
(-2,-3,-1),
(-2,1,3),
(2,1,-1),
(2,-3,-1),
(2,-3,3),
(2,1,3)
}

Let's see where those points end up in 3D space:

Well, there certainly is some potential for this to end up as a cube.
The vertices are located at the corners of the cube-like thing there; each corner is actually a translated, rotated and scaled Empty in blender.

The next thing I will do is rework the indices as they SHOULD have been, build the corresponding triangles and hopefully figure out how to interpret these damn integer-wide HEX values..

Stay tuned for progress.

If you can help, go ahead.

[David]

I was using http://onesadcookie.com/svn/libBinaryIO/ for low-level file reading and writing, which may have done something strange with the integer byte order.

[rudel_ic]

It's weird that you didn't use shorts in the first place for indexing because there can never be a number of vertices that exceeds the short width anyway.. Oh well.

[rudel_ic]

It can't be the byte order anyway; no matter what byte order is assumed, since there are HEX values like 3F and 80 in there, these values can't be integers if they're supposed to represent indices.
To be honest, I doubt that they index the vertices for triangles; I pointed out above that there are as many distinct indices as there are triangles, but it should be as many distinct indices as there are vertices.
Simply because each triangle is defined by its vertices, and there are no more than 8 of them. But there are 12 indices, as we can see.
It can't be triangles indexed for vertices either, because every vertex has 6 adjacent triangles in this case, not 3.
There has to be some weird wand-wiggling involved to make triangles out of those values.

Let's toss numbers around.
We have 144 bytes for indexing.
That would be 2*2*2*2*3*3.
Indexing vertices for triangles:
12 triangles * 3 vertex indices * 4 bytes for integer width = 144 bytes.
Indexing triangles for vertices:
8 vertices * (up to) 9 adjacent triangles * 2 bytes for short width = 144 bytes.

The latter variant is very obscure, but it's still a possible way to define triangles.
I still end up with weird superbig indices though. There must be some crazy stuff going on, like bitshifting or whatever. Still, it would be some crazy stuff that fixes values like 0x3F80 (or 0x803F or 0x3F800000 or 0x0000803F etc) and 0x0002 (or 0x0200 or 0x00020001 or 0x01000200 or 0x00010002 etc).

Short, this could be anything.

Thanks for the pointer to libBinaryIO, but the source I found there didn't contain any hints. There may be the solution to this in some ancient CVS repository.. I'll look into it.
Should be around 2003, right?

[WingedWolf93]

Woah, im not good at Hex editing and such, but i wish you all luck, making a .Solid editor would be hilarious!
If even it would also support a way to use bones, then we could get new characters! =D
Anyway, i hope to see some progress soon.
Wish luck!

[rudel_ic]

Thanks, .solids first though, animation is a real whopper, we'll see about that. Don't get your hopes up.

The whole point of this is getting to a reasonable art pipeline for modders, for example making a model in blender, exporting as X model, converting to .solid, replacing original .solid.

Thus, adding stuff is out of the question, of course, just replacing stuff will be possible that way.

Currently, I'm nowhere with the integers, so until that issue is solved, there will be no progress reports apart from my attempts at cracking it.

[Count Roland]

©øø? ?ˆ®?

[rudel_ic]

I haven't really done anything yet.. If I can't figure out what's up with these indices, there will be no .solid conversion tool. The indices are crucial.

[Makrond]

Something there doesn't add up... I can't put my finger on it.

I'm interested in your cueing up... were you arranging the indices in numerical order? If so, then you've missed one.

Regardless, I've seen computers in general do strange things. Looking at the original hex table, the 3F 80 values seem to be near 00 00 00 00 values - and from the little I know of hex, they could be type separators.

That may be absolutely no help to you whatsoever, you've probably noticed it yourself, but I thought I'd mention it anyway.

[rudel_ic]

I've missed two values actually: 262144 and 3. Thanks.

So we end up with 14 distinct values.

I'll look at the type separators idea, thanks for the hint. It is helpful.

Although we're not talking assembly here, this is just binary data without type description overhead. The types are most likely programmatically deducted.

But the 3F 80 and 33 80 must be something else than data. I'll run with that for a bit.

I had a disassembly session with Lugaru.exe, but that also didn't get me anywhere. I got to the point where Box.solid was referenced as a string, but couldn't find the fileread call in the referenced functions that ensued. I must have overlooked it, will probably have a go at it again.

[rudel_ic]

Makrond, actually, it turns out your observation reveals a pattern.

Look at the original HEX:

Now look at a certain deletion pattern:

The values that are left are actually pretty reasonable for integers. I think this could lead to an interesting conclusion. Is the 3F 80 / 33 80 stuff actually type definition overhead, meaning I got it wrong in the previous post?

We will see, I'll have to check out whether or not it adds up.

Makrond, that was a great move.

[rudel_ic]

Great, now we're in type definition hell.

Let's roll with the idea that 3fblabla defines types because THAT'S HOW WE ROLL!

First, we'll list them up in order:

Code: Select all

1.  3F 80 00 00 33 00 00 00 3F 80 00 00 3F 80 00 00 3F 80 00 00 00 00 00 00

2.  3F 80 00 00 33 00 00 00 33 00 00 00 00 00 00 00 3F 80 00 00 00 00 00 00

3.  3F 80 00 00 3F 80 00 00 00 00 00 00 3F 80 00 00 33 80 00 00 33 80 00 00

4.  00 00 00 00 3F 80 00 00 00 00 00 00 00 00 00 00 3F 80 00 00 3F 80 00 00

5.  3F 80 00 00 33 00 00 00 33 00 00 00 00 00 00 00 3F 7F FF FE 00 00 00 00

6.  33 00 00 00 3F 80 00 00 33 00 00 00 3F 80 00 00 00 00 00 00 00 00 00 00

7.  3F 80 00 00 00 00 00 00 00 00 00 00 3F 80 00 00 33 80 00 00 3F 80 00 00

8.  3F 80 00 00 00 00 00 00 3F 80 00 00 3F 80 00 00 00 00 00 00 00 00 00 00

9.  3F 80 00 00 3F 80 00 00 33 00 00 00 00 00 00 00 3F 7F FF FE 3F 7F FF FE

10. 3F 80 00 00 3F 80 00 00 33 00 00 00 00 00 00 00 3F 7F FF FE 3F 7F FF FE

11. 00 00 00 00 00 00 00 00 3F 80 00 00 00 00 00 00 3F 80 00 00 00 00 00 00

12. 3F 80 00 00 00 00 00 00 3F 80 00 00 00 00 00 00 3F 80 00 00 3F 80 00 00

Apart from 9 and 10, they are all distinct, if I'm not mistaken.

So it's obvious that these are not type definitions, there are simply not that many types.

Educated guess: We can abandon these bytes; they are overhead from the lowlevel file access API David used.

The cool thing is that it's trivial to find out when to skip these bytes upon streaming. All that has to be kept in mind is that they have to be hardcoded into the converter so that the new .solid still contains them.

What's left is figuring out what data there is without the overhead padding.

More later.