hacking file formats with python: PyFFI

[amorilia]

Hi everyone,

I'm one of the developers at niftools (see my signature, mirex maybe remembers me), and I accidently stumbled on your forum; very nice work you guys have been doing here!

I'm working on file formats myself as well, however since almost two years so I've taken a strong focus on interfacing them in order to create custom content, not just in the nif format, but in any block structured format that admits a simple enough (i.e. block structured) file format description.

In particular, besides the nif format, I've also decoded most of the cgf format for Far Cry and Crysis, helped by Crytek's mod sdk, and I managed to support both nif and cgf formats on Python via a single unified library called "PyFFI", which stands for Python File Format Interface. The reason that people here might be interested in this, is that it's extremely easy to add new formats to the library (the main difference between the nif part and the cgf part of the library is the file format description, which is written in xml, and this description format is very easy to learn). This is the main point of why I wrote PyFFI, and it turned out to work rather well. I found it useful for decoding formats as well, as it is easy to display analyzed data with it (in a nifskope-like fashion, if you'd be familiar with that).

So if anyone here is interested in interfacing file formats with minimal coding effort... yet another tool at your disposal. See my signature for the link. I hope it is useful.

I was also wondering how you guys went about interfacing new formats. I'd be happy to learn from fellow decoders how they go about this.

Cheers,
Amorilia

EDIT: not sure why but my signature is not attached, so here are the links:
http://niftools.sourceforge.net
http://pyffi.sourceforge.net

[mirex]

Hi Amorilia, welcome ! Is there some example of file format description in xml that is used to describe a 3d file with PyFFI ? I'd like to see that. I was looking through http://pyffi.sourceforge.net site but could not find it.

I can tell you how I did the reverse engineering. I usually just opened a file in hex-editor ( my favourite is hex editor in file manager Dos Navigator ) and I was looking around, trying to think out what is what. There is usually some header at start of file, some data look like a groups of something, some bytes look like 4 bytes of integer number, some look like a float numbers ...

Then when I had some rough image of what could be in file, I have written some convertor for it (or a importer into a Biturn) and I tried to conver the file, and see it in 3D ( if it was 3D model ) or 2D ( if it was bitmap ).

And then it was just about the tuning .. finding another values in the files, thinking about what is missing, and looking into file which parts of file are unidentified. Trying to convert different files of same format if they work allright or something does not.

Thats all how I do it. Cheers !

[amorilia]

Hi Mirex! Thanks for your reply. Here are some example xml files for a variety of formats:

http://pyffi.svn.sourceforge.net/viewvc ... iew=markup
http://pyffi.svn.sourceforge.net/viewvc ... iew=markup
http://pyffi.svn.sourceforge.net/viewvc ... iew=markup
http://niftools.svn.sourceforge.net/vie ... iew=markup

The last two are for 3d formats. NIF is used in oblivion, fallout 3, civ4, ... CGF is used in Far Cry and Crysis.

I guess where pyffi would help is writing a converter, since everything is almost automatic for reading/writing files, and also visualizing the data through qskope (if you know nifskope, then qskope is the same but for arbitrary file formats) as soon as you have an xml description and a fairly small bit of extra code.

Here's a quick example of actual interface code for a simple format:

http://pyffi.svn.sourceforge.net/viewvc ... iew=markup

For the nif and cgf formats the reader and writer code is of course a bit more complicated, but the main ideas are just the same.

Cheerio,
Amorilia

[amorilia]

Well, if anyone ever stumbles on this, pyffi has moved from svn to git, so the links above are broken; here is the link to the new git repostory where you can find the xml and examples:

http://github.com/amorilia/pyffi/tree/master