Unlzexe

Tools, assembly, and file formats.
Post Reply
levellass
Posts: 3001
Joined: Wed Oct 11, 2006 12:03 pm
Location: Ngaruawahia New Zealand

Unlzexe

Post by levellass »

Currently none of my programs work with LZ compressed executables, they need unlzexe to decompress them first. I'd like to change that by adding a snippet of code to decompress things.

I am assuming that all Keen executables follow the same plan, a decompressor, followed by a chunk of data that is decompressed to the working executable/ I'm assuming the decompressor to be the same in all cases (Or nearly so.) and that the LZ compresssion works similarly to that employed by Keen 1 for its graphics.

Before I bother to actually poke around and possibly waste time, are these assumptions correct? (I doubt all of them are.) and if not, what changes will I have to make before I can decompress the executables correctly? (I am of course especially interested in the compression method itself.)


I'm looking your way Lemm!
User avatar
Malvineous
Posts: 113
Joined: Sat Mar 13, 2004 12:54 am
Location: Brisbane, Australia
Contact:

Post by Malvineous »

The C source code for a few versions of unlzexe is readily available if that helps. This isn't a recent one, but it might provide a few pointers as to what it's doing.

I had thought of doing the same thing for my modding tools, but unfortunately you have to deal with the structure of .exe files with the unpacker. I think this is because it's not just one .exe compressed inside another, but portions of code compressed inside an .exe - so when you decompress those portions of code (which are designed to be written directly to memory and immediately executed), you have to then rearrange them back into a valid .exe file again yourself.

If you're using QuickBASIC for your programs, perhaps it would be easier to convert one of the unlzexe versions into a library, so you can call it from your own code just like any other sub. That way you could detect whether the source file is compressed, and if it is then decompress it before proceeding with the rest of the code. Of course it might be just as easy to port unlzexe to BASIC in that case and make your own library written entirely in BASIC that you can reuse in your modding tools.
levellass
Posts: 3001
Joined: Wed Oct 11, 2006 12:03 pm
Location: Ngaruawahia New Zealand

Post by levellass »

The c code is... not quite gibberish to me, but close. I'd much prefer there to be a page somewhere describing the decompression, possibly on modding wiki. That way understanding it would not be dependent on understanding any specific language.

I do know that 'A good source of inspiration was the program EXEPACK from Microsoft which did a very simple compression on the executables. I used almost the same scheme. The main point was to decompress the executable while not using more memory than its decompressed size. It is not so easy because the compressor must guaranty that at each time during the decompression the compressed and decompressed parts do not overlap. ' (From the LZEXE homepage.) The moddingwiki's page is rather sparse: http://www.shikadi.net/moddingwiki/LZEXE

The whole segments thing shouldn't be a problem from what I hear, it's only that way to cut down on how much memory is needed for decompression. Can you possibly tell me what it is the c code does? Because I have very little idea at present.
User avatar
Malvineous
Posts: 113
Joined: Sat Mar 13, 2004 12:54 am
Location: Brisbane, Australia
Contact:

Post by Malvineous »

Unfortunately I'm not familiar enough with compression or .exe files to be able to help very much. The decompression algorithm seems highly optimised, which doesn't make it easy to understand at the best of times.

Having said that it doesn't look quite as bad as I thought. It seems the .exe header is stored uncompressed at a different offset, then the rest of the original .exe is compressed in its entirety, with no attention paid to its structure. I'm not sure how the embedded decompressor handles things, but it looks like all unlzexe has to do is copy out the original header into a new .exe then append the decompressed data and that's it.

Maybe you could start by trying to copy what the mkreltbl() function does, which is to recreate the original .exe header and see if you can get it to mimic the real unlzexe output? Once that's done then you can try adding LZ decompression into the mix.

gerstrong might also be able to help, as he ported unlzexe to C++ recently so probably has a better idea of what's going on.
levellass
Posts: 3001
Joined: Wed Oct 11, 2006 12:03 pm
Location: Ngaruawahia New Zealand

Post by levellass »

Excellent; how may I contact him, do you know?
gerstrong
Posts: 63
Joined: Sun Jan 25, 2009 3:21 pm

Post by gerstrong »

Looking for me? ;-)

I'm not out of the world...

Regarding this

http://clonekeenplus.git.sourceforge.ne ... pp;hb=HEAD

I'm not sure if it's the best code, but a recent adaptation even tested on portable devices. It is very fast also, because the compressed file is loaded to the memory, unpacked right there and read.

As it is for CG I never write to uncompressed data to a file, but it is possible and wouldn't be very hard.

What can I explain you here?

Why don't you write me an email, I don't come in here that often, so my response could be delayed.

Kind regards
levellass
Posts: 3001
Joined: Wed Oct 11, 2006 12:03 pm
Location: Ngaruawahia New Zealand

Post by levellass »

I'm not sure I know what your email address is.

Basically I need to know how the LZEXE executable is compressed that I might write my own code to decompress it. (It will probably be decompressed into memory, but that's neither here nor there.) Your code looks nifty, but I am no good at understanding what it actually entails, my grasp of it is very poor indeed.
gerstrong
Posts: 63
Joined: Sun Jan 25, 2009 3:21 pm

Post by gerstrong »

So tell me what you don't understand so I can explain it... or is it the entire Code you don't understand. how much experience do you have with C/C++?

Please let me help you
levellass
Posts: 3001
Joined: Wed Oct 11, 2006 12:03 pm
Location: Ngaruawahia New Zealand

Post by levellass »

very, VERY little experience with C and C++, and no ability at all to translate from code to English. I can see vaguely what it does, but not nearly clear enough to see how I should translate it into other languages. (So yes, entire code.) Even a rough explanation of what it does would be of great benefit to me.
gerstrong
Posts: 63
Joined: Sun Jan 25, 2009 3:21 pm

Post by gerstrong »

okay, I sent a PM on PCKF.
Post Reply