Instant Carma!

Utilities for editing Keen:Galaxy and Keen:Dreams levels.
Post Reply
User avatar
ckguy
Posts: 464
Joined: Tue Oct 14, 2003 11:20 am
Location: Providence, RI, US
Contact:

Instant Carma!

Post by ckguy » Mon Oct 29, 2007 11:02 pm

This is big news.

Instant Carma! is a new program I just wrote this afternoon. It only took me like two hours. How can a program written in such a short amount of time be big news? Because it does something I'm sure many of us have been waiting for for a long time: Separate carmackization.

That's right--this program implements fake carmackization, as set out in my post "Separate carmackization" 2 1/2 years ago. (I was really surprised that it was that long--how time flies!) It was fairly easy to write once I got down to it--the only problem was the usual impossible-to-find bugs that crop up in programs that translate one non-human-readable file into another non-human-readable file.

Anyway, stick instantcarma.exe in your ted5 folder and run it after ted5 and before ckpatch. This program removes the need for fixmhead, and for the goX.bat file. Check out readme.txt for info on the custmaps/gamemaps snare. Source included in instantcarma.exw.

This has a couple of effects on my other Keen projects: my level pack will probably speed up in development a bit, because I won't have to wait five minutes to test every little change. And the proposed Keen 4-6 level editor has been put on indefinite hold because TED5 is fine for my editing purposes.

Also (since I forgot to put these in readme.txt--they'll be there in release two if there is one, I promise) big thanks to Andrew Durdin for both the "Plumbing the Depths of Keen" article, and for the source of TEDSETUP.C, both of which were invaluable in writing this program.

And lastly, I'm sorry about the name of the program; I couldn't resist ... Image (I also couldn't resist that ... )

Have fun guys! (ShadowMaster, you know what to say ... )

User avatar
adurdin
Site Founder
Posts: 549
Joined: Fri Aug 29, 2003 11:27 pm
Location: Edinburgh, Scotland
Contact:

Re: Instant Carma!

Post by adurdin » Tue Oct 30, 2007 8:00 am

CK Guy wrote:That's right--this program implements fake carmackization
Brilliant!

User avatar
Shadow Master
Posts: 129
Joined: Fri Oct 20, 2006 3:14 pm
Location: Santiago, Chile
Contact:

Re: Instant Carma!

Post by Shadow Master » Tue Oct 30, 2007 12:32 pm

CK Guy wrote:That's right--this program implements fake carmackization
What the fake?
CK Guy wrote:Have fun guys! (ShadowMaster, you know what to say ... )
What should I say? Ah well, thanks for providing us the source. I guess I'll have fun figuring out how to translate that Euphoria code to standard C++. EDIT: It isn't difficult, I just needed the algorithm in a more plain format than TED5's source code.
EDIT2: your effort has given me inspiration to continue working on EModKeen 3. Although odds are your program will be merged into EModKeen source code, otherwise non-Windows users will not be able to enjoy my work. Do you mind if I publish it under the same BSD-style license of LModkeen 2?

levellass
Posts: 2902
Joined: Wed Oct 11, 2006 12:03 pm
Location: Ngaruawahia New Zealand

Post by levellass » Tue Oct 30, 2007 1:36 pm

Just asking here, is Ted5 supposed to produce custmaps files? Mine outputs GAMEMAPS.CK4 files every time, and all I have in my GoX batch files is the line to run CKxPatch -nowait.

EDIT: Ah, I see, 'fake' carmackization because it doesn't really carmackize anything, it leaves the maps uncompressed and writes a new map header file that lets the game use them. How convenient!

And it seems that GoX has a lot of needless stuff in it when it's set up, no wonder it ruins things. I now have GoX set up to run IC before CKPatch, all in one smooth movement. IC can even be run while Ted is still going in the background; this *will* be useful!

User avatar
ckguy
Posts: 464
Joined: Tue Oct 14, 2003 11:20 am
Location: Providence, RI, US
Contact:

Post by ckguy » Tue Oct 30, 2007 2:13 pm

ShadowMaster - you were supposed to say "This is god." Oh well. You caught the reference over at teh pckf, I don't know why you didn't get it here ...

Anyway fake carmackization: it doesn't actually compress the MAPTEMP files (it actually gets very slightly larger). Real carmackization compresses by replacing sequences of words that have appeared previously in the file file by a reference to them: How long ago was it, and how long was it?

In some cases, the bytes in the data stream might look like a reference back in the file; in these cases the compressor has to do something special to make sure the decompresser isn't tricked into thinking it is a reference back in the file. This is the only part of "compression" that my program does--the minimum possible amount of work to get MAPTEMP into a valid GAMEMAPS file.

So anyway, I'll at least tidy up the documentation for release two. The CUSTMAPS thing--I kind of added that in there at the last second. What I meant was that GOx.BAT copies GAMEMAPS to CUSTMAPS and then PATCHx.PAT has the line "%gamemaps custmaps.ckX".

I'm also thinking about rewriting the program in a more traditional (and popular) language so that more people can read it and understand it. And there'll definitely be some comments added.

[Edit, after having read the end of Shadow Master's post, which I don't know why I didn't notice before.] Sure, that would be great! I'd like it if you put in something along the lines of "it is based on a program written by CK Guy that was based on the work of Andrew Durdin." If you get tired of piecing the algorithm together, here it is:

Code: Select all

Write word xABCD to maphead
Set gamemaps_level_offset to 0
Loop level from 0 to 99
	Seek in mapthead to 294 + 4 * level
	Get dword from mapthead, store in maptemp_level_offset
	If maptemp_level_offset = -1 (xFFFFFFFF) then
		Write dword 0 to maphead
		Continue with next iteration of loop
	End if
	Set gamemaps_plane_offset to gamemaps_level_offset + 38
	Write dword gamemaps_level_offset to maphead
	Set gamemaps_level_length to 38
	Loop plane from 0 to 2
		Seek in maptemp to maptemp_level_offset + 4 * plane
		Read dword from maptemp, store in maptemp_plane_offset
		Seek in maptemp to maptemp_level_offset + 12 + 2 * plane
		Read word from maptemp, store in maptemp_plane_length
		Seek in maptemp to maptemp_plane_offset
		Seek in gamemaps to gamemaps_plane_offset
		Write word maptemp_plane_length to gamemaps
		Set gamemaps_plane_length to 2
		Loop maptemp_plane_length / 2 times
			Read word from maptemp, store in data
			Set high to data AND xFF00
			If high = xA700 or high = xA800 then
				Write word high to gamemaps
				Write byte data AND xFF to gamemaps
				Increase gamemaps_plane_length by 3
			Else
				Write word data to gamemaps
				Increase gamemaps_plane_length by 2
			End if
		End loop
		Seek in gamemaps to gamemaps_level_offset + 4 * plane
		Write dword gamemaps_plane_offset to gamemaps
		Seek in gamemaps to gamemaps_level_offset + 12 + 2 * plane
		Write word gamemaps_plane_length to gamemaps
		Increase gamemaps_plane_offset by gamemaps_plane_length
		Increase gamemaps_level_length by gamemaps_plane_length
	End loop
	Seek in maptemp to maptemp_level_offset + 18
	Seek in gamemaps to gamemaps_level_offset + 18
	Copy 20 bytes from maptemp to gamemaps
	Increase gamemaps_level_offset by gamemaps_level_length
End loop
This is actually not exactly what the original program did, but I did test out the changes and they work. The big difference is the lack of the "!ID!" signature dword at the end of each level header in GAMEMAPS before the plane 0 data starts. Also, the string "TED5v1.0" is not written in the beginning of GAMEMAPS (and was not in the original program either) but this does not affect the loading of the levels. So the output of this program is not 100% Plumbing the Depths of Keen-compliant, but Keen doesn't notice.

And lastly: LLa, have you gotten TED5 to work? The last time I heard about it, you were, um, a little bit angry at TED5. Has this been resolved?

[Edit, after having read the edit on LLa's post, which I'm not sure if it was there or not before.] Well, that's great! And running IC/Keen with TED5 still running in the background--that's a good idea, because then you only need for TED5 to decompress the graphics once per editing session.

[Edit.] One more thing. I should probably mention how my program fits in with Shadow Master's A tale of three programs... ModKeen2, TEDSetup and TED5. Put TED5, Keen, TEDSETUP, ModKeen2, CKxPatch, UNLZEXE, and my program all in the same folder. Decompress the Keen executable, run TEDSETUP. Now--never run TEDSETUP again. Export the graphics with ModKeen2. Edit them, import them. So from now on, there are a couple things to note.
  • You may safely run ModKeen2 to import maps whenever you want, and TED5 and Keen won't get confused.
  • After each time you run TED5, run Instant Carma!. In my TED5 folder, I have a batch file that runs TED5 with DOSBox and then runs my fake carmackizer.
At no point in the process is FIXMHEAD needed, because TED5 never carmackizes and never has the chance to output the faulty MAPHEAD file.

[Edit] To edit at maximum speed, have both TED5 and Keen running in DOSBox windows. After each TED5 edit, run IC. Then switch back over to Keen 5 and [W]warp back to the level. This does not work 100% consistently, but it does save a bit of time when it does work. Just restart Keen whenever it crashes, instead of before each edit. And you'll never have to restart TED5.

Hopefully that covers it. (Whew, long post, with many, many edits, most of which aren't even flagged with [Edit] before them.)
Last edited by ckguy on Wed Oct 31, 2007 4:56 am, edited 1 time in total.

User avatar
ckguy
Posts: 464
Joined: Tue Oct 14, 2003 11:20 am
Location: Providence, RI, US
Contact:

Post by ckguy » Tue Oct 30, 2007 9:07 pm

Release two uploaded. (Same place--see first post.) It's written in PureBasic, and the exe is 4.5k instead of over 50k. Make sure you check out my previous post, it's been edited many times, and in all likelihood, contains something it didn't last time you read it. :-)

levellass
Posts: 2902
Joined: Wed Oct 11, 2006 12:03 pm
Location: Ngaruawahia New Zealand

Post by levellass » Wed Oct 31, 2007 4:07 am

Real carmackization compresses by replaces sequences of words that have appeared previously in the file file by a reference to them:
'replacing' not 'replaces'; that can cause a bit of confusion there.


I didn't notice the compression first time around; but you're right, it does do a little packing, though the GAMEMAPS file ends up much larger than it would if it were fully compressed. It seems that not only can I run this with Ted5, but I can also compress the game maps while testing my changes in DosBox. How wonderful!

I think the ToTP post needs to be edited to fit a few new things in.

User avatar
ckguy
Posts: 464
Joined: Tue Oct 14, 2003 11:20 am
Location: Providence, RI, US
Contact:

Post by ckguy » Wed Oct 31, 2007 12:21 pm

More trimming of the output file:

You don't need to copy all of TED5's level names from from MAPTEMP to GAMEMAPS. So this is reflected two places in the algorithm above:

Code: Select all

Set gamemaps_plane_offset to gamemaps_level_offset + 38
becomes

Code: Select all

Set gamemaps_plane_offset to gamemaps_level_offset + 22
and

Code: Select all

Copy 20 bytes from maptemp to gamemaps
becomes

Code: Select all

Copy 4 bytes from maptemp to gamemaps
I'm becoming so obsessed with streamlining the output file ... maybe I'll even implement some real carmack compression. (Maybe.)

levellass
Posts: 2902
Joined: Wed Oct 11, 2006 12:03 pm
Location: Ngaruawahia New Zealand

Post by levellass » Wed Oct 31, 2007 12:40 pm

You wish to trim a few bytes from a 1/3 megabyte file?

I expect to see Carmac compression sometime next year.

User avatar
Shadow Master
Posts: 129
Joined: Fri Oct 20, 2006 3:14 pm
Location: Santiago, Chile
Contact:

Post by Shadow Master » Wed Oct 31, 2007 5:45 pm

CK Guy wrote:ShadowMaster - you were supposed to say "This is god." Oh well. You caught the reference over at teh pckf, I don't know why you didn't get it here ...
Ah yes. This is god. I just forgot that.
This is actually not exactly what the original program did, but I did test out the changes and they work. The big difference is the lack of the "!ID!" signature dword at the end of each level header in GAMEMAPS before the plane 0 data starts. Also, the string "TED5v1.0" is not written in the beginning of GAMEMAPS (and was not in the original program either) but this does not affect the loading of the levels. So the output of this program is not 100% Plumbing the Depths of Keen-compliant, but Keen doesn't notice.
Does it skip writing the signatures "!ID!" and "TED5v1.0" for some good reason, or is it just a lack of code?
[Edit.] One more thing. I should probably mention how my program fits in with Shadow Master's A tale of three programs... ModKeen2, TEDSetup and TED5. Put TED5, Keen, TEDSETUP, ModKeen2, CKxPatch, UNLZEXE, and my program all in the same folder. Decompress the Keen executable, run TEDSETUP. Now--never run TEDSETUP again. Export the graphics with ModKeen2. Edit them, import them. So from now on, there are a couple things to note.
  • You may safely run ModKeen2 to import maps whenever you want, and TED5 and Keen won't get confused.
  • After each time you run TED5, run Instant Carma!. In my TED5 folder, I have a batch file that runs TED5 with DOSBox and then runs my fake carmackizer.
At no point in the process is FIXMHEAD needed, because TED5 never carmackizes and never has the chance to output the faulty MAPHEAD file.

[Edit] To edit at maximum speed, have both TED5 and Keen running in DOSBox windows. After each TED5 edit, run IC. Then switch back over to Keen 5 and [W]warp back to the level. This does not work 100% consistently, but it does save a bit of time when it does work. Just restart Keen whenever it crashes, instead of before each edit. And you'll never have to restart TED5.

Hopefully that covers it. (Whew, long post, with many, many edits, most of which aren't even flagged with [Edit] before them.)
Feel free to add it to the sticky thread, I'm sorta busy right now, and need to try the program first, probably this weekend.

User avatar
ckguy
Posts: 464
Joined: Tue Oct 14, 2003 11:20 am
Location: Providence, RI, US
Contact:

Post by ckguy » Wed Oct 31, 2007 11:17 pm

Shadow Master wrote:Does it skip writing the signatures "!ID!" and "TED5v1.0" for some good reason, or is it just a lack of code?
Um. Shorter output file? You can do whatever you want with regard to these conformance issues in your implementation in EModKeen. Completely compliant GAMEMAPS files would be just as easy to write. (You could even output the compliant junk bytes after "TED5v1.0" and before the real info starts. [joke]) And updating the sticky topic, I'll do that next.

Post Reply