Writing a level loading function; trouble with decompression

Tools, assembly, and file formats.
Post Reply
jigglez
Posts: 4
Joined: Sat Jun 27, 2009 6:51 pm

Writing a level loading function; trouble with decompression

Post by jigglez »

Hi everyone,

I'm new, nice to meet you all. I had to jump through some hoops to get registered since I don't know much about keen (blasphemy!)

Anyways, I am writing a level loading func for Wolf3D which uses the same file format as Keen for levels.

I am having trouble un-carmikizing and/or un-RLE'ing (can't figure out which one is wrong)

Specifically, after unRLEW'ing the files, the uncompressed data is huge, much larger than the total length of a single plane array == 64*64*2 == 8192 bytes, for a 2-byte word.

Thanks for any possible help.



Here are the results I get:

File == w38.map from Wolf3D

MAP NAME: Wolf4 Boss
MUSIC NAME: music/WARMARCH.ogg
ROW WIDTH (w): 64
NUM ROWS (h): 64
CEILING COLOUR: 56 56 56 0
FLOOR COLOUR: 112 112 112 0
LENGTH ARRAY FOR PLANES, IN WORDS: 705 370 10
OFFSET ARRAY FOR PLANES, IN BYTES: 77 782 1152
RLE TAG: 43981
PLANE1 BYTE SIZE (UNCOMPRESSED): 48994
PLANE2 BYTE SIZE (UNCOMPRESSED): 6588
PLANE3 BYTE SIZE (UNCOMPRESSED): 4096
jigglez
Posts: 4
Joined: Sat Jun 27, 2009 6:51 pm

Post by jigglez »

Here are some details for my decompression, for the same map file w38.map:

Its hard to believe that a word would be needed to be repeated 0xABCD (the RLE tag) times, as in this:

DECOMPRESSING... CUR_INDEX = 1179, COUNT = 43981, VALUE = 9



** Decompression output****


@@@ TESTING CARMACK COMP/DECOMP
SIZE OF compressed_data_byte_length = 1410
SIZE OF temp_data = 1083
DONE DECOMPRESS!
SIZE OF out_data_array = 2486
@@@ TESTING RLEW COMP/DECOMP
EXPECTED SIZE TO DECOMPRESS = 8192
SIZE OF compressed_data_byte_length = 288
SIZE OF in_byte_array = 2484
DECOMPRESSING... CUR_INDEX = 37, COUNT = 6, VALUE = 1
DECOMPRESSING... CUR_INDEX = 93, COUNT = 13, VALUE = 109
DECOMPRESSING... CUR_INDEX = 101, COUNT = 29, VALUE = 1
DECOMPRESSING... CUR_INDEX = 133, COUNT = 8, VALUE = 109
DECOMPRESSING... CUR_INDEX = 173, COUNT = 31, VALUE = 1746
DECOMPRESSING... CUR_INDEX = 225, COUNT = 45, VALUE = 35
DECOMPRESSING... CUR_INDEX = 267, COUNT = 45, VALUE = 48
DECOMPRESSING... CUR_INDEX = 435, COUNT = 11, VALUE = 1458
DECOMPRESSING... CUR_INDEX = 777, COUNT = 17, VALUE = 120
DECOMPRESSING... CUR_INDEX = 1023, COUNT = 173, VALUE = 3284
DECOMPRESSING... CUR_INDEX = 1099, COUNT = 4, VALUE = 8
DECOMPRESSING... CUR_INDEX = 1105, COUNT = 44, VALUE = 274
DECOMPRESSING... CUR_INDEX = 1117, COUNT = 11, VALUE = 1458
DECOMPRESSING... CUR_INDEX = 1125, COUNT = 36, VALUE = 37
DECOMPRESSING... CUR_INDEX = 1179, COUNT = 43981, VALUE = 9
DECOMPRESSING... CUR_INDEX = 1555, COUNT = 2533, VALUE = 99
DECOMPRESSING... CUR_INDEX = 1641, COUNT = 48, VALUE = 0
DECOMPRESSING... CUR_INDEX = 1649, COUNT = 6, VALUE = 4
DECOMPRESSING... CUR_INDEX = 1667, COUNT = 138, VALUE = 0
DECOMPRESSING... CUR_INDEX = 1675, COUNT = 37, VALUE = 9
DECOMPRESSING... CUR_INDEX = 1731, COUNT = 6, VALUE = 3
DECOMPRESSING... CUR_INDEX = 1755, COUNT = 26, VALUE = 0
DECOMPRESSING... CUR_INDEX = 1817, COUNT = 5, VALUE = 48
DECOMPRESSING... CUR_INDEX = 1823, COUNT = 415, VALUE = 48
DECOMPRESSING... CUR_INDEX = 1911, COUNT = 34, VALUE = 0
DECOMPRESSING... CUR_INDEX = 1959, COUNT = 5, VALUE = 48
DECOMPRESSING... CUR_INDEX = 1965, COUNT = 15, VALUE = 2211
DECOMPRESSING... CUR_INDEX = 2019, COUNT = 23, VALUE = 0
DECOMPRESSING... CUR_INDEX = 2027, COUNT = 38, VALUE = 0
DECOMPRESSING... CUR_INDEX = 2067, COUNT = 9, VALUE = 3
DECOMPRESSING... CUR_INDEX = 2197, COUNT = 7, VALUE = 12
DECOMPRESSING... CUR_INDEX = 2207, COUNT = 46, VALUE = 0
DECOMPRESSING... CUR_INDEX = 2301, COUNT = 7, VALUE = 164
DONE DECOMPRESS!
SIZE OF raw_data = 48994
User avatar
adurdin
Site Founder
Posts: 549
Joined: Fri Aug 29, 2003 11:27 pm
Location: Edinburgh, Scotland
Contact:

Post by adurdin »

jigglez wrote:I'm new, nice to meet you all. I had to jump through some hoops to get registered since I don't know much about keen
Good to have you here. I'm glad you were able to get past the bot filter ok :)
I am having trouble un-carmikizing and/or un-RLE'ing (can't figure out which one is wrong)
...
Its hard to believe that a word would be needed to be repeated 0xABCD (the RLE tag) times
Yes, obviously something just before has not decompressed correctly, leaving you reading the tag when you expect a count.

It's possible your carmack-decompression is faulty. You can easily test this by comparing the expected decompressed length (that's stored just before the compressed data) with the actual length after you've carmack-decompressed it, but before any RLE decompression. If they're different, the carmack-decompression is faulty.

I'm not able to try decompression the wolf3d files myself to compare what I get with your results; however if you want a known-good carmack and RLE decompresser, grab the modkeen2 source from http://files.keenmodding.org/modkeen2.zip and extract the relevant functions from it. If you know C, it should be trivial to do a test run using the code there and compare with your results. Of course, the Wolfenstein 3D source code itself would be another good comparison.
jigglez
Posts: 4
Joined: Sat Jun 27, 2009 6:51 pm

Post by jigglez »

ahh thanks for that information and advice. I am actually writing the code in Ruby, and comparing it to the source from the orig. Wolf 3D (actually the iphone one :D )


you're right, in fact the carmack headers for size after un-carmacking dont match -- i wasnt sure if they were supposed to or not, since I didnt have anyone to ask (and i didnt feel like ripping out those functions :P )

but since you know they are supposed to match, i must be doing something wrong :P

will be back once i fix it :P

however, i am pretty sure i checked it a few times and against the original C code too. well, that is the nature of bugs.. :D
levellass
Posts: 3001
Joined: Wed Oct 11, 2006 12:03 pm
Location: Ngaruawahia New Zealand

Post by levellass »

I myself never both reading the count for things like this, when I uncarmackize I simply read data until there's no more left (Same with all RLE I do.) Can someone tell me if this is actually necessary? (I can understand the executable, etc doing it as it can help to know how big the file(s) you're dealing with are, but for just plain decompression, does this serve any purpose?)
jigglez
Posts: 4
Joined: Sat Jun 27, 2009 6:51 pm

Post by jigglez »

I don't have enough experience to be able to tell you if its necessary to check the counts or not. *however* -- having looked at the decompress functions as they were written for Wolf3D, they used the byte counts in both carmack and rle to determine when to stop the decompression. It didn't seem necessary to me either..


Also - i fixed my bug :D i got bored of the proj due to the bug then came back and after many hours, found that I was unpacking the original data from the .map file in Ruby with "C" (unsigned) instead of "c" (signed). Yurghhhh
User avatar
adurdin
Site Founder
Posts: 549
Joined: Fri Aug 29, 2003 11:27 pm
Location: Edinburgh, Scotland
Contact:

Post by adurdin »

Levellass: the main reason for needing to know the size up-front is so you know how much memory the uncompressed data will take up. Obviously this is more important in languages like C with manual memory management, but it can make any implementation more efficient.
The other reason to check it is as a safeguard: if the data file has been corrupted, you might well end up writing much more (bogus) data than expected (or much less). In C, this would result in a buffer overflow, and you'd probably crash; but even in other languages it's useful as a simple sanity check that the file has not been corrupted.
User avatar
adurdin
Site Founder
Posts: 549
Joined: Fri Aug 29, 2003 11:27 pm
Location: Edinburgh, Scotland
Contact:

Post by adurdin »

Levellass: the main reason for needing to know the size up-front is so you know how much memory the uncompressed data will take up. Obviously this is more important in languages like C with manual memory management, but it can make any implementation more efficient.
The other reason to check it is as a safeguard: if the data file has been corrupted, you might well end up writing much more (bogus) data than expected (or much less). In C, this would result in a buffer overflow, and you'd probably crash; but even in other languages it's useful as a simple sanity check that the file has not been corrupted.
levellass
Posts: 3001
Joined: Wed Oct 11, 2006 12:03 pm
Location: Ngaruawahia New Zealand

Post by levellass »

The other reason to check it is as a safeguard: if the data file has been corrupted, you might well end up writing much more (bogus) data than expected (or much less). In C, this would result in a buffer overflow, and you'd probably crash; but even in other languages it's useful as a simple sanity check that the file has not been corrupted.
Interesting, I just set a reasonable limit on how much data I store, many graphics have a set size I set space for (Ignoring any overflow.) and those that don't I know Keen's limitations so I set a 'maximum' size after which an error is reported. I guess C is different, I have little experience with it.
Post Reply