Hi everyone,
I'm new, nice to meet you all. I had to jump through some hoops to get registered since I don't know much about keen (blasphemy!)
Anyways, I am writing a level loading func for Wolf3D which uses the same file format as Keen for levels.
I am having trouble un-carmikizing and/or un-RLE'ing (can't figure out which one is wrong)
Specifically, after unRLEW'ing the files, the uncompressed data is huge, much larger than the total length of a single plane array == 64*64*2 == 8192 bytes, for a 2-byte word.
Thanks for any possible help.
Here are the results I get:
File == w38.map from Wolf3D
MAP NAME: Wolf4 Boss
MUSIC NAME: music/WARMARCH.ogg
ROW WIDTH (w): 64
NUM ROWS (h): 64
CEILING COLOUR: 56 56 56 0
FLOOR COLOUR: 112 112 112 0
LENGTH ARRAY FOR PLANES, IN WORDS: 705 370 10
OFFSET ARRAY FOR PLANES, IN BYTES: 77 782 1152
RLE TAG: 43981
PLANE1 BYTE SIZE (UNCOMPRESSED): 48994
PLANE2 BYTE SIZE (UNCOMPRESSED): 6588
PLANE3 BYTE SIZE (UNCOMPRESSED): 4096
Writing a level loading function; trouble with decompression
Here are some details for my decompression, for the same map file w38.map:
Its hard to believe that a word would be needed to be repeated 0xABCD (the RLE tag) times, as in this:
DECOMPRESSING... CUR_INDEX = 1179, COUNT = 43981, VALUE = 9
** Decompression output****
@@@ TESTING CARMACK COMP/DECOMP
SIZE OF compressed_data_byte_length = 1410
SIZE OF temp_data = 1083
DONE DECOMPRESS!
SIZE OF out_data_array = 2486
@@@ TESTING RLEW COMP/DECOMP
EXPECTED SIZE TO DECOMPRESS = 8192
SIZE OF compressed_data_byte_length = 288
SIZE OF in_byte_array = 2484
DECOMPRESSING... CUR_INDEX = 37, COUNT = 6, VALUE = 1
DECOMPRESSING... CUR_INDEX = 93, COUNT = 13, VALUE = 109
DECOMPRESSING... CUR_INDEX = 101, COUNT = 29, VALUE = 1
DECOMPRESSING... CUR_INDEX = 133, COUNT = 8, VALUE = 109
DECOMPRESSING... CUR_INDEX = 173, COUNT = 31, VALUE = 1746
DECOMPRESSING... CUR_INDEX = 225, COUNT = 45, VALUE = 35
DECOMPRESSING... CUR_INDEX = 267, COUNT = 45, VALUE = 48
DECOMPRESSING... CUR_INDEX = 435, COUNT = 11, VALUE = 1458
DECOMPRESSING... CUR_INDEX = 777, COUNT = 17, VALUE = 120
DECOMPRESSING... CUR_INDEX = 1023, COUNT = 173, VALUE = 3284
DECOMPRESSING... CUR_INDEX = 1099, COUNT = 4, VALUE = 8
DECOMPRESSING... CUR_INDEX = 1105, COUNT = 44, VALUE = 274
DECOMPRESSING... CUR_INDEX = 1117, COUNT = 11, VALUE = 1458
DECOMPRESSING... CUR_INDEX = 1125, COUNT = 36, VALUE = 37
DECOMPRESSING... CUR_INDEX = 1179, COUNT = 43981, VALUE = 9
DECOMPRESSING... CUR_INDEX = 1555, COUNT = 2533, VALUE = 99
DECOMPRESSING... CUR_INDEX = 1641, COUNT = 48, VALUE = 0
DECOMPRESSING... CUR_INDEX = 1649, COUNT = 6, VALUE = 4
DECOMPRESSING... CUR_INDEX = 1667, COUNT = 138, VALUE = 0
DECOMPRESSING... CUR_INDEX = 1675, COUNT = 37, VALUE = 9
DECOMPRESSING... CUR_INDEX = 1731, COUNT = 6, VALUE = 3
DECOMPRESSING... CUR_INDEX = 1755, COUNT = 26, VALUE = 0
DECOMPRESSING... CUR_INDEX = 1817, COUNT = 5, VALUE = 48
DECOMPRESSING... CUR_INDEX = 1823, COUNT = 415, VALUE = 48
DECOMPRESSING... CUR_INDEX = 1911, COUNT = 34, VALUE = 0
DECOMPRESSING... CUR_INDEX = 1959, COUNT = 5, VALUE = 48
DECOMPRESSING... CUR_INDEX = 1965, COUNT = 15, VALUE = 2211
DECOMPRESSING... CUR_INDEX = 2019, COUNT = 23, VALUE = 0
DECOMPRESSING... CUR_INDEX = 2027, COUNT = 38, VALUE = 0
DECOMPRESSING... CUR_INDEX = 2067, COUNT = 9, VALUE = 3
DECOMPRESSING... CUR_INDEX = 2197, COUNT = 7, VALUE = 12
DECOMPRESSING... CUR_INDEX = 2207, COUNT = 46, VALUE = 0
DECOMPRESSING... CUR_INDEX = 2301, COUNT = 7, VALUE = 164
DONE DECOMPRESS!
SIZE OF raw_data = 48994
Its hard to believe that a word would be needed to be repeated 0xABCD (the RLE tag) times, as in this:
DECOMPRESSING... CUR_INDEX = 1179, COUNT = 43981, VALUE = 9
** Decompression output****
@@@ TESTING CARMACK COMP/DECOMP
SIZE OF compressed_data_byte_length = 1410
SIZE OF temp_data = 1083
DONE DECOMPRESS!
SIZE OF out_data_array = 2486
@@@ TESTING RLEW COMP/DECOMP
EXPECTED SIZE TO DECOMPRESS = 8192
SIZE OF compressed_data_byte_length = 288
SIZE OF in_byte_array = 2484
DECOMPRESSING... CUR_INDEX = 37, COUNT = 6, VALUE = 1
DECOMPRESSING... CUR_INDEX = 93, COUNT = 13, VALUE = 109
DECOMPRESSING... CUR_INDEX = 101, COUNT = 29, VALUE = 1
DECOMPRESSING... CUR_INDEX = 133, COUNT = 8, VALUE = 109
DECOMPRESSING... CUR_INDEX = 173, COUNT = 31, VALUE = 1746
DECOMPRESSING... CUR_INDEX = 225, COUNT = 45, VALUE = 35
DECOMPRESSING... CUR_INDEX = 267, COUNT = 45, VALUE = 48
DECOMPRESSING... CUR_INDEX = 435, COUNT = 11, VALUE = 1458
DECOMPRESSING... CUR_INDEX = 777, COUNT = 17, VALUE = 120
DECOMPRESSING... CUR_INDEX = 1023, COUNT = 173, VALUE = 3284
DECOMPRESSING... CUR_INDEX = 1099, COUNT = 4, VALUE = 8
DECOMPRESSING... CUR_INDEX = 1105, COUNT = 44, VALUE = 274
DECOMPRESSING... CUR_INDEX = 1117, COUNT = 11, VALUE = 1458
DECOMPRESSING... CUR_INDEX = 1125, COUNT = 36, VALUE = 37
DECOMPRESSING... CUR_INDEX = 1179, COUNT = 43981, VALUE = 9
DECOMPRESSING... CUR_INDEX = 1555, COUNT = 2533, VALUE = 99
DECOMPRESSING... CUR_INDEX = 1641, COUNT = 48, VALUE = 0
DECOMPRESSING... CUR_INDEX = 1649, COUNT = 6, VALUE = 4
DECOMPRESSING... CUR_INDEX = 1667, COUNT = 138, VALUE = 0
DECOMPRESSING... CUR_INDEX = 1675, COUNT = 37, VALUE = 9
DECOMPRESSING... CUR_INDEX = 1731, COUNT = 6, VALUE = 3
DECOMPRESSING... CUR_INDEX = 1755, COUNT = 26, VALUE = 0
DECOMPRESSING... CUR_INDEX = 1817, COUNT = 5, VALUE = 48
DECOMPRESSING... CUR_INDEX = 1823, COUNT = 415, VALUE = 48
DECOMPRESSING... CUR_INDEX = 1911, COUNT = 34, VALUE = 0
DECOMPRESSING... CUR_INDEX = 1959, COUNT = 5, VALUE = 48
DECOMPRESSING... CUR_INDEX = 1965, COUNT = 15, VALUE = 2211
DECOMPRESSING... CUR_INDEX = 2019, COUNT = 23, VALUE = 0
DECOMPRESSING... CUR_INDEX = 2027, COUNT = 38, VALUE = 0
DECOMPRESSING... CUR_INDEX = 2067, COUNT = 9, VALUE = 3
DECOMPRESSING... CUR_INDEX = 2197, COUNT = 7, VALUE = 12
DECOMPRESSING... CUR_INDEX = 2207, COUNT = 46, VALUE = 0
DECOMPRESSING... CUR_INDEX = 2301, COUNT = 7, VALUE = 164
DONE DECOMPRESS!
SIZE OF raw_data = 48994
- adurdin
- Site Founder
- Posts: 549
- Joined: Fri Aug 29, 2003 11:27 pm
- Location: Edinburgh, Scotland
- Contact:
Good to have you here. I'm glad you were able to get past the bot filter ok :)jigglez wrote:I'm new, nice to meet you all. I had to jump through some hoops to get registered since I don't know much about keen
Yes, obviously something just before has not decompressed correctly, leaving you reading the tag when you expect a count.I am having trouble un-carmikizing and/or un-RLE'ing (can't figure out which one is wrong)
...
Its hard to believe that a word would be needed to be repeated 0xABCD (the RLE tag) times
It's possible your carmack-decompression is faulty. You can easily test this by comparing the expected decompressed length (that's stored just before the compressed data) with the actual length after you've carmack-decompressed it, but before any RLE decompression. If they're different, the carmack-decompression is faulty.
I'm not able to try decompression the wolf3d files myself to compare what I get with your results; however if you want a known-good carmack and RLE decompresser, grab the modkeen2 source from http://files.keenmodding.org/modkeen2.zip and extract the relevant functions from it. If you know C, it should be trivial to do a test run using the code there and compare with your results. Of course, the Wolfenstein 3D source code itself would be another good comparison.
ahh thanks for that information and advice. I am actually writing the code in Ruby, and comparing it to the source from the orig. Wolf 3D (actually the iphone one :D )
you're right, in fact the carmack headers for size after un-carmacking dont match -- i wasnt sure if they were supposed to or not, since I didnt have anyone to ask (and i didnt feel like ripping out those functions :P )
but since you know they are supposed to match, i must be doing something wrong :P
will be back once i fix it :P
however, i am pretty sure i checked it a few times and against the original C code too. well, that is the nature of bugs.. :D
you're right, in fact the carmack headers for size after un-carmacking dont match -- i wasnt sure if they were supposed to or not, since I didnt have anyone to ask (and i didnt feel like ripping out those functions :P )
but since you know they are supposed to match, i must be doing something wrong :P
will be back once i fix it :P
however, i am pretty sure i checked it a few times and against the original C code too. well, that is the nature of bugs.. :D
I myself never both reading the count for things like this, when I uncarmackize I simply read data until there's no more left (Same with all RLE I do.) Can someone tell me if this is actually necessary? (I can understand the executable, etc doing it as it can help to know how big the file(s) you're dealing with are, but for just plain decompression, does this serve any purpose?)
I don't have enough experience to be able to tell you if its necessary to check the counts or not. *however* -- having looked at the decompress functions as they were written for Wolf3D, they used the byte counts in both carmack and rle to determine when to stop the decompression. It didn't seem necessary to me either..
Also - i fixed my bug :D i got bored of the proj due to the bug then came back and after many hours, found that I was unpacking the original data from the .map file in Ruby with "C" (unsigned) instead of "c" (signed). Yurghhhh
Also - i fixed my bug :D i got bored of the proj due to the bug then came back and after many hours, found that I was unpacking the original data from the .map file in Ruby with "C" (unsigned) instead of "c" (signed). Yurghhhh
- adurdin
- Site Founder
- Posts: 549
- Joined: Fri Aug 29, 2003 11:27 pm
- Location: Edinburgh, Scotland
- Contact:
Levellass: the main reason for needing to know the size up-front is so you know how much memory the uncompressed data will take up. Obviously this is more important in languages like C with manual memory management, but it can make any implementation more efficient.
The other reason to check it is as a safeguard: if the data file has been corrupted, you might well end up writing much more (bogus) data than expected (or much less). In C, this would result in a buffer overflow, and you'd probably crash; but even in other languages it's useful as a simple sanity check that the file has not been corrupted.
The other reason to check it is as a safeguard: if the data file has been corrupted, you might well end up writing much more (bogus) data than expected (or much less). In C, this would result in a buffer overflow, and you'd probably crash; but even in other languages it's useful as a simple sanity check that the file has not been corrupted.
- adurdin
- Site Founder
- Posts: 549
- Joined: Fri Aug 29, 2003 11:27 pm
- Location: Edinburgh, Scotland
- Contact:
Levellass: the main reason for needing to know the size up-front is so you know how much memory the uncompressed data will take up. Obviously this is more important in languages like C with manual memory management, but it can make any implementation more efficient.
The other reason to check it is as a safeguard: if the data file has been corrupted, you might well end up writing much more (bogus) data than expected (or much less). In C, this would result in a buffer overflow, and you'd probably crash; but even in other languages it's useful as a simple sanity check that the file has not been corrupted.
The other reason to check it is as a safeguard: if the data file has been corrupted, you might well end up writing much more (bogus) data than expected (or much less). In C, this would result in a buffer overflow, and you'd probably crash; but even in other languages it's useful as a simple sanity check that the file has not been corrupted.
Interesting, I just set a reasonable limit on how much data I store, many graphics have a set size I set space for (Ignoring any overflow.) and those that don't I know Keen's limitations so I set a 'maximum' size after which an error is reported. I guess C is different, I have little experience with it.The other reason to check it is as a safeguard: if the data file has been corrupted, you might well end up writing much more (bogus) data than expected (or much less). In C, this would result in a buffer overflow, and you'd probably crash; but even in other languages it's useful as a simple sanity check that the file has not been corrupted.