Making quick problems with PNG encodings
I am trying to build a fast 8 bit color PNG encoder. Unfortunately, I must be wrong about the spec part. Smaller image sizes seem to work, but larger ones will only open in some image viewers. This image (with a few DEFLATE blocks) gives the "Decompression error in IDAT" error in my image viewer, but it opens fine in my browser:
This image only has one DEFLATE block, but it also gives an error:
Below I will cover what I have put in my IDAT block if you can easily spot any errors (note, images and steps have been modified based on the answers, but the problem is still there):
-
IDAT length
-
"IDAT" in ascii (literally bytes 0x49 0x44 0x41 0x54)
-
Zlib Header 0x78 0x01
Steps 4-7 for each deflation block, since the data can be split:
-
Byte 0x00 or 0x01, whichever is middle or last.
-
The number of bytes in a block (up to 2 ^ 16-1), stored as the small end of a 16-bit integer
-
1's complement of this integer representation.
-
Image data (each scan line starts with a zero byte for the no filter parameter in PNG and is followed by bytes of the grayscale pixel data width)
-
Adler-32 checksum of all image data
-
CRC of all IDAT data
I've tried pngcheck
on linux and it doesn't show any errors. If no one sees what is wrong, can you point me in the right direction with a debugging tool?
Lately I have been using libpng library to create my own decoder and debug from there.
Some people have suggested that this might be my calculation of the adler-32 function:
static uint32_t adler32(uint32_t height, uint32_t width, char** pixel_array)
{
uint32_t a=1,b=0,w,h;
for(h=0;h<height;h++)
{
b+=a;
for(w=0;w<width;w++)
{
a+=pixel_array[h][w];
b+=a;
}
}
return (uint32_t)(((b%65521)*65536)|(a%65521));
}
Note that since the pixel_array passed to the function does not contain a null byte at the beginning of each scan line (required for PNG), there is an extra b + = a (and implicit a + = 0) outer loop at the beginning of each iteration.
source to share
- Byte 0x00 or 0x80, depending on whether it is middle or last.
Change 0x80
to 0x01
and you should be fine.
0x80
is displayed as a saved block that is not the last block. All that is considered is a bit that is zero, which indicates the middle block. All data is in this "middle" block, so the decoder will reconstruct the complete image. Some liberal PNG decoders may ignore errors when trying to decode the next block that is missing and then ignore the missing check values (Adler-32 and CRC-32), etc. So it appears ok in browsers even though it is an invalid PNG file.
There are two things in the Adler-32 code. First, you are accessing data from the array char
. char
signed, so your bytes 0xff
are appended not as 255, but rather as -127. You need to make an array unsigned char
or apply it to this before fetching byte values from it.
Second, you are moduloing the operation too late. You must execute % 65521
before overflow uint32_t
. Otherwise, you will not get the modulo sums, as the algorithm requires. A simple fix would be to do % 65521
before a
and b
immediately after the width loop inside the height loop. This will work as long as you can guarantee the width is less than 5551 bytes. (Why 5551 is left as an exercise for the reader.) If you can't guarantee it, then you'll need to inline another loop to consume the bytes from the string until you get to 5551 of them, modulo and then continue the line. Or, to smooth more slowly, just run the counter and modulo when it reaches the limit.
Here's an example of a version that works for any width:
static uint32_t adler32(uint32_t height, uint32_t width, unsigned char ** pixel_array)
{
uint32_t a = 1, b = 0, w, h, k;
for (h = 0; h < height; h++)
{
b += a;
w = k = 0;
while (k < width) {
k += 5551;
if (k > width)
k = width;
while (w < k) {
a += pixel_array[h][w++];
b += a;
}
a %= 65521;
b %= 65521;
}
}
return (b << 16) | a;
}
source to share
I am getting an error with pngcheck
: "zlib: inflate error = -3 (data error)". Since your PNG forest structure looks fine, you need to peek into the IDAT
low view block with a hex view. (I'm going to introduce this while working through it.)
The headline looks good; IDAT
length is ok. Your zlib flags 78 01
("No / low compression", see also What does the zlib header look like? ) Where one of my own tools uses 78 9C
("Default compression"), but again, these flags are only informative.
Next: zlib internals (per RFC1950 ).
Immediately after the compression flags ( CMF
in RFC1950), it expects FLATE compressed data, which is the only compression scheme supported by zlib. And that's in another RFC lock : RFC1951 .
Each individual block of compression is appended with a byte:
3.2.3. Details of the block format
Each block of compressed data starts with 3 header bits containing the following data:
first bit BFINAL next 2 bits BTYPE
...
BFINAL is set if and only if it is the last data block set.BTYPE defines how the data is compressed as follows:
00 - no compression 01 - compressed with fixed Huffman codes 10 - compressed with dynamic Huffman codes 11 - reserved (error)
Thus, this value can be set to 00
for "not last block, uncompressed" and 01
for "last block, uncompressed", immediately followed by the length (2 bytes) and its bitwise inversion, after 3.2.4. Uncompressed blocks (BTYPE = 00) :
3.2.4. Uncompressed blocks (BTYPE = 00)
Any input bits up to the next byte boundary are ignored. The rest of the block consists of the following information:
0 1 2 3 4... +---+---+---+---+================================+ | LEN | NLEN |... LEN bytes of literal data...| +---+---+---+---+================================+
LEN is the number of data bytes in the block. NLEN is one addition to LEN.
These are the last 4 bytes in the segment IDAT
. Why do small images work and no longer? Since you only have 2 bytes for the length. 1 You need to split the image into blocks of no more than 65535 bytes (in my own PNG maker, I seem to have used 32768, probably "for safety",). If the last block, write 01
else 00
. Then add two times two bytes LEN
, correctly encoded, followed by exactly the LEN
data bytes. Repeat until the end.
The Adler-32 checksum is not part of the Flate compressed data and should not be counted in data blocks LEN
. (However, it is still part of the block IDAT
.)
After re-reading your question to make sure that I have addressed all of your problems (and confirmed that I have written Adler-32 correctly), I realized that you are describing all the steps to the right - except that the last block is' (edit later: uh, you might be right about that!) - but what it doesn't show in this PNG example. See if you can get it to work after all the writing steps.01
and not80
Kudos for doing this "by hand". This is a great "to spec" exercise, and if you get it working it might be worth trying and adding compression. I try to avoid ready-made libraries; the only tutorial I did for my own PNG encoder / decoder was using Rich Geldreich miniz.c
as implementing proper Flate encoding / decoding is out of my ken's scope.
1 This is not the whole story. Browsers are especially forgiving of HTML errors; they seem to forgive PNG's mistakes as well. Safari displays your image just fine, as well as the Preview. But they can just use the generic OS X PNG decoder because Photoshop rejects the file.
source to share