I am working with large tiled (no overlap) datasets of ortho imagery that has been mosaicked in ArcGIS and exported as a single, merged TIF for the entire AOI.
These images need to be compressed so that they can be easily distributed to the public, however when compressing to JPEG straight from ArcGIS, some pretty horrendous compression artifacts appear in the imagery along the edge of the nodata boundary.
I am trying to find a way to eliminate these artifacts, while still maintaining a decent compression.
I have tried using GDAL nearblack
followed by gdalwarp -srcnodata 0 -dstalpha input output
and finally followed by gdal_translate input output -b 1 -b 2 -b 3 -mask 4 COMPRESS=JPEG –config GDAL_TIFF_INTERNAL_MASK YES
This has created a significant reduction in the black pixelated artifacts around the edge of the imagery – however these artifacts still do exist until I zoom into a scale of 1:1500 and then they disappear.
Any ideas of how to (if possible) eliminate these artifacts altogether?
Best Answer
TL;DR The short version is to use a command like the following one which will make you as happy as it is possible with GDAL.
Longer story
Your problem is not really in the compression artifacts but how to handle transparency with JPEG compressed TIFF files. It looks also that you have done right things and probably you have already read this thread from GDAL-devel mailing list http://osgeo-org.1560.x6.nabble.com/gdal-dev-Creating-jpeg-compressed-tiff-with-alpha-td4985300.html
I made a few tests with current GDAL development version 2.0-dev and it appears that the trick which are mentioned in the thread are mostly implemented into GDAL now in rather uses friendly way.
The following screenshots hopefully clarify what happens and what is doable with GDAL.
First image shows a detail from the border between data and nodata in warped and jpeg compressed image. I wouldn't say that there are any visible artifacts.
However, when setting the viewer to make black pixels transparent so that the blue background shows there are for sure nasty pixels. They appear because jpeg compression is lossy and some nodata pixels which were originally totally black (RGB 0 0 0) are only nearly black after compression, like RGB 0 0 1.
Compare the result with lossless LZW compression which keeps black as black. Compare also file sizes: LZW compressed 848 MB vs. 246 MB as JPEG compressed tiff. Note that the visual quality on the data area is about the same and the nasty things happen only at the border and if image is set to transparent.
The truth is that transparency can't be handled with transparent pixels with JPEG compression not other lossy compression methods. The mechanism that works is to attach to image an additional band that keeps the information about transparency. Transparency can be stored into an alpha channel or into auxiliary mask which can be even in a separate file. Both methods can be used with GDAL but unfortunately in your case neither of them lead to perfect result.
First shot is to convert nodata pixels into an alpha band with lossless method and compress the result into jpeg compressed tiff. The result is rather good but not perfect because as described in the mailing list thread, GDAL is compressing also the alpha channel with lossy jpeg method. However, while with nodata pixels the transparency is either zero of one but with alpha channel on range 0-255 the effect of lossy compression is not so bad.
The other method that GDAL supports is to write the data/nodata infromation into separate mask band. This info takes only 1 bit to store and it can be effectively compressed so the payload in disk space is minimal, in this case 1 MB extra for a 26855 by 23536 pixels sized warped image. The only drawback is that I do not know any viewer that supports the GDAL external mask layers. Not ever QGIS which is heavily based on GDAL with image handling understands them. There seems to a related ticket but that does not mention external masks http://hub.qgis.org/issues/3348.