Add TurboVNC encoding support.

TurboVNC is a variant of TightVNC that uses the same client/server protocol (RFB version 3.8t),
and thus it is fully cross-compatible with TightVNC and TigerVNC (with one exception, which is noted below.)
Both the TightVNC and TurboVNC encoders analyze each rectangle, pick out regions of solid color to send
separately, and send the remaining subrectangles using mono, indexed color, JPEG, or raw encoding, depending
on the number of colors in the subrectangle.  However, TurboVNC uses a fundamentally different selection
algorithm to determine the appropriate subencoding to use for each subrectangle.  Thus, while it sends a
protocol stream that can be decoded by any TightVNC-compatible viewer, the mix of subencoding types in this
protocol stream will be different from those generated by a TightVNC server.

The research that led to TurboVNC is described in the following report:
http://www.virtualgl.org/pmwiki/uploads/About/tighttoturbo.pdf.
In summary:  20 RFB captures, representing "common" 2D and 3D application workloads (the 3D workloads were
run using VirtualGL), were studied using the TightVNC encoder in isolation.  Some of the analysis features
in the TightVNC encoder, such as smoothness detection, were found to generate a lot of CPU usage with little
or no benefit in compression, so those features were disabled.  JPEG encoding was accelerated using
libjpeg-turbo (which achieves a 2-4x speedup over plain libjpeg on modern x86 or ARM processors.)  Finally,
the "palette threshold" (minimum number of colors that the subrectangle must have before it is compressed
using JPEG or raw) was adjusted to account for the fact that JPEG encoding is now quite a bit faster
(meaning that we can now use it more without a CPU penalty.)  TurboVNC has additional optimizations,
such as the ability to count colors and encode JPEG images directly from the framebuffer without first
translating the pixels into RGB.  The TurboVNC encoder compares quite favorably in terms of compression
ratio with TightVNC and generally encodes a great deal faster (often an order of magnitude or more.)

The version of the TurboVNC encoder included in this patch is roughly equivalent to the one found in version
0.6 of the Unix TurboVNC Server, with a few minor patches integrated from TurboVNC 1.1.  TurboVNC 1.0
added multi-threading capabilities, which can be added in later if desired (at the expense of making
libvncserver depend on libpthread.)

Because TurboVNC uses a fundamentally different mix of subencodings than TightVNC, because it uses
the identical protocol (and thus a viewer really has no idea whether it's talking to a TightVNC or
TurboVNC server), and because it doesn't support rfbTightPng (and in fact conflicts with it-- see below),
the TurboVNC and TightVNC encoders cannot be enabled simultaneously.

Compatibility:

In *most* cases, a TurboVNC-enabled viewer is fully compatible with a TightVNC server, and vice versa.
TurboVNC supports pseudo-encodings for specifying a fine-grained (1-100) quality scale and specifying
chrominance subsampling.  If a TurboVNC viewer sends those to a TightVNC server, then the TightVNC server
ignores them, so the TurboVNC viewer also sends the quality on a 0-9 scale that the TightVNC server can
understand.  Similarly, the TurboVNC server checks first for fine-grained quality and subsampling
pseudo-encodings from the viewer, and failing to receive those, it then checks for the TightVNC 0-9
quality pseudo-encoding.

There is one case in which the two systems are not compatible, and that is when a TightVNC or TigerVNC
viewer requests compression level 0 without JPEG from a TurboVNC server.  For performance reasons,
this causes the TurboVNC server to send images directly to the viewer, bypassing Zlib.  When the
TurboVNC server does this, it also sets bits 7-4 in the compression control byte to rfbTightNoZlib (0x0A),
which is unfortunately the same value as rfbTightPng.  Older TightVNC viewers that don't handle PNG
will assume that the stream is uncompressed but still encapsulated in a Zlib structure, whereas newer
PNG-supporting TightVNC viewers will assume that the stream is PNG.  In either case, the viewer will
probably crash.  Since most VNC viewers don't expose compression level 0 in the GUI, this is a
relatively rare situation.

Description of changes:

configure.ac
-- Added support for libjpeg-turbo.  If passed an argument of --with-turbovnc, configure will now run
   (or, if cross-compiling, just link) a test program that determines whether the libjpeg library being
   used is libjpeg-turbo.  libjpeg-turbo must be used when building the TurboVNC encoder, because the
   TurboVNC encoder relies on the libjpeg-turbo colorspace extensions in order to compress images directly
   out of the framebuffer (which may be, for instance, BGRA rather than RGB.)  libjpeg-turbo can optionally
   be used with the TightVNC encoder as well, but the speedup will only be marginal (the report linked
   above explains why in more detail, but basically it's because of Amdahl's Law.  The TightVNC encoder
    was designed with the assumption that JPEG had a very high CPU cost, and thus JPEG is used only sparingly.)
-- Added a new configure variable, JPEG_LDFLAGS.  This is necessitated by the fact that libjpeg-turbo
   often distributes libjpeg.a and libjpeg.so in /opt/libjpeg-turbo/lib32 or /opt/libjpeg-turbo/lib64,
   and many people prefer to statically link with it.  Thus, more flexibility is needed than is provided
   by --with-jpeg.  If JPEG_LDFLAGS is specified, then it overrides the changes to LDFLAGS enacted by
   --with-jpeg (but --with-jpeg is still used to set the include path.)  The addition of JPEG_LDFLAGS
   necessitated replacing AC_CHECK_LIB with AC_LINK_IFELSE (because AC_CHECK_LIB automatically sets
   LIBS to -ljpeg, which is not what we want if we're, for instance, linking statically with libjpeg-turbo.)
-- configure does not check for PNG support if TurboVNC encoding is enabled.  This prevents the
   rfbSendRectEncodingTightPng() function from being compiled in, since the TurboVNC encoder doesn't
   (and can't) support it.

common/turbojpeg.c, common/turbojpeg.h
-- TurboJPEG is a simple API used to compress and decompress JPEG images in memory.  It was originally
   implemented because it was desirable to use different types of underlying technologies to compress
   JPEG on different platforms (mediaLib on SPARC, Quicktime on PPC Macs, Intel Performance Primitives, etc.)
   These days, however, libjpeg-turbo is the only underlying technology used by TurboVNC, so TurboJPEG's
   purpose is largely just code simplicity and flexibility.  Thus, since there is no real need for
   libvncserver to use any technology other than libjpeg-turbo for compressing JPEG, the TurboJPEG wrapper
   for libjpeg-turbo has been included in-tree so that libvncserver can be directly linked with libjpeg-turbo.
   This is convenient because many modern Linux distros (Fedora, Ubuntu, etc.) now ship libjpeg-turbo as
   their default libjpeg library.

libvncserver/rfbserver.c
-- Added logic to check for the TurboVNC fine-grained quality level and subsampling encodings and to
   map Tight (0-9) quality levels to appropriate fine-grained quality level and subsampling values if
   communicating with a TightVNC/TigerVNC viewer.

libvncserver/turbo.c
-- TurboVNC encoder (compiled instead of libvncserver/tight.c)

rfb/rfb.h
-- Added support for the TurboVNC subsampling level

rfb/rfbproto.h
-- Added constants for the TurboVNC fine quality level and subsampling encodings as well as the rfbTightNoZlib
   constant and notes on its usage.

1 files changed, 30 insertions, 4 deletions

diff --git a/rfb/rfbproto.h b/rfb/rfbproto.h
index c6dfd2c..d99fbe5 100644
--- a/rfb/rfbproto.h
+++ b/rfb/rfbproto.h
@@ -13,7 +13,9 @@
 */
 
 /*
+ *  Copyright (C) 2009-2010 D. R. Commander. All Rights Reserved.
  *  Copyright (C) 2005 Rohit Kumar, Johannes E. Schindelin
+ *  Copyright (C) 2004-2008 Sun Microsystems, Inc. All Rights Reserved.
  *  Copyright (C) 2000-2002 Constantin Kaplinsky.  All Rights Reserved.
  *  Copyright (C) 2000 Tridia Corporation.  All Rights Reserved.
  *  Copyright (C) 1999 AT&T Laboratories Cambridge.  All Rights Reserved.
@@ -457,6 +459,8 @@ typedef struct {
 
 /*
  * Special encoding numbers:
+ *   0xFFFFFD00 .. 0xFFFFFD05 -- subsampling level
+ *   0xFFFFFE00 .. 0xFFFFFE64 -- fine-grained quality level (0-100 scale)
  *   0xFFFFFF00 .. 0xFFFFFF0F -- encoding-specific compression levels;
  *   0xFFFFFF10 .. 0xFFFFFF1F -- mouse cursor shape data;
  *   0xFFFFFF20 .. 0xFFFFFF2F -- various protocol extensions;
@@ -465,6 +469,17 @@ typedef struct {
  *   0xFFFFFFF0 .. 0xFFFFFFFF -- cross-encoding compression levels.
  */
 
+#ifdef LIBVNCSERVER_HAVE_TURBOVNC
+#define rfbEncodingFineQualityLevel0   0xFFFFFE00
+#define rfbEncodingFineQualityLevel100 0xFFFFFE64
+#define rfbEncodingSubsamp1X           0xFFFFFD00
+#define rfbEncodingSubsamp4X           0xFFFFFD01
+#define rfbEncodingSubsamp2X           0xFFFFFD02
+#define rfbEncodingSubsampGray         0xFFFFFD03
+#define rfbEncodingSubsamp8X           0xFFFFFD04
+#define rfbEncodingSubsamp16X          0xFFFFFD05
+#endif
+
 #define rfbEncodingCompressLevel0  0xFFFFFF00
 #define rfbEncodingCompressLevel1  0xFFFFFF01
 #define rfbEncodingCompressLevel2  0xFFFFFF02
@@ -719,12 +734,19 @@ typedef struct {
  *   bit 3:    if 1, then compression stream 3 should be reset;
  *   bits 7-4: if 1000 (0x08), then the compression type is "fill",
  *             if 1001 (0x09), then the compression type is "jpeg",
- *             if 1001 (0x0A), then the compression type is "png",
- *             if 0xxx, then the compression type is "basic",
+ *             (TurboVNC) if 1010 (0x0A), then the compression type is "basic"
+ *               and no Zlib compression was used,
+ *             (TurboVNC) if 1110 (0x0E), then the compression type is "basic",
+ *               no Zlib compression was used, and a "filter id" byte follows
+ *               this byte,
+ *             (TightVNC) if 1010 (0x0A), then the compression type is "png",
+ *             if 0xxx, then the compression type is "basic" and Zlib
+ *               compression was used,
  *             values greater than 1010 are not valid.
  *
- * If the compression type is "basic", then bits 6..4 of the
- * compression control byte (those xxx in 0xxx) specify the following:
+ * If the compression type is "basic" and Zlib compression was used, then bits
+ * 6..4 of the compression control byte (those xxx in 0xxx) specify the
+ * following:
  *
  *   bits 5-4:  decimal representation is the index of a particular zlib
  *              stream which should be used for decompressing the data;
@@ -836,7 +858,11 @@ typedef struct {
 #define rfbTightExplicitFilter         0x04
 #define rfbTightFill                   0x08
 #define rfbTightJpeg                   0x09
+#ifdef LIBVNCSERVER_HAVE_TURBOVNC
+#define rfbTightNoZlib                 0x0A
+#else
 #define rfbTightPng                    0x0A
+#endif
 #define rfbTightMaxSubencoding         0x0A
 
 /* Filters to improve compression efficiency */