LibVNCClient: Add H.264 encoding for framebuffer updates

This patch implements support in LibVNCClient for framebuffer updates
encoded as H.264 frames. Hardware accelerated decoding is performed
using VA API.

This is experimental support to let the community explore the possibilities
offered by the potential bandwidth and latency reductions that H.264 encoding
allows. This may be particularly useful for use cases such as online gaming,
hosted desktops, hosted set top boxes...

This patch only provides the client side support and is meant to be used
with corresponding server-side support, as provided by an upcoming patch for
qemu ui/vnc module (to view the display of a virtual machine executing under
QEMU).

With this H.264-based encoding, if multiple framebuffer update messages
are generated for a single server framebuffer modification, the H.264
frame data is sent only with the first update message. Subsequent update
framebuffer messages will contain only the coordinates and size of the
additional updated regions.

Instructions/Requirements:
* The patch should be applied on top of the previous patch I submitted with
minor enhancements to the gtkvncviewer application:
http://sourceforge.net/mailarchive/message.php?msg_id=30323804
* Currently only works with libva 1.0: use branch "v1.0-branch" for libva and
intel-driver. Those can be built as follows:
   cd libva
   git checkout v1.0-branch
   ./autogen.sh
   make
   sudo make install
   cd ..
   git clone git://anongit.freedesktop.org/vaapi/intel-driver
   cd intel-driver
   git checkout v1.0-branch
   ./autogen.sh
   make
   sudo make install

Signed-off-by: David Verbeiren <[email protected]>

1 files changed, 18 insertions, 0 deletions

diff --git a/rfb/rfbproto.h b/rfb/rfbproto.h
index acb2e7b..43f41e8 100644
--- a/rfb/rfbproto.h
+++ b/rfb/rfbproto.h
@@ -514,6 +514,9 @@ typedef struct {
 #define rfbEncodingSupportedEncodings 0xFFFE0002
 #define rfbEncodingServerIdentity     0xFFFE0003
 
+#ifdef LIBVNCSERVER_CONFIG_LIBVA
+#define rfbEncodingH264               0x48323634
+#endif
 
 /*****************************************************************************
  *
@@ -869,6 +872,21 @@ typedef struct {
 #endif
 
 /*- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
+ * h264 - h264 encoding.  We have an rfbH264Header structure
+ * giving the number of bytes following.  Finally the data follows is
+ * h264 encoded frame.
+ */
+
+typedef struct {
+    uint32_t nBytes;
+	uint32_t slice_type;
+	uint32_t width;
+	uint32_t height;
+} rfbH264Header;
+
+#define sz_rfbH264Header 16
+
+/*- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  * XCursor encoding. This is a special encoding used to transmit X-style
  * cursor shapes from server to clients. Note that for this encoding,
  * coordinates in rfbFramebufferUpdateRectHeader structure hold hotspot