Fix a locking problem in libvncserver

There seems to be a locking problem in libvncserver, with respect to how
condition variables are used.

On certain machines in our lab, when using a vncviewer to view a display
that has a very high rate of updates, we will occasionally see the VNC
server process crash.  In one stack trace that was obtained, an assertion
had tripped in glibc's pthread_cond_wait, which was called from
clientOutput.

Inspection of clientOutput suggests that WAIT is being called incorrectly.
The mutex that protects a condition variable should always be locked when
calling wait, and on return from the wait will still be locked.  The
attached patch fixes the locking around this condition variable, and one
other that I found by grepping the source for similar occurrences.

Signed-off-by: Charles Coffing <[email protected]>

2 files changed, 2 insertions, 3 deletions

diff --git a/libvncserver/main.c b/libvncserver/main.c
index 3af90ca..52bd4e7 100644
--- a/libvncserver/main.c
+++ b/libvncserver/main.c
@@ -455,12 +455,11 @@ clientOutput(void *data)
 		haveUpdate = sraRgnAnd(updateRegion,cl->requestedRegion);
 		sraRgnDestroy(updateRegion);
 	    }
-	    UNLOCK(cl->updateMutex);
 
             if (!haveUpdate) {
                 WAIT(cl->updateCond, cl->updateMutex);
-		UNLOCK(cl->updateMutex); /* we really needn't lock now. */
             }
+	    UNLOCK(cl->updateMutex);
         }
         
         /* OK, now, to save bandwidth, wait a little while for more
diff --git a/libvncserver/rfbserver.c b/libvncserver/rfbserver.c
index 0c7f584..434d59d 100644
--- a/libvncserver/rfbserver.c
+++ b/libvncserver/rfbserver.c
@@ -500,9 +500,9 @@ rfbClientConnectionGone(rfbClientPtr cl)
       do {
 	LOCK(cl->refCountMutex);
 	i=cl->refCount;
-	UNLOCK(cl->refCountMutex);
 	if(i>0)
 	  WAIT(cl->deleteCond,cl->refCountMutex);
+	UNLOCK(cl->refCountMutex);
       } while(i>0);
     }
 #endif