I found a bad bug where the loss of a reply caused a dead lock of
of primary and secondary services during sync up. It happens very
rarely, but i can happen.

I was wondering of people could share their thoughts in designing
robust client-server protocols.

----

Depends on what you mean by "robust". One definition is "any process that continues to run will halt with an output value in a fixed number of steps, regardless of delays or failures by other processes". This definition is discussed in: 

	* M.P. Herlihy and N. Shavit. ''The Topological Structure of Asynchronous Computability''. Journal of the ACM, November 1999. (http://www.cs.brown.edu/people/mph/HerlihyS99/p858-herlihy.pdf)

which proves that for a large set of practical problems, this kind of robustness is an impossibility (mathematically).

Another paper:

	* M. Herlihy and E. Ruppert. ''On the Existence of Booster Types''. In Thirty-Second IEEE Symposium on Foundations of Computer Science (FOCS), November 12-14, 2000 (http://www.cs.brown.edu/people/mph/HerlihyRu00/focs.pdf)

discusses the different models of asynchronous concurrent computing, and builds a hierarchy of models that can be implemented on top of other models, losing a degree of robustness at each stage.

However, there is also a positive paper that applies the same methods:

	* M.P. Herlihy, V. Luchangco, M. Moir and W.M. Scherer. ''Software Transactional Memory for Dynamic-sized Data Structures'', Proceedings of the Twenty-Second Annual ACM SIGACT-SIGOPS Symposium on Principles of Distributed Computing (PODC), July 2003 (http://www.cs.brown.edu/people/mph/HerlihyLM03b/main.pdf)

It doesn't promise the same degree of robustness (called "wait-free") that has been proven impossible, but does promise a weaker form of robustness, called here "obstruction-free".

''Is there something more concrete programmers can use in client-server messaging protocols. It's not clear how to turn those papers into systems.''

SORRY! I had the wrong reference for the third one (corrected now). The right reference '''is''' directly applicable in programming!