reconnect

Several days ago there was a bug reported that our XML-RPC client over SSL will compliant that "no data is available" when trying to read the HTTP header from the server. Since it is a P2-critical bug, I spent a lot of time debugging it immediately.

GDB shows me that "BIO_read()" returned zero when reading the HTTP header, and its manual said that:
A 0 or -1 return is not necessarily an indication of an error. In par-
ticular when the source/sink is non-blocking or of a certain type it
may merely be an indication that no data is currently available and
that the application should retry the operation later.

But finally I can see that when we switching to HTTP from HTTPS, the client works just fine. And later, we are confirmed that the SSL handling in the server side has a race condition and the connection would be shutdown by then.

In order to make the client more robust, I worked our a patch to do reconnection. The result is rather exciting -- I did thousands of XML-RPC calls over a persistent connection, and everything works perfectly.

At last, I am glad to share that, in section 6.3 of "UNIX Network Programming Volume 1", the author described the conditions that a descriptor being select() is ready for read:

b. The read half of the connection is closed (i.e., a TCP connection that has received a FIN). A read operation on the socket will not block and will return 0 (i.e., EOF).

Actually our underlying socket is handled by select() , and I checked that BIO_get_close() really returns 1 when the connection is closed by the server.

timeout

Eventually I managed to add a timeout value in the server side code so that an idle connection will be shutdown [patch] within a given time. The patch is much simpler than what's expected, and at the same time, I realized that I did NOT turn libxr back to non-blocking I/O, but non-buffering I/O.

I solved another problem which blocked our development this afternoon. It turned out to be SSL re-handshake problem -- as I guessed. Adding a Squid proxy between our client and server application is not that transparent as we thought, it will request the server program to do SSL re-handshake.

First git branch

I just created my first git branch of libxr with the command line:
# git checkout -b non-blocking
# git commit -a

Of course, the first commit is a patch which turns libxr back to non-blocking I/O. The next step is to setup the time out value of waiting incoming HTTP header.

BTW, I am also planning to spare sometime writing a document about libxr, with XeTeX.

P1 bug

These days I've been occupied by a P1 bug and, unfortunately, the fix is far from ready.

The XML-RPC library we use has a thread pool with each thread serves incoming request, and the thread will not be ripped back to the pool until the client shuts down the connection. The problem is, when the number of established connection prevails the number of worker threads, the following connection will be blocked.

Since the library uses buffered I/O thus blocking, the first step I need to take is to change it back to non-blocking I/O style -- If a connection has been idle for a while, we might have a chance to shut it down.

My brain now is jammed with confusion and I felt quit dizzy, as many newbies who are trying openssl programming. Just too many stuff!

1. buffered I/O
2. chained BIO
   
3. blocking I/O
4. non-blocking I/O
5. BIO with select()