Home > Uncategorized > Tricks and Tips with NIO part I: Why you must handle OP_WRITE

Tricks and Tips with NIO part I: Why you must handle OP_WRITE

I’m getting a lot of questions about NIO in general, and since most of them apply not only to HTTP handling, I’ve decided to blog about my experience with NIO in Grizzly. The observations I’ve measured might not apply to all NIO based servers implementation, but I suspect it will cover the majority of them. Anyway there is not much documentations about NIO in general (except basic tutorial), so it might not hurt to blog about it, whatever I’m right or wrong. But first, I recommend reading an NIO tutorial (if you don’t know what NIO is) before reading this blog :-).

When building a scalable NIO server, you always have to handle three important NIO operation set bit:

  • OP_ACCEPT: Operation-set bit for socket-accept operations
  • OP_READ: Operation-set bit for read operations
  • OP_WRITE: Operation-set bit for write operations

Handling OP_ACCEPT and OP_READ has been well documented in several NIO tutorial. Strangely, the OP_WRITE is sometimes not described at all. Not handling the OP_WRITE correctly can make your server performance pretty bad, and on win32, can produce disastrous performance problem, like freezing the OS by eating all the CPU. How come? Well, let start with an example. Usually, you will write to a SocketChannel by doing:

     while ( bb.hasRemaining() ) {
        int len = socketChannel.write(bb);
        if (len < 0){
           throw new EOFException();

This code will works most of the time….until the Selector on which the SocketChannel has been registered is exhausted, e.g the Selector isn’t able to let the socketChannel flush the content of the ByteBuffer. , which means:

     while ( bb.hasRemaining() ) {
        // *** socketChannel will always return 0
        int len = socketChannel.write(bb);       
        if (len < 0){
           throw new EOFException();

Hence the CPU will be consumed by looping over and over, producing disastrous performance problem (try it in win32 :-)). OK, but what can we do? There is several ways of handling this. In GlassFish, Grizzly uses a pool of temporary Selector to register the SocketChannel on it:

        try {
            while ( bb.hasRemaining() ) {
                int len = socketChannel.write(bb);
                if (len  2)
                            throw new IOException("Client disconnected");
                    } else {
                } else {
                    attempts = 0;
        } finally {
            if (key != null) {
                key = null;

            if ( writeSelector != null ) {
                // Flush the key.

If the main Selector is exhausted, a temporary Selector will be used to handle the OP_WRITE. In Grizzly, since the OP_READ also use the pool of Selector, the pool might return null. In that case, the main Selector will be re-used instead, like the Mina framework is doing. The Jetty Web server seems to create a temporary Selector lazily (I don’t know Jetty enough to know the life cycle of SocketChannelOutputStream, but I suspect this object is recycled amongst requests so there is not an infinite Selector creation). EmberIO doesn’t seems to handle OP_WRITE at all, which is surprising knowing the popularity of this framework.

Another alternative is to use a ThreadLocal to store a temporary Selector. Unfortunately the benchmarks I did demonstrated that this approach is slower that using temporary pool of Selector.

That’s it. Next time I will discuss the evil method SelectionKey.attach() gold candidate for memory leak.


_uacct = “UA-3111670-1”;

Categories: Uncategorized
  1. December 12, 2014 at 11:24 am

    Hello Jeanfrancois,

    thanks for sharing this. Unfortunately, your third code snippet is a bit broken (some parts got eaten by wordpress), so I am not sure if I got everything correctly.

    None the less, wouldn’t the following approach be easier:

    1. write, until socket.write(bb) returns null
    2. if bb.hasRemaining() = 0, we’re done
    3. else, wait until the next OP_WRITE is fired for the socket, and continue at 1.

    This should avoid flooding the selector and busy looping. Or did I miss something here?


  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: