Archive

Archive for July, 2006

Tricks and Tips with NIO part IV: Meet the Selectors

When building scalable NIO based server, it is important to not restrict your server to use a single Selector. Multiples Selectors can always be used to handle OP_ACCEPT, OP_READ and OP_WRITE to avoid overloading the main Selector.

Using multiples Selectors to load balance OP_ACCEPT and OP_READ

Usually, an NIO based server will do the following:

 

            serverSocketChannel = ServerSocketChannel.open();
            selector = Selector.open();

            serverSocket = serverSocketChannel.socket();
            serverSocket.setReuseAddress(true);
            if ( inet == null)
                serverSocket.bind(new InetSocketAddress(port),
                   ssBackLog);
            else
                serverSocket.bind(new InetSocketAddress(inet,port),
                                  ssBackLog);

            serverSocketChannel.configureBlocking(false);
            serverSocketChannel.register(selector, SelectionKey.OP_ACCEPT);

 

Mainly, you create an instance of ServerSocketChannel, get a Selector instance (called the main Selector), bind it to a dedicated port, configure the ServerSocketChannel non blocking, and then register the Selector for OP_ACCEPT interest key. Latter when processing the OP_ACCEPT, you register the SocketChannel to the previously created Selector

 

    protected void handleAccept(SelectionKey key) throws IOException{
        ServerSocketChannel server = (ServerSocketChannel) key.channel();
        SocketChannel channel = server.accept();

        if (channel != null) {
           channel.configureBlocking(false);
           SelectionKey readKey =
                  channel.register(selector, SelectionKey.OP_READ);
           setSocketOptions(((SocketChannel)readKey.channel()).socket())
        }
        ....

 

Although this not clearly stated inside the NIO API documentation, you don’t need to register the SocketChannel using the “main” Selector. Instead, you might want to load balance amongst several Selectors:

 

        SlaveSelector[] selectors = new SlaveSelector[selectorCount];
        for(int i=0; i < selectorCount; i++){
              SlaveSelector slave = new SlaveSelector();
              slave.start();
              selectors[i] = slave;
        }

 

Where the SlaveSelector class looks like:

 

public class SlaveSelector extends Thread{

    private Selector selector;

    public SlaveSelector throws IOException{
         selector = Selector.open();
    }

    /**
     * List of Channels to process.
     */
    ArrayList channels = new ArrayList();

    /**
     * Add a Channel to be processed by this
     * Selector
     */
    public synchronized void addChannel(SocketChannel channel)
            throws IOException, ClosedChannelException {
        channels.add(channel);
        selector.wakeup();
    }

    /**
     * Register all Channels with an OP_READ opeation.
     */
    private synchronized void registerNewChannels() throws IOException {
        int size = channels.size();
        for (int i = 0; i < size; i++) {
            SocketChannel sc = channels.get(i);
            sc.configureBlocking(false);
            try {
                SelectionKey readKey =
                        sc.register(selector, SelectionKey.OP_READ);
                setSocketOptions(((SocketChannel)
                                  readKey.channel()).socket());
            } catch (ClosedChannelException cce) {
            }
        }
        channels.clear();
    }
    .....

 

Thus, when handling OP_ACCEPT, instead of using the main Selector, you register using one “slave” Selector:

 

    protected void handleAccept(SelectionKey key) throws IOException{
        ServerSocketChannel server = (ServerSocketChannel) key.channel();
        SocketChannel channel = server.accept();

        if (channel != null) {
            SlaveSelector slave = getSlaveSelector();
            slave.addChannel(channel);
        }
    }

    private synchronized SlaveSelector getSlaveSelector() {
        if (curSlave == slaveSelectors.length)
            curSlave = 0;
        return slaveSelectors[curSlave++];
    }

 

The Grizzly implementation of this trick can be found here. Note that distributing the “load” should be carefully evaluated, because the cost of maintaining multiple Selectors can reduce scalability instead of improving it. At least an NIO server should have an option to enable such functionality.

pont.jpg

 

Using a pool of Selectors for handling OP_READ and OP_WRITE

 

The next trick is also used in Grizzly. When Grizzly handles OP_READ, it always delegate the SocketChannel read(s) processing to a thread pool. Grizzly have several strategies to determine if all the HTTP header bytes have been fully read from the SocketChannel. One of the available strategy is to read the available bytes without trying to determine if all the bytes have been read or not, but instead use a pool of Selectors to help in case bytes still need to be read.

The way I did it in Grizzly is by wrapping a ByteBuffer inside a ByteBufferInputStream:

 

public class ByteBufferInputStream extends InputStream {
    /**
     * The wrapped ByteBuffer
     */
    protected ByteBuffer byteBuffer;

    public ByteBufferInputStream (final ByteBuffer byteBuffer) {
        this.byteBuffer = byteBuffer;
    }

    /**
     * Read bytes from the wrapped ByteBuffer.
     */
    public int read (byte[] b, int offset, int length)
            throws IOException {
        if (!byteBuffer.hasRemaining()) {
            int eof = 0;
            for (int i=0; i < readTry; i++) {
                eof = doRead();

                if ( eof != 0 ){
                    break;
                }
            }

            if (eof  byteBuffer.remaining()) {
            length = byteBuffer.remaining();
        }
        byteBuffer.get(b, offset, length);

        return (length);
    }

    .....    

    /**
     * Read bytes using the read a temporary Selector
     */
    protected int doRead() throws IOException{
        if ( key == null ) return -1;

        byteBuffer.clear();
        int count = 1;
        int byteRead = 0;
        Selector readSelector = null;
        SelectionKey tmpKey = null;

        try{
            SocketChannel socketChannel =
                   (SocketChannel)key.channel();
            while (count > 0){
                count = socketChannel.read(byteBuffer);// [1]
                if ( count > -1 )
                    byteRead += count;
                else
                    byteRead = count;
            }            

            if ( byteRead == 0 ){
                readSelector = SelectorFactory.getSelector(); //[2]

                if ( readSelector == null ){
                    return 0;
                }
                count = 1;
                tmpKey = socketChannel
                        .register(readSelector,SelectionKey.OP_READ);
                tmpKey.interestOps(
                     tmpKey.interestOps() | SelectionKey.OP_READ);
                int code = readSelector.select(readTimeout);
                tmpKey.interestOps(
                    tmpKey.interestOps() & (~SelectionKey.OP_READ));

                if ( code == 0 ){
                    return 0;
                    // Return on the main Selector and try again.
                }

                while (count > 0){
                    count = socketChannel.read(byteBuffer);
                }
            }
        } finally {
            if (tmpKey != null)
                tmpKey.cancel();

            if ( readSelector != null){
                // Bug 6403933
                try{
                    readSelector.selectNow();
                } catch (IOException ex){
                    ;
                }
                SelectorFactory.returnSelector(readSelector);
            }
        }
        byteBuffer.flip();
        return byteRead;
    }
}

 

First, socketChannel.read(byteBuffer) is executed until it return 0. Then the byteBuffer is wrapped by the ByteBufferInputStream and “passed” to the HTTP Parser class. The HTTP Parser will try to parse the http request line and headers with the bytes available inside the byteBuffer. If the HTTP Parser class ask for more bytes, being unable to parse all the headers, and the socketChannel.read() ([1] in the code above) return 0 (meaning no bytes have been read), internally I register the socketChannel on a different Selector by getting a temporary Selector from a pool of ready to use Selectors [2]. Then, using the temporary Selector, try to read the missing bytes (or at least more bytes).

There is several advantages when doing this. First, you don’t need to go back to the main Selector, which most probably run on another Thread, by attaching the current state of the processing to the SelectionKey. Second, when the main Selector is ready (bytes are available), re-process the OP_READ by retrieving the previous state of the ByteBuffer, getting another Thread from the pool, and try to see if this time all the bytes are available.

I know I know, on slow network (or strange client) it might be better to go back to the main Selector instead of holding a Thread, waiting for the temporary Selector to read the missing bytes. Hence uses this trick carefully by properly configuring the time before Selector.select(..) times out, to avoid holding a Thread.

The same trick can be applied when handing OP_WRITE:

 

        try {
            while ( bb.hasRemaining() ) {
                int len = socketChannel.write(bb);
                attempts++;
                if (len  2)
                            throw new IOException("Client disconnected");
                    } else {
                        attempts--;
                    }
                } else {
                    attempts = 0;
                }
            }
        } finally {
            if (key != null) {
                key.cancel();
                key = null;
            }

            if ( writeSelector != null ) {
                // Cancel the key.
                writeSelector.selectNow();
                SelectorFactory.returnSelector(writeSelector);
            }
        }

 

Same as for OP_READ, you need to make sure the Selector.select(..) is carefully configured.

 

C’est le temps des vacances!

 

OK, as usual, thanks for the feedback! I will take a couple of weeks out of this world, avoiding NIO monsters and Grizzly bugs, so you have a break of me…until I come with part V, which will consist of explaining a couple of workarounds when boom! the VM crash, or oups!, why do I get “javax.net.ssl.SSLHandshakeException: no cipher suites in common” when NIO + SSL play together.

technorati:

_uacct = “UA-3111670-1″;
urchinTracker();

Categories: Uncategorized

Tricks and Tips with NIO part III: To Thread or Not to Thread

partIII.jpg

This time I will share some observations I’ve experimented when handling OP_ACCEPT, OP_READ and OP_WRITE using Threads. When I started working on Grizzly, I’ve designed the framework open enough so I can easily add thread pool mostly everywhere during the request processing. At that time there weren’t a lot of NIO framework available neither clear recommendations about what to do and what to avoid. To avoid having to redesign Grizzly every weeks, I’ve decided to make OP_READ, OP_ACCEPT and OP_WRITE processing configurable. By configurable, I mean being able to execute different strategies, e.g. being able to execute the processing of those operations on their own thread or using the same thread as the Selector:


            if ( myExecutor == null ){
                myExecutor = Executors.newFixedThreadPool(maxThreads);
            }

            try{
                selectorState = selector.select(selectorTimeout);
            } catch (CancelledKeyException ex){
                ;
            }

            readyKeys = selector.selectedKeys();
            iterator = readyKeys.iterator();
            while (iterator.hasNext()) {
                key = iterator.next();
                iterator.remove();
                if (key.isValid()) {                  
                   if ((key.readyOps() & SelectionKey.OP_ACCEPT) 
                         == SelectionKey.OP_ACCEPT){
                      myExecutor.execute(getAcceptHandler(key));
                   } else if ((key.readyOps() & SelectionKey.OP_READ) 
                         == SelectionKey.OP_READ) {
                      myExecutor.execute(getReadHandler(key));
                   } 
                   ....
                } else {
                    cancelKey(key);
                }
            }

From the code above, the getAcceptHandler(key) will return a Runnable object which most usually does:


        ServerSocketChannel server = (ServerSocketChannel) key.channel();
        SocketChannel channel = server.accept();

        if (channel != null) {
           channel.configureBlocking(false);
           SelectionKey readKey = 
                channel.register(selector, SelectionKey.OP_READ);
           setSocketOptions(((SocketChannel)readKey.channel()).socket());
        }

where getReadHandler(key) will do:


        key.interestOps(key.interestOps() & (~SelectionKey.OP_READ));
        socketChannel = (SocketChannel)key.channel();
        while ( socketChannel.isOpen() && (
          ((count = socketChannel.read(byteBuffer))> -1)){
              // Do something

An alternative is to execute the getAcceptHandler(key)and getReadHandler(key) on the same thread as the Selector.select(..):


            try{
                selectorState = selector.select(selectorTimeout);
            } catch (CancelledKeyException ex){
                ;
            }

            readyKeys = selector.selectedKeys();
            iterator = readyKeys.iterator();
            while (iterator.hasNext()) {
                key = iterator.next();
                iterator.remove();
                if (key.isValid()) {                  
                   if ((key.readyOps() & SelectionKey.OP_ACCEPT) 
                         == SelectionKey.OP_ACCEPT){
                      getAcceptHandler(key);
                   } else if ((key.readyOps() & SelectionKey.OP_READ) 
                         == SelectionKey.OP_READ) {
                      getReadHandler(key);
                   } 
                   ....
                } else {
                    cancelKey(key);
                }
            }

and of course, without having to create a Runnable object. Finally, the other alternative is a mix of the first two strategy:


            try{
                selectorState = selector.select(selectorTimeout);
            } catch (CancelledKeyException ex){
                ;
            }

            readyKeys = selector.selectedKeys();
            iterator = readyKeys.iterator();
            while (iterator.hasNext()) {
                key = iterator.next();
                iterator.remove();
                if (key.isValid()) {                  
                   if ((key.readyOps() & SelectionKey.OP_ACCEPT) 
                         == SelectionKey.OP_ACCEPT){
                      getAcceptHandler(key);
                   } else if ((key.readyOps() & SelectionKey.OP_READ) 
                         == SelectionKey.OP_READ) {
                      myExecutor.execute(getReadHandler(key));
                   } 
                   ....
                } else {
                    cancelKey(key);
                }
            }

or executing the getAcceptHandler(key) using myExecutor and
getReadHandler on the same thread as the Selector.select(..). Like I said earlier, Grizzly can be configured to support all strategies.

Which strategy perform the best

I’ve benchmarked all of the above strategies and find that the one that perform the best is:


            try{
                selectorState = selector.select(selectorTimeout);
            } catch (CancelledKeyException ex){
                ;
            }

            readyKeys = selector.selectedKeys();
            iterator = readyKeys.iterator();
            while (iterator.hasNext()) {
                key = iterator.next();
                iterator.remove();
                if (key.isValid()) {                  
                   if ((key.readyOps() & SelectionKey.OP_ACCEPT) 
                         == SelectionKey.OP_ACCEPT){
                      getAcceptHandler(key);
                   } else if ((key.readyOps() & SelectionKey.OP_READ) 
                         == SelectionKey.OP_READ) {
                      myExecutor.execute(getReadHandler(key));
                   } 
                   ....
                } else {
                    cancelKey(key);
                }
            }

c’est a dire executing the OP_ACCEPT on the same thread as the Selector.select(..), and using a Thread for executing the OP_READ. I’ve shared my observations with my colleagues and they also came to the same conclusion.

Well, what about OP_WRITE

I didn’t forget the OP_WRITE. I also tested the strategies described above and came to the conclusion than OP_WRITE should be handled using the same Thread as the one handling OP_READ. One thing that might explain why I’m getting such results is the use of temporary Selector when the main Selection is not able to flush the socket outgoing buffer (see part I for more details), or when socketChannel.read() return 0. Another important observation is all the tests I’ve ran are either HTTP or IIOP based protocol. Other protocols might perform differently, although I suspect it will not make such a difference.

Are you getting different results? This is not easy to measure because you have to make sure the framework itself is not the bottleneck.

As usual, feedback is more than welcome. Next time I will discuss using more than one Selector under high load. Merci!!

technorati:

_uacct = “UA-3111670-1″;
urchinTracker();

Categories: Uncategorized

The Grizzly Comet or why space shuttle Discovery launch was delayed.

Space shuttle Discovery was delayed recently, and the real reason was kept secret. Something strange was observed by the Hubble Space Telescope. The Hubble Ultra Deep Field(HUDF) image was showing a new star coming extremely fast to earth. Even after washing the main mirror with AJAX, the HUDF was clear: the Grizzly Comet is entering our atmosphere….

comet.jpg

This time I will discuss the new Comet support in Grizzly (sometimes called request polling, http streaming or continuation), build on top of Grizzly’s Asynchronous Request Processing(ARP). From Wikipedia:

Comet is a programming technique that enables web servers
to send data to the client without having any need for the client
to request for it. It allows creation of event-driven web 
applications which are hosted in the browser.

After I’ve blogged about ARP, I’ve started defining some Comet APIs on top of it. I was waiting for free time to define the perfect API. I think I was dreaming….I will never have free time with Grizzly! Since Comet Request processing is more and more popular (Jetty supports it for a while, Tomcat just have a fresh implementation…and a new NIO based connector (finally!!!), and GlassFish have ARP), I’ve decided to make available my own implementation. I didn’t update my implementation based on Greg Wilkins’ proposal, but hey, this is build on top of ARP and once the Servlet EG approves Comet support, it will be easy to implement it.

The next couple of paragraphs will introduce the API. I did the famous Chat implementation using jMaki, and will soon publish it once Greg reviewed my modifications to his jMaki application :-)

Comet API

A component (Servlet, JSP, JSF or a java class) that wants to support Comet requests first need to register to the CometEngine:


        CometEngine cometEngine = CometEngine.getEngine();
        CometContext context = cometEngine.register(contextPath);    

Mainly, you first get an instance of CometEngine, then register the context path on which Comet requests will be allowed. The CometContext is the main object a component will use to interact with others Comet requests, implemented as CometHandler.


         // Class that implement CometHandler interface
         CometResponseHandler handler = new CometResponseHandler();
         handler.attach(httpServletResponse);
         CometContext cometContext = 
            cometEngine.getCometContext(contextPath);
         cometContext.addCometHandler(handler);
         cometContext.notify("User X just entered the room");

The CometHandler interface is quite simple:


    /**
     * Attach an instance of E to this class.
     */
    public void attach(E attachment);
    
    /**
     * Receive CometEvent notification.
     */
    public void onEvent(CometEvent event) throws IOException;   
    
    /**
     * Receive CometEvent notification when the underlying 
     * tcp communication is started by the client
     */
    public void onInitialize(CometEvent event) throws IOException;   
    
    /**
     * Receive CometEvent notification when the underlying 
     * tcp communication is closed by the CometHandler
     */
    public void onTerminate(CometEvent event) throws IOException;    
      
    /**
     * Receive CometEvent notification when the underlying 
     * tcp communication is interrupted by the Grizzly ARP.
     */
    public void onInterrupt(CometEvent event) throws IOException;

Below is an example of a CometHandler implementation


    public class CometResponseHandler implements CometHandler{
        
        private HttpServletResponse httpServletResponse;
        
        public void attach(HttpServletResponse httpServletResponse){
            this.httpServletResponse = httpServletResponse;
        }
                
        public void onEvent(CometEvent event) throws IOException{   
            System.out.println("==== onEvent =====");
            try{
                PrintWriter printWriter = httpServletResponse.getWriter();
                // We just received a new chat message from another user.
                // Flush it to the browser.
                printWriter.println(event.attachment());
                printWriter.flush();
            } catch (Throwable t){
               t.printStackTrace(); 
            }  
        }

        public void onInitialize(CometEvent event) throws IOException{  
        }

        public void onTerminate(CometEvent event) throws IOException{
            onInterrupt(event);
        }

        public void onInterrupt(CometEvent event) throws IOException{
            CometContext cometContext = event.getCometContext();  
            cometContext.removeCometHandler(this);
        }        
    }

Once added to a CometContext, the CometHandler will be invoked everytime the CometContext.notify(Object) is invoked (ex: A new message is added to a Chat forum):


        cometContext.notify(new Message
          ("Grizzly Comet", userId + " has entered our atmosphere."));

The CometHandler can be invoked by the Grizzly ARP via the CometEngine and by others CometHandler. The CometEngine will notify CometHandler when the request is received (onInitialize) and when the request is interrupted (onInterrupt). The polled request will be resumed when the delay set on the CometContext expires:


         context.setExpirationDelay(60 * 1000); //60 seconds

Using the Chat example, a Servlet will use the CometContext to notify the CometHandler when a new chat user is added or when a new message is received:

    public void doPost(HttpServletRequest request, 
                       HttpServletResponse  response)
            throws IOException, ServletException {                      
         try {
            String action = request.getParameter("action");
            // negotiate a userid
            if ("valid-register".equals(action)) {
                ....
            } else if ("register".equals(action)) {
                ....
                CometEngine cometEngine = CometEngine.getEngine();
                CometContext cometContext = 
                        cometEngine.getCometContext(contextPath);
                cometContext.notify(
                   new Message("Chatter", userId + " has joined."));               
            } else if ("add-message".equals(action)) {
                .....
                CometEngine cometEngine = CometEngine.getEngine();
                CometContext cometContext = 
                        cometEngine.getCometContext(contextPath);
                cometContext.notify(new Message(userId,msg);
            }
            ....

From an AJAX application, an http connection will be openned and a CometHandler will be created using the HttpServletResponse object created for that connection. The CometHandler.onEvent(..) will wait for CometEvent, and based on the CometEvent.attachment(), will proceed by pushing back data to the AJAX application. The CometEvent.attachment() can be any type. For a Chat application, you will most probably attach a String containing the new message.

Technical details

In Grizzly ARP, the Comet request isn’t holding a Thread, but instead polled using NIO Selector (unfortunately using the SelectionKey.attach(..) for now ;-)). For scalability, it is very important to avoid holding a Thread during the request polling. Another important detail is the CometHandler doesn’t need to manage any Threads when notifying others CometHandler. This is implemented by the Grizzly ARP directly and not exposed to the CometHandler API. Of cource CometHandler can do whatever they want, hence they can decide to implement different strategies using Threads to notify others CometHandler.

The time where the polling happens can be configured. In the chat example, the polling will occurs after the Servlet.service() has been invoked. In the GMail/JavaMail example, the polling happens before Servlet.service(). You can decide when polling will happens using the CometEngine.register(..) method:


        CometEngine cometEngine = CometEngine.getEngine();
        CometContext context = 
           cometEngine.register(contextPath, 
                                CometEngine.BEFORE_REQUEST_PROCESSING);    

see CometEngine implementation for more details.

When notifying CometHandler, you can specify the type of CometEvent you want to push:


         CometEngine cometEngine = CometEngine.getEngine();
         CometContext cometContext = 
             cometEngine.getCometContext(contextPath);
         cometContext.notify("Closing the chat room",
                             CometEvent.TERMINATE);

see CometEvent for more details.

I’m still thinking about how the CometContext is retrieved from a component:


         cometContext = cometEngine.getCometContext(contextPath);

A Servlet from another application can easily compute the contextPath and retrieve the “external” CometContext. Although I can envision very interesting applications who are sharing CometContext and CometHandler, a security mechanism needs to be added to allow configuring the shareability of CometContext.

Finally, Grizzly ARP works with clean and SSL connection (both supported by NIO non blocking socket), thus Comet support can be used for secure and non secure communication.

comet-2.jpg

La suite des choses

The Comet support is fairly new and available only starting in GlassFish 9.1 ea-b10. That means the implementation most probably need improvements. As usual, any feedback is welcomed. I will soon add to the Grizzly workspace a couple examples on how to use the API with AJAX client. More important, if you don’t like the current implementation, it is very easy to extend Grizzly ARP with a completely different Comet support approach. I saw some implementation at JavaOne :-)

P.S To enable Comet Support, just add, in GlassFish domain.xml


        <http-listener acceptor-threads="1" address="0.0.0.0" 
           blocking-enabled="false" default-virtual-server="server"
           enabled="true" family="inet" id="http-listener-1" port="8080"
           security-enabled="false" server-name="" xpowered-by="true">
                <property name="cometSupport" value="true"/>
        </http-listener>

and make sure, in case you are using a Servlet|JSP, that it is initialized when GlassFish starts up by adding in your web.xml:


        <load-on-startup>0</load-on-startup>

UPDATED: A second blog that describe an example can be read here.

_uacct = “UA-3111670-1″;
urchinTracker();

technorati:

Categories: Uncategorized

Tricks and Tips with NIO part II: Why SelectionKey.attach() is evil

First, thanks for all the good feedback on part I. Please use the thread instead of sending me private email, as everybody can contribute to the answer (and I will not forgot to respond :-) ). Now If I can have the same kind of feedback for Grizzly code, I will be very happy (subliminal marketing here :-) )

OK this time I want to discuss the java.nio.channels.SelectionKey.attach(). I recommend you read about SelectionKey and the way you handle them before reading this blog. As usual, my first try might contains unclear parts (I should really start blogging in French instead :-) ).

The Java documentation for SelectionKey.attach() states:

Attaches the given object to this key.

An attached object may later be retrieved via the attachment method. Only one 
object may be attached at a time; invoking this method causes any previous 
attachment to be discarded. The current attachment may be discarded by 
attaching null.

Wow…the devil exists, and he lives inside the NIO API!


Why? Well, let takes a simple example. Usually, you handle SelectionKey by doing:


            selectorState = 0;
            enableSelectionKeys();                

            try{
                selectorState = selector.select(selectorTimeout);
            } catch (CancelledKeyException ex){
                ;
            }
            readyKeys = selector.selectedKeys();
            iterator = readyKeys.iterator();
            while (iterator.hasNext()) {
                key = iterator.next();
                iterator.remove();
                if (key.isValid()) {
                    handleConnection(key);
                } else {
                    cancelKey(key);
                }
            }

The Selector.select() will always return the set of SelectionKey whose ready-operation sets were updated. Then the handleConnection implementation will most likely looks like:


        if ((key.readyOps() & SelectionKey.OP_ACCEPT) == 
                SelectionKey.OP_ACCEPT){
            handleAccept(key);
        } else if ((key.readyOps() & SelectionKey.OP_READ) == 
                 SelectionKey.OP_READ) {
            handleRead(key);
        }

Next in handleRead(key), you will do:


            socketChannel = (SocketChannel)key.channel();
            while ( socketChannel.isOpen() && 
                    (count = socketChannel.read(byteBuffer))> -1)){
                // do something
            }

Well, the scary part is the // do something.

Gold Candidate for a Memory Leak (GCML)

At this stage, socketChannel is ready to read bytes. Hence you invoke socketChannel.read(byteBuffer), and you find that you haven’t read all the bytes from the socket (or you are ready to handle the next request), so you decide to register the SelectionKey back to the Selector by doing:


            selectionKey.interestOps(
                    selectionKey.interestOps() | SelectionKey.OP_READ);

and…and…and do something like:


            selectionKey.attach(...)

Boum…the little is where the devil is hiding! What you are attaching to the SelectionKey is very dangerous, because there is some probability that your SelectionKey might never return to a ready-operation state, leaving the SelectionKey and its evil attachment forever inside the Selector keys set. Does it sound like a GC…ML (GCML)? But what’s the point, nobody will ever do that, because we are all very talented engineers, and we always take care of cleaning our SelectionKey from the Selector keys set, right?

The problem might comes when your framework needs to handle thousand of connections, and you need to keep-alive those connections for a very long time (from 60 seconds to 5 minutes). Most framework (and unfortunately a lot of tutorials and talks) will usually attach their framework object to the SelectionKey (ex: The Reactor pattern). Those framework objects will most probably include:

  • A ByteBuffer
  • Some keep-alive object (let’s assume a Long)
  • A SocketChannel
  • A Framework Handler (like the Reactor pattern)
  • etc.

So you can ends up with thousand of objects taking vacations, enjoying idle time inside the Selector keys set. If you didn’t implement any mechanism to make periodical look inside the Selector keys set, then you will most probably ends up with a memory leak (or your framework performance will be impacted). Worse, you might never notice the problem….

BelleIsle-2.jpg

Recommended solutions?

My first experimentation of NIO was using that kind of approach (like Mina, like EmberIO and like our ORB NIO implementation). Then Scott started working with me on Grizzly and pointed the problems after a couple of benchmarks. Under HTTP stress, you might ends up with 10 000 connections (so 10 000 actives SelectionKey). If they all have as an attachment a ByteBuffer or an Handler, then a lot of memory will be consumed, reducing your scalability and having fun eating all your memory.

Even if most Virtual Machine are very good those days, I consider this as a very bad design anyway, unless you have a very good reason. But don’t get me wrong, I’m not saying the framework listed above are bad, I’m just pointing some problems.

The next couple of paragraphs will describe some solutions

How do I retrieve the SocketChannel if I don’t attach it to my framework object.

Most existing framework include, inside their framework object, the SocketChannel associated with the SelectionKey. This is wrong, because the SocketChannel can always be retrieved using SelectionKey.channel().

How do I deal with incomplete socketChannel read.

When you do socketChannel.read(), you can never predict when all bytes are read from the socket buffer. Most of the time, you will have to register the SelectionKey back to the Selector, waiting for more bytes to be available. In that case, you will most probably attach the incomplete ByteBuffer to the SelectionKey, and continue adding bytes to it once the SelectionKey is ready. Instead, I would recommend you register the SelectionKey to a temporary Selector (I will blog about this trick in more details):


        try{
            SocketChannel socketChannel = (SocketChannel)key.channel();
            while (count > 0){
                count = socketChannel.read(byteBuffer);
            }

            if ( byteRead == 0 ){
                readSelector = SelectorFactory.getSelector();
                tmpKey = socketChannel
                        .register(readSelector,SelectionKey.OP_READ);
                tmpKey.interestOps(tmpKey.interestOps() | SelectionKey.OP_READ);
                int code = readSelector.select(readTimeout);
                tmpKey.interestOps(
                    tmpKey.interestOps() & (~SelectionKey.OP_READ));

                if ( code == 0 ){
                    return 0; 
                }

                while (count > 0){
                    count = socketChannel.read(byteBuffer);
                }
            }
        } catch (Throwable t){

In this example, you try to read more bytes using a temporay Selector (on the same Thread, without having to return to your original Selector, which most of the time run on another thread). With this trick, you don’t need to attach anything to the SelectionKey.

But there is a drawback. If the temporary Selector.select() blocks (because the SelectionKey isn’t ready, most probably because the client isn’t sending all the bytes), you will block a processing Thread for “readTimeout” seconds, ending up in a similar well known situation called blocking socket ;-) (one thread per connection). That wouldn’t have been the case if you had registered the SelectionKey back the original Selector with a ByteBuffer attached. So here you gonna need to decide based on your use of NIO: do you want a dormant ByteBuffer attached to a SelectionKey or a Thread blocking for readTimeout.

In Grizzly, both approaches can be configured, but by default the thread will block for 15 seconds and cancel the SelectionKey if the client isn’t doing anything. You can configure Grizzly to attach the ByteBuffer if you really like to use memory :-) . We did try on slow network, with broken client, etc., and blocking a Thread scale better than having a dormant ByteBuffer,

Have one ByteBuffer per Thread, not per object framework.

The good news about not using SelectionKey.attach() is you only need to create a ByteBuffer per Thread, instead of a ByteBuffer per SelectionKey. So for 10 000 connections, instead of having 10 000 ByteBuffer, you will only have X ByteBuffer, where X = the number of active Threads. This significantly improve scalability by not overloading the VM with dormant ByteBuffer. As an example, in Grizzly, the keep-alive mechanism is implemented as follow:

  • Thread-1: Selector.select()
  • Thread-2: Do socketChannel.read() until the HTTP 1.1 request is fully read
  • Thread-2: Process the request and send the response
  • Thread-2: register the SelectionKey back to the Selector (without SelectionKey.attach())
  • Thread-1: Selector.select(). If the SelectionKey ready, then
  • Thread-2: Do socketChannel.read() until the HTTP 1.1 request is fully read
  • etc.

As you can see, Thread-2 is not blocked between keep-alive requests. Theorically, you would probably be able to serve hundreds of requests with only two threads. Not to say no dormant ByteBuffer, no pending framework objects, etc.

Wow that one was very long. Agree, disagree?…..

Next time I will discuss when its appropriate to spawn a thread to handle OP_ACCEPT, OP_READ and OP_WRITE, and when its not.

_uacct = “UA-3111670-1″;
urchinTracker();

technorati:

Categories: Uncategorized
Follow

Get every new post delivered to your Inbox.

Join 50 other followers