Search Results

Keyword: ‘"async http client"’

Async Http Client 1.2.0 released

Read the official announcement including the changes log

http://codemeself.blogspot.com/2010/10/async-http-client-120.html

Thanks to every body that contributed to that amazing release.

For any questions you can use our Google Group, on irc.freenode.net #asynchttpclient or use Twitter to reach me! You can checkout the code on Github, download the jars from Maven Central or use Maven:

<dependency>
    <groupId>com.ning</groupId>
    <artifactId>async-http-client</artifactId>
    <version>1.2.0</version>
</dependency>

Categories: Uncategorized

Async Http Client 1.1.0 Released

September 4, 2010 3 comments

After months of collecting feedback and adding new features, I’m happy to announce the availability of the Async Http Client Library version 1.1.0!

We also promoted the project to it’s own github organization (since I’m now working for Sonatype :-))

http://github.com/AsyncHttpClient/async-http-client/

We got a lot of feedback from the community and worked hard to include all the requirements. You can see the library in action in Play Framework, JClouds, Atmosphere, Apache Camel, EhCache, Maven (coming soon), ahc-cl (Clojure) and many more! Get started by looking at our super simple sample.

What’s new with this version:

  • Connection pool is unlimited by default
  • Basic and Digest Authentication supported out-of-the-box
  • Multi Headers support
  • Removal of Google Collection and Log4j dependency (significantly reduce the jar)
  • Latest Netty version (3.3.2)
  • OAuth signature now supported
  • Proxy per request now supported
  • Performance improved
  • JDK Uri/Url now used instead of our own Url class.

Complete list of what has been fixed and added can be read here and here. Change logs available here.

For any questions you can use our Google Group, on irc.freenode.net #asynchttpclient or use Twitter to reach me! You can checkout the code on Github, download the jars from Maven Central or use Maven:

<dependency>
    <groupId>com.ning</groupId>
    <artifactId>async-http-client</artifactId>
    <version>1.1.0</version>
</dependency>

Async Http Client 1.0.0 released

A few month ago, we, at Ning, open sourced our Async Http Client  library:

Today we are announcing the 1.0.0 GA availability. This release includes many improvements, bug fixes and features based on our community’s feedback. You can browse the complete (and long) change logs:

http://github.com/ning/async-http-client/commits/master/

In short:

  • Default Async Provider powered by Netty 3.2.1.Final (just released!)
  • Support both http and https
  • Support the majority of  HTTP method like OPTIONS, GET, PUT, POST, DELETE, etc.
  • Works well with JRuby, Python, Groovy, etc.
  • Works well with asynchronous WebServer and Framework like Jetty Continuation, Atmosphere, etc.
  • Many useful utilities to optimally manipulate the request/response.
  • More than 100 unit tests you can look at to get started.
  • And many many more.

You can now use the following dependency

<dependency>
<groupId>com.ning</groupId>
<artifactId>async-http-client</artifactId>
<version>1.0.0</version>
</dependency>

or clone our repository from:

http://github.com/AsyncHttpClient/async-http-client

Our next release roadmap can be browsed from here:

http://groups.google.com/group/asynchttpclient/web/road-map-to-1-0-0-…

Finally, if you are interested to contribute or have any questions, join our group here:

http://groups.google.com/group/asynchttpclient

Thanks to all the contributors from Ning and outside Ning!

Categories: Atmosphere, Ning

Going Asynchronous using AsyncHttpClient: The Complex

The Async Http Client library purpose is to allow Java applications to easily execute HTTP requests and asynchronously process the HTTP responses. In this second part on the topic, I will describe more complex operations that can be done with the AsyncHttpClient like resumable download, zero in memory bytes copy, oAuth calculation, optimal transfer listener and performance tricks.

Resumable Donwload

The AsyncHttpClient supports resumable download in two differents scenarios:

  • IOException: If an IOException occurs (for whatever reason), you can configure the library to restart the download automatically without having to restart the download from the beginning.
  • JVM crashes: If your application or the JVM goes down during a file download, the library can also restart the download automatically when the same download is requested.

You can configure the AsyncHttpClient Library to survive IOException using the IOException Filter:

  AsyncHttpClient c = new AsyncHttpClient(
          new AsyncHttpClientConfig.Builder()
            .addIOExceptionFilter(
               new ResumableIOExceptionFilter()).build());

  ResumableAsyncHandler a =
      new ResumableAsyncHandler(
         new ResumableRandomAccessFileListener());

  a.setResumableListener(
       new ResumableRandomAccessFileListener(
           new RandomAccessFile( "file.avi", "rw" ) ) );

 Response r = c.prepareGet("http://host:port/file.avi")
       .execute(a).get();

If you need something more high level and configurable, you can use a ResumableAsyncHandler, and or implement a ResumableProcessor:

 AsyncHttpClient c = new AsyncHttpClient();
 ResumableAsyncHandler a =
     new ResumableAsyncHandler( new PropertiesBasedResumableProcessor() );
 a.setResumableListener(
     new ResumableRandomAccessFileListener(
        new RandomAccessFile( "file.avi", "rw" ) ) );

 Response r = c.prepareGet( "http://localhost:8081/file.AVI" )
        .execute( a ).get();

You can also simply use a ResumableListener (or use the ResumableRandomAccessFileListener, which does what’s described below):

public interface ResumableListener {

    public void onBytesReceived(ByteBuffer byteBuffer) throws IOException;

    public void onAllBytesReceived();

    public long length();
}

As simple as:

 AsyncHttpClient c = new AsyncHttpClient();
 final RandomAccessFile file = new RandomAccessFile( "file.avi", "rw" );        
 ResumableAsyncHandler a = new ResumableAsyncHandler();
 a.setResumableListener( new ResumableListener() {

    public void onBytesReceived(ByteBuffer byteBuffer) throws IOException {
       file.seek( file.length() );
       file.write( byteBuffer.array() );
    }

    public void onAllBytesReceived() {
       file.close();
    }

    public long length() {
       return file.length();
    }
} );
Response r = c.prepareGet( "http://localhost:8081/file.AVI" )
        .execute( a ).get();

Make it simple: TransferListener

In some scenario an application may need to manipulate the received bytes in more than one place, e.g. saves the bytes on disk but also accumulate it for checksum checking later. In that case, instead of using an AsyncHandler and mixes logic inside an AsyncHandler.onBodyPartReceived, it is recommended to use the TransferListener simple API:

public interface TransferListener {

    public void onRequestHeadersSent
       (FluentCaseInsensitiveStringsMap headers);

    public void onResponseHeadersReceived
        (FluentCaseInsensitiveStringsMap headers);

    public void onBytesReceived(ByteBuffer buffer)
         throws IOException;

    public void onBytesSent(ByteBuffer buffer);

    public void onRequestResponseCompleted();

    public void onThrowable(Throwable t);
}

All you need to do in that case is to create a TransferCompletionHandler and add as many TransferListener as you need:

 AsyncHttpClient client = new AsyncHttpClient();
 TransferCompletionHandler tl = new TransferCompletionHandler();
 tl.addTransferListener(new TransferListener(){...});

 Response response = httpClient.prepareGet("http://...").execute(tl).get();

Zero Bytes Copy

When uploading or downloading bytes, it is important to try to avoid buffering bytes in memory.

Upload

On the upload side, the mechanism is enabled by default when setting the Request’s body to a File:

 AsyncHttpClient client = new AsyncHttpClient();

 File file = new File("file.avi");
 Future f = client.preparePut("http://localhost").setBody(file).execute();

If you can’t use a File, the recommended way (as described in part I) is to use a BodyGenerator. It is strongly recommended to avoid using InputStream as the library will unfortunately buffer the entire content in memory in order to set the content-lenght, which can cause out of memory error.

Download

On the download side, you can use the HttpResponseBodyPart.writeTo to avoid loading bytes in memory and unnecessary copy:

 AsyncHttpClient client = new AsyncHttpClient();

 File tmp = new File("zeroCopy.txt");
 final FileOutputStream stream = new FileOutputStream(tmp);
 Future f = client.prepareGet("http://localhost/largefile.avi")
       .execute(new AsyncHandler() {
     public void onThrowable(Throwable t) {
     }

     public STATE onBodyPartReceived(HttpResponseBodyPart bodyPart)
         throws Exception {
        bodyPart.writeTo(stream);
        return STATE.CONTINUE;
     }

     { .... }
});
Response resp = f.get();

Limiting the number of connections to improve raw performance

By default the library uses a connection pool and re-use connections as needed. It is important to not let the connection pool grow too large as it takes resources in memory. One way consist of setting the maximum number of connection per host or in total:

 AsyncHttpClientConfig config = new AsyncHttpClientConfig.Builder()
     .setMaximumConnectionsPerHost(10)
     .setMaximumConnectionsTotal(100)
     .build();
 AsyncHttpClient c = new AsyncHttpClient(config);

There is no magic number, so you will need to try it and decide which one gives the best result.

Using OAuth

You can use the library to pull data from any OAuth site (like Twitter). This is as simple as:

 private static final String CONSUMER_KEY = "dpf43f3p2l4k3l03";
 private static final String CONSUMER_SECRET = "kd94hf93k423kf44";
 public static final String TOKEN_KEY = "nnch734d00sl2jdk";
 public static final String TOKEN_SECRET = "pfkkdhi9sl3r4s00";
 public static final String NONCE = "kllo9940pd9333jh";
 final static long TIMESTAMP = 1191242096;

 public void oAuth() {
   ConsumerKey consumer =
     new ConsumerKey(CONSUMER_KEY, CONSUMER_SECRET);
   RequestToken user =
     new RequestToken(TOKEN_KEY, TOKEN_SECRET);
   OAuthSignatureCalculator calc =
      new OAuthSignatureCalculator(consumer, user);
   AsyncHttpClient client = new AsyncHttpClient();

   Response response = client.prepareGet("http://...")
      .setSignatureCalculator(calc).execute().get();

What’s Next

In the next part of this series I will explain more complex operations that can be done with the AsyncHttpClient library like:

  • Supporting the WebSocket protocol.
  • Concurrent use of AsyncHandler

For any questions you can use our Google Group, irc.freenode.net #asynchttpclient or use Twitter to reach me! You can checkout the code on Github, download the jars from Maven Central or use Maven:

<dependency>
    <groupId>com.ning</groupId>
    <artifactId>async-http-client</artifactId>
    <version>1.4.1</version>
</dependency>

Categories: Async Http client

Going Asynchronous using AsyncHttpClient: The Basic

December 21, 2010 12 comments

The Async Http Client library purpose is to allow Java applications to easily execute HTTP requests and asynchronously process the HTTP responses. In this blog I will explain how to use the library and what features are supported.

Executing request synchronously or asynchronously.

The first thing to decide when using the library is if your application can handle asynchronous response or not. If not, the library has been designed using the Future API, hence you can always execute synchronous call by blocking on the Future.get() method:

   AsyncHttpClient client = new AsyncHttpClient();
   Response response = client.prepareGet(("http://sonatype.com")
     .execute().get();

The above means the request will block until the full Response has been received. It also made your application’s blocking, waiting for the response to comes back. This could be potentially an issue to block for every request, specially when doing POST or PUT operations where you don’t necessarily need to wait for the response. A simple way consists of not calling the Future.get()

AsyncHttpClient client = new AsyncHttpClient();
Response response =
   client.preparePut(("http://sonatype.com/myFile.avi").execute();

A better way than above would consist of using an AsyncHandler. The AynchHandler API is fairly simple and just consist of 5 methods to implements:

public interface AsyncHandler<T>  {
    void onThrowable(Throwable t);

    STATE onBodyPartReceived(HttpResponseBodyPart bodyPart)
      throws Exception;

    STATE onStatusReceived(HttpResponseStatus responseStatus)
      throws Exception;

    STATE onHeadersReceived(HttpResponseHeaders headers)
      throws Exception;

    T onCompleted() throws Exception;
}

The method’s order of invocation when the response start arriving consist of:

  1. onStatusReceived: The status line has been processed.
  2. onHeadersReceived: All response’s headers has been processed.
  3. onBodyPartReceived: A body parts has been received. This method can be invoked many time depending of the response’s bytes body.
  4. onCompleted: Invoked when the full response has been read, or if the processing get aborted (more on this below)
  5. onThrowable: Invoked if something wrong happened inside the previous methods or when an I/O exception occurs.

Note that for all methods onXXXReceived, the return value is an enum which can take the value of CONTINUE or ABORT. Returning CONTINUE tells the library to continue processing the response, where ABORT means stop processing the response and automatically invoke the onCompleted(). This is particularly helpful if your application just need to looks for the response’s status or headers, without the need to process the entire response’s body. An implementation would looks like (T can be anything):

AsyncHttpClient client = new AsyncHttpClient();
client.prepareGet("http://sonatype.com")
  .execute(new AsyncHandler<T>() {

     void onThrowable(Throwable t) {
     }
     public STATE onBodyPartReceived(HttpResponseBodyPart bodyPart)
       throws Exception{
        return STATE.CONTINUE;
     }
     public STATE onStatusReceived(HttpResponseStatus responseStatus)
       throws Exception {
        return STATE.CONTINUE;
     }
     public STATE onHeadersReceived(HttpResponseHeaders headers)
       throws Exception {
        return STATE.CONTINUE;
     }
     T onCompleted() throws Exception {
       return T;
   }
});

Creating a Request object

The AsynHttpClient uses the builder pattern when it is time to create Request object. The simplest way consist of:

    RequestBuilder builder = new RequestBuilder("PUT");
    Request request = builder..setUrl("http://")
     .addHeader("name", "value")
     .setBody(new File("myUpload.avi"))
     .build();
    AsyncHttpClient client = new AsyncHttpClient();
    client.execute(request, new AsyncHandler&lt;...&gt;() {
         .....
    } );

If you need to work with File, the library supports the zero copy in memory concept, e.g the File can be uploaded or downloaded without loading its associated bytes in memory, preventing out of memory errors in case you need to upload or download many large files. Although the library support the following:

    Request request = builder..setUrl("http://")
      .addHeader("name", "value")
      .setBody(myInputStream))
      .build();

it is discouraged to use InputStream as the library will need to buffer bytes in memory in order to determine the length of the stream, and instead highly recommended to either use a File or the BodyGenerator API to avoid loading unnecessary bytes in memory:

 public interface BodyGenerator {
    Body createBody() throws IOException;
}

where a Body is defined as:

public interface Body {
    long getContentLength();
    long read(ByteBuffer buffer)
      throws IOException;
    void close() throws IOException;
}

This way the library will never read unnecessary bytes in memory, which could significantly improve the performance your application.

The RequestBuilder can also be used to create per Request configuration, like setting a Proxy or request timeout:

    PerRequestConfig requestConfig = new PerRequestConfig();
    requestConfig.setRequestTimeoutInMs(5 * 1000);
    requestConfig.setProxy(new ProxyServer(...));
    Future responseFuture =
     client.prepareGet("http://").setPerRequestConfig(requestConfig)
        .execute();

Creating a Response object

The AsyncHandler is typed, e.g you can return any object from the AsyncHandler.onCompleted(). One useful object of the library is the Response object and it’s associate builder. You can incrementally create a Response object using the ResponseBuilder.accumulate() method:

MyAsyncHandler<Response> asyncHandler = new MyAsyncHanfler<Response>() {
  private final Response.ResponseBuilder builder =
          new Response.ResponseBuilder();

  public STATE onBodyPartReceived(final HttpResponseBodyPart content)
    throws Exception {
      builder.accumulate(content);
      return STATE.CONTINUE;
  }

  public STATE onStatusReceived(final HttpResponseStatus status)
     throws Exception {
      builder.accumulate(status);
      return STATE.CONTINUE;
  }

  public STATE onHeadersReceived(final HttpResponseHeaders headers)
     throws Exception {
      builder.accumulate(headers);
      return STATE.CONTINUE;
  }

  public Response onCompleted() throws Exception {
      return builder.build();
  }
}

Response response = client.prepareGet("http://sonatype.com")
     .execute(asyncHandler).get();

One thing to consider when creating a Response object is the size of the response body. By default, a Response object will accumulate all response’s bytes in memory, and that could potentially create an out of memory error. If you are planning to use the API for downloading large files, it is not recommended to accumulate bytes in memory and instead flush the bytes on disk as soon as they are available. Note that you can still use the Response object, except you don’t accumulate the response’s bytes as demonstrated below:

MyAsyncHandler<Response> asyncHandler = new MyAsyncHanfler<Response>() {
   private final Response.ResponseBuilder builder =
      new Response.ResponseBuilder();

   public STATE onBodyPartReceived(final HttpResponseBodyPart content)
     throws Exception {
       content.write(myOutputStream);
       return STATE.CONTINUE;
   }

   public STATE onStatusReceived(final HttpResponseStatus status)
     throws Exception {
       builder.accumulate(status);
       return STATE.CONTINUE;
   }

   public STATE onHeadersReceived(final HttpResponseHeaders headers)
      throws Exception {
       builder.accumulate(headers);
       return STATE.CONTINUE;
   }

   public Response onCompleted() throws Exception {
       return builder.build();
   }
}

Response response = client.prepareGet("http://sonatype.com")
   .execute(asyncHandler).get();

Note that in the above scenario invoking Response.getResponseBodyAsStream() or getResponseBody() will return an IllegalStateException because the body wasn’t accumulated by the Response object.

Configuring the AsyncHttpClient: Compression, Connection Pool, Proxy, Times out, Thread Pools, Security, etc.

You can configure the AsyncHttpClient class using the AsyncHttpClientConfig’s Builder:

    Builder builder = new AsyncHttpClientConfig.Builder();
    builder.setCompressionEnabled(true)
        .setAllowPoolingConnection(true)
        .setRequestTimesout(30000)
        .build();
    AsyncHttpClient client = new AsyncHttpClient(builder.build());

You can set the ExecutorServices as well if you don’t want to use the default, which is a cached threads pool:

    Builder builder = new AsyncHttpClientConfig.Builder();
    builder.setExecutorService(myOwnThreadPool);
    AsyncHttpClient client = new AsyncHttpClient(builder.build());

You can also configure the connection pool the library is using and implement your own polling strategy:

    Builder builder = new AsyncHttpClientConfig.Builder();
    builder.setConnectionsPool(new ConnectionsPoo<U,V>() {
          public boolean offer(U uri, V connection) {...}
          public V poll(U uri)  {...}
          public boolean removeAll(V connection)  {...}
          public boolean canCacheConnection()  {...}
          public void destroy()  {...}
     });
    AsyncHttpClient client = new AsyncHttpClient(builder.build());

It is recommended to use the default connections pool for performance reason, but you are always free to design a better one.
You can also set the SSL information, Filters, etc. Those topics will be covered inside their own section.

Configuring SSL

Configuring the library to support SSL is simple. By default you don’t have to configure anything if you don’t need to use your own certificates etc.

   AsyncHttpClient client = new AsyncHttpClient();
   Response response = client.prepareGet(("https://sonatype.com")
     .execute().get();

The library will detect it’s an SSL request and appropriately locate the key store, trust store etc. If you need to configure those objects, all you need to do is to create an SSLContext and set it using the AsyncHttpClient’s Builder as showed below:

  InputStream keyStoreStream = ....
  char[] keyStorePassword = "changeit".toCharArray();
  KeyStore ks = KeyStore.getInstance("JKS");
  ks.load(keyStoreStream, keyStorePassword);

  char[] certificatePassword = "changeit".toCharArray();
  KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509");
  kmf.init(ks, certificatePassword);

  KeyManager[] keyManagers = kmf.getKeyManagers();
  TrustManager[] trustManagers = new TrustManager[]{DUMMY_TRUST_MANAGER};
  SecureRandom secureRandom = new SecureRandom();

  SSLContext sslContext = SSLContext.getInstance("TLS");
  sslContext.init(keyManagers, trustManagers, secureRandom);
  Builder builder = new AsyncHttpClientConfig.Builder();
  builder.setSSLContext(myOwnThreadPool);
  AsyncHttpClient client = new AsyncHttpClient(builder.build());

Using Filters

The library supports three types of Filter who can intercept, transform, decorate and replay transactions: Request, Response and IOException.

Request Filter

Request Filters are useful if you need to manipulate the Request or AsyncHandler object before the request is made. As an example, you can throttle requests using the following RequestFilter implementation:

public class ThrottleRequestFilter implements RequestFilter {
    private final int maxConnections;
    private final Semaphore available;
    private final int maxWait;

    public ThrottleRequestFilter(int maxConnections) {
        this.maxConnections = maxConnections;
        this.maxWait = Integer.MAX_VALUE;
        available = new Semaphore(maxConnections, true);
    }

    public ThrottleRequestFilter(int maxConnections, int maxWait) {
        this.maxConnections = maxConnections;
        this.maxWait = maxWait;
        available = new Semaphore(maxConnections, true);
    }

    public FilterContext filter(FilterContext ctx) throws FilterException {

        try {
            if (!available.tryAcquire(maxWait, TimeUnit.MILLISECONDS)) {
                throw new FilterException(
                    String.format("No slot available for Request %s "
                            "with AsyncHandler %s",
                            ctx.getRequest(), ctx.getAsyncHandler()));
            };
        } catch (InterruptedException e) {
            throw new FilterException(
                    String.format("Interrupted Request %s" +
                         "with AsyncHandler %s",
                            ctx.getRequest(), ctx.getAsyncHandler()));
        }

        return new FilterContext(
             new AsyncHandlerWrapper(ctx.getAsyncHandler()), ctx.getRequest());
    }

    private class AsyncHandlerWrapper implements AsyncHandler<T> {

        private final AsyncHandler asyncHandler;

        public AsyncHandlerWrapper(AsyncHandler asyncHandler) {
            this.asyncHandler = asyncHandler;
        }

        public void onThrowable(Throwable t) {
            asyncHandler.onThrowable(t);
        }

        public STATE onBodyPartReceived(HttpResponseBodyPart bodyPart)
                throws Exception {
            return asyncHandler.onBodyPartReceived(bodyPart);
        }

        public STATE onStatusReceived(HttpResponseStatus responseStatus)
              throws Exception {
            return asyncHandler.onStatusReceived(responseStatus);
        }

        public STATE onHeadersReceived(HttpResponseHeaders headers)
              throws Exception {
            return asyncHandler.onHeadersReceived(headers);
        }

        public T onCompleted() throws Exception {
            available.release();
            return asyncHandler.onCompleted();
        }
    }

In the above, we decorate the original AsyncHandler and use semaphore to throttle requests. To add RequestFilter, all you need to do is to configure it on the AsyncHttpClientConfig:

 AsyncHttpClientConfig.Builder b =
                 new AsyncHttpClientConfig.Builder();
 b.addRequestFilter(new ThrottleRequestFilter(100));
 AsyncHttpClient c = new AsyncHttpClient(b.build());

Response Filter

Like with Request, you can also filter the Response’s bytes before an AsyncHandler gets called. Response Filters are always invoked before the library executes the logic for authentication, proxy challenging, redirection etc. That means an application can takes control of those operations at any moment using a Response Filter. As an example, the following Response Filter redirect request from google.ca to google.com in case .ca is not responding:

 AsyncHttpClientConfig.Builder b = new AsyncHttpClientConfig.Builder();
 b.addResponseFilter(new ResponseFilter() {

   public FilterContext filter(FilterContext ctx) throws FilterException {

      if ( ctx.getResponseStatus().getStatusCode() == 503 ) {
          return new FilterContext.FilterContextBuilder(ctx)
                       .request(new RequestBuilder("GET")
                       .setUrl("http://google.com").build())
                       .build();
                 }
            }});

 AsyncHttpClient c = new AsyncHttpClient(b.build());

IOException Filter

The AsyncHttpClient library support IOExceptionFilter that can be used to replay a request in case server a server goes down or unresponsive, a network outage occurs, or nay kind of I/O abnormal situation. In those cases, the library will catch the IOException and delegate the IOException handling to the Filter. As an example, the following filter will resume an interrupted download instead of restarting downloading the file from the beginning:

    AsyncHttpClient c = new AsyncHttpClient(
          new AsyncHttpClientConfig.Builder()
            .addIOExceptionFilter(
              new ResumableIOExceptionFilter()).build());

    Response r = c.prepareGet("http://host:port/LargeFile.avi")
       .execute(new AsyncHandler(){...}).get();

The IOExceptionFilter is defined as

public class ResumableIOExceptionFilter implements IOExceptionFilter {
    public FilterContext filter(FilterContext ctx) throws FilterException {
        if (ctx.getIOException() != null ) {
            Request request = new RequestBuilder(ctx.getRequest())
                .setRangeOffset(file.length());
            return new FilterContext.FilterContextBuilder(ctx)
               .request(request)
               .replayRequest(true)
               .build();
        }
        return ctx;
    }
}

In the above we just catch any IOException and replay the request using the Range header to tell the remote server to restart sending bytes at that position. This way we don’t need to re download the entire file.

Uploading file: Progress Listener

When uploading bytes, an application might need to take some action depending on where the upload status is. The AsyncHttpClient library support a special AsyncHandler called ProgressAsyncHandler that can be used to track the upload operation:

public interface ProgressAsyncHandler<T> extends AsyncHandler<T> {
    STATE onHeaderWriteCompleted();
    STATE onContentWriteCompleted();
    STATE onContentWriteProgress(long amount, long current, long total);
}

The methods are called in the following order:

  • onHeaderWriteCompleted: invoked when the headers has been flushed to the remote server
  • onContentWriteProgress: as soon as some response’s body bytes are written. Might be invoked many times.
  • onContentWriteCompleted: invoked when the response has been sent or aborted.

Like with AsyncHandler, you can always always abort the processing at any moment in the upload process

Configuring Authentication: BASIC, DIGEST or NTLM

Configuring authentication with AsyncHttpClient is simple. You can configure it at the Request level using the RealmBuilder:

    AsyncHttpClient client = new AsyncHttpClient();
    Realm realm = new Realm.RealmBuilder()
               .setPrincipal(user)
               .setPassword(admin)
               .setUsePreemptiveAuth(true)
               .setScheme(AuthScheme.BASIC)
               .build();
    client.prepareGet("http://...").setRealm(realm).execute();

You can also set the realm at the AsyncHttpClientConfig level:

    Builder builder = new AsyncHttpClientConfig.Builder();
    Realm realm = new Realm.RealmBuilder()
               .setPrincipal(user)
               .setPassword(admin)
               .setUsePreemptiveAuth(true)
               .setScheme(AuthScheme.BASIC)
               .build();
    builder.setRealm(realm).build();
    AsyncHttpClient client = new AsyncHttpClient(builder.build());

The authentication type supported are BASIC, DIGEST and NTLM. You can also customize your own authentication mechanism by using the Response Filter.

Configuring a Proxy

The AsyncHttpClient library supports proxy, proxy authentication and proxy tunneling. Just need to create a ProxyServer instance:

AsyncHttpClient client = new AsyncHttpClient();
        Future<Response> f = client
                .prepareGet("http://....)
                .setProxyServer(new ProxyServer("127.0.0.1", 8080))
                .execute();

If you need to use an SSL tunnel, all you need to do is:

 ProxyServer ps =
          new ProxyServer(ProxyServer.Protocol.HTTPS, "127.0.0.1", 8080);
 AsyncHttpClient asyncHttpClient = new AsyncHttpClient();
 RequestBuilder rb = new RequestBuilder("GET")
             .setProxyServer(ps)
             .setUrl("https://twitpic.com:443");

  Future responseFuture = asyncHttpClient
               .executeRequest(rb.build(), new AsyncCompletionHandlerBase() {
    @Override
    public void onThrowable(Throwable t) {}

    @Override
    public Response onCompleted(Response response) throws Exception {
      return response;
    }});

   Response r = responseFuture.get();

You can also set the authentication token on the ProxyServer instance

 ProxyServer ps = new ProxyServer(ProxyServer.Protocol.HTTPS,
                                  "127.0.0.1",
                                  8080,
                                  "admin",
                                  "password");
  AsyncHttpClient asyncHttpClient = new AsyncHttpClient();
  RequestBuilder rb = new RequestBuilder("GET")
        .setProxyServer(ps).setUrl("https://twitpic.com:443");

  Future responseFuture = asyncHttpClient
               .executeRequest(rb.build(), new AsyncCompletionHandlerBase() {
    @Override
    public void onThrowable(Throwable t) {}

    @Override
    public Response onCompleted(Response response) throws Exception {
      return response;
    }});

  Response r = responseFuture.get();

You can also set the ProxyServer at the AsyncHttpClientConfig level. In that case, all request will share the same proxy information.

Switching Provider

By default, the AsyncHttpClient is using the powerful Netty’s framework as the HTTP processor. There might be environment where you can’t use Netty. Fortunately, the AsyncHttpClient library supports two other http runtime: the JDKAsyncHttpProvider, which build around the  URLConnection, and ApacheAsyncHttpProvider which build on top of the Apache HttpClient. To change provider, all you need to do is:

 AsyncHttpClient client = new AsyncHttpClient(
    new ApacheAsyncHttpProvider(new AsyncHttpClientConfig.Builder().build()));

Same for the JDK:

 AsyncHttpclient client = new AsyncHttpClient(
    new JDKAsyncHttpProvider(new AsyncHttpClientConfig.Builder().build()));

Also every AsyncHttpClientProvider can be configured with their native functionality. As an example, you can switch the NettyAsyncHttpProvider to use blocking I/O instead of NIO:

 NettyAsyncHttpProviderConfig config = new NettyAsyncHttpProviderConfig();
 config.setProperty(NettyAsyncHttpProviderConfig.USE_BLOCKING_IO, "true");

 AsyncHttpClientConfig c = 
    new AsyncHttpClientConfig()
      .setAsyncHttpClientProviderConfig(config).build();

 AsyncHttpClient client = new AsyncHttpClient(
    new NettyAsyncHttpProvider(config));

Using the WebDav protocol

The AsyncHttpClient has build in support for the WebDav protocol. The API can be used the same way normal HTTP request are made, and everything discussed in this blog works with WebDAV as well:

 AsyncHttpClient c = new AsyncHttpClient();
 Request mkcolRequest = new RequestBuilder("MKCOL")
      .setUrl("http://host:port/folder1").build();
 Response response = c.executeRequest(mkcolRequest).get();

or

  AsyncHttpClient c = new AsyncHttpClient();
  Request propFindRequest = new RequestBuilder("PROPFIND")
     .setUrl("http://host:port).build();
  Response response = c.executeRequest(propFindRequest, new AsyncHandler(){...}).get();

Using the ready to go AsyncHandler

The framework is shipping with ready to go AsyncHandler:

  • The AsyncCompletionHandlerBase is an implementation of the AsyncCompletionHandler&lt;Response&gt;. That means Future.get() will always return a instance of Response.
  • The WebDavCompletionHandlerBase can be used to parse the XML response of a PROPFIND request. As an example, response.getStatusCode() will return 200 instead of 207 if you don’t use that AsyncHandler.
  • The ResumableAsyncHandler can be used to survive interrupted download, either produced by an IOException or because the JVM went down. This handler will be explained in details in the next blog about AsyncHttpClient.

What’s Next

In the next part of this article (here)  I will explain more complex operations that can be done with the AsyncHttpClient library like:

  • Configuring resumable download using the ResumableAsyncHandler, ResumableProcessor and ResumableListener
  • How to efficiently uses the TransferListener
  • How to efficiently uses the zero bytes copy mechanism
  • How to configure the AsyncHttpClient for performance
  • How to manage 100-Continue server response
  • OAuth build in support
  • Supporting the WebSocket protocol.

For any questions you can use our Google Group, irc.freenode.net #asynchttpclient or use Twitter to reach me! You can checkout the code on Github, download the jars from Maven Central or use Maven:

<dependency>
    <groupId>com.ning</groupId>
    <artifactId>async-http-client</artifactId>
    <version>1.4.1</version>
</dependency>

Leaving Ning

Today I’ve officially resigned from Ning. Ning was a wonderful place to work, but I wanted to spend more time with my three little monsters and avoid traveling to California so often. I’ve never worked in a team of skilled architects like that and I will miss all the learning I was doing every day. Thanks to all of you and I’m sure Ning will be a great success!! I hope we all keep in touch!


What I’m gonna do? My quest for my next company start this week … things may move quite fast as you all know :-) For sure I will take a couple of weeks off as I’ve left Sun on a Friday and the next Monday I was at Ning. Bad idea but I was so trilled to join a great place like Ning!

I will not disappear completely as I can’t stop improving my Atmosphere Framework and support our growwwwwing community..so I will not be 100% on vacation. I also want to explore Akka as this project is so interesting and the community there is just awesome, and i can be dangerous as I have commit access :-)! I will also continue to actively work on the Async Http Client…actually I will spend more time on it now!!.

So, Just follow me on Twitter for summer news :-).

Categories: Uncategorized

New Open Source Project Alert! A New Asynchronous Http Client library!

Introducing the new Asynchronous HTTP Client library, which allow applications to execute HTTP requests and asynchronously process the HTTP response .

IMG_0395.jpg

Read the official announcement here. Brian McCalister, Thomas Dudziak and I worked on that library since I’ve joinned NIng back in December.

The library purpose is to allow Java applications to easily execute HTTP requests and asynchronously process the HTTP responses. You can get it at

http://github.com/AsyncHttpClient/async-http-client

The Async HTTP Client library is simple to use. First, in order to add it to your Maven project, simply add this dependency:

<dependency>
    <groupId>com.ning</groupId>
    <artifactId>async-http-client</artifactId>
    <version>1.0.0-SNAPSHOT</version>
</dependency>

For now, you’ll have to add the Sonatype snapshot repo to your settings in order to be able to access the snapshot builds:

http://oss.sonatype.org/service/local/repositories/snapshots/content

Then in your code you can simply do:

    import com.ning.http.client.*;
    import java.util.concurrent.Future;

    AsyncHttpClient asyncHttpClient = new AsyncHttpClient();
    Future<Response> f = asyncHttpClient.prepareGet("http://www.ning.com/").execute();
    Response r = f.get();

You can also accomplish asynchronous operation without using a Future if you want to receive and process the response in your handler:

    import com.ning.http.client.*;
    import java.util.concurrent.Future;

    AsyncHttpClient asyncHttpClient = new AsyncHttpClient();
    asyncHttpClient.prepareGet("http://www.ning.com/").execute(new AsyncCompletionHandler(){

        @Override
        public Response onCompleted(Response response) throws Exception{
            // Do something with the Response
            // ...                                   `
            return response;
        }

        @Override
        public void onThrowable(Throwable t){
            // Something wrong happened.
        }
    });

You can also mix Future with AsyncHandler to only retrieve part of the asynchronous response

    import com.ning.http.client.*;
    import java.util.concurrent.Future;

    AsyncHttpClient asyncHttpClient = new AsyncHttpClient();
    Future<Integer> f = asyncHttpClient.prepareGet("http://www.ning.com/")
          .execute(new AsyncCompletionHandler(){

        @Override
        public Integer onCompleted(Response response) throws Exception{
            // Do something with the Response
            return response.getStatusCode();
        }

        @Override
        public void onThrowable(Throwable t){
            // Something wrong happened.
        }
    });

    int statuѕCode = f.get();

You have full control on the Response life cycle, so you can decide at any moment to stop processing what the server is sending back:

      import com.ning.http.client.*;
      import java.util.concurrent.Future;

      AsyncHttpClient c = new AsyncHttpClient();
      Future<String> f = c.prepareGet("http://www.ning.com/").execute(new AsyncHandler() {
          private StringBuilder builder = new StringBuilder();

          @Override
          public STATE onStatusReceived(HttpResponseStatus status) throws Exception {
              int statusCode = status.getStatusCode();
               // The Status have been read
               // If you don't want to read the headers,body, or stop processing the response
               return STATE.CONTINUE; // Or STATE.ABORT
          }

          @Override
          public STATE onHeadersReceived(HttpResponseHeaders h) throws Exception {
              Headers headers = h.getHeaders();
               // The headers have been read
               // If you don't want to read the body, or stop processing the response
               return STATE.ABORT;
          }

          @Override
          public STATE onBodyPartReceived(HttpResponseBodyPart bodyPart) throws Exception {
               builder.append(new String(bodyPart.getBodyPartBytes()));
               return STATE.CONTINUE;
          }

          @Override
          public String onCompleted() throws Exception {
               // Will be invoked once the response has been fully read or a
               // ResponseComplete exception has been thrown.
               return builder.toString();
          }

          @Override
          public void onThrowable(Throwable t) {
          }
      });

      String bodyResponse = f.get();

Finally, you can also configure the AsyncHttpClient via it’s AsyncHttpClientConfig object:

        import com.ning.http.client.*;
        import java.util.concurrent.Future;

        AsyncHttpClientConfig cf = new AsyncHttpClientConfig.Builder()
                 .setProxyServer(new ProxyServer("127.0.0.1", 38080)).build();
        AsyncHttpClient c = new AsyncHttpClient(cf);

The library uses Java non blocking I/O for supporting asynchronous operations. The default asynchronous provider is build on top of Netty (http://www.jboss.org/netty), the Java NIO Client Server Socket Framework from JBoss, but the library exposes a configurable provider SPI which allows to easily plug in other frameworks.

Keep up to date on the library development by joining the Asynchronous HTTP Client discussion group or ask your questions directly by following me on twitter.

(*) This blog has been written in collaboration with Thomas Dudziak

technorati:

Categories: Ning
Follow

Get every new post delivered to your Inbox.

Join 50 other followers