mercredi 28 mai 2014

Indexing keys and values in MapDB

MapDB is a high performance pure java database, it provides concurrent collections (Maps, Sets and Queues) backed by disk storage or off-heap memory.
It provides a powerful mechanism to synchronize collections that can be used to build multiple indexes on a primary collection. Follows is an example showing how to index keys and also values of main collection.

1. define a serializable class
// this class should implement serializable in order to be stored
public class Person implements Serializable {
  String firstname; 
  String lastname; 
  Integer age; 
  boolean male;

  public Person(String f, String l, Integer a, boolean m) {     
    this.firstname = f;
    this.lastname = l; 
    this.age = a; 
    this.male = m;
  }

  public boolean isMale() {
    return male;
  }
  @Override
  public String toString() {
    return "Person [firstname=" + firstname + ", lastname=" + lastname + ", age=" + age + ", male=" + male + "]";
  }
}

2. Define a map of persons by id
// stores person under id
BTreeMap<Integer, Person> primary = DBMaker.newTempTreeMap();
primary.put(111, new Person("bIs9r", "NWmqoxFf", 92, true)); 
primary.put(111, new Person("4KXp8", "QrPsabf1", 31, false)); 
primary.put(111, new Person("eJLIo", "SJwJidWk", 6, true)); 
primary.put(111, new Person("LGW58", "vteM4khp", 42, false)); 
primary.put(111, new Person("tIM8R", "Rzq75ONh", 57, false)); 
primary.put(111, new Person("KqKRE", "BnpUV4dW", 26, true)); 

3. Define a gender-based index
// stores value hash from primary map
NavigableSet<Fun.Tuple2<Boolean, Integer>> genderIndex = new TreeSet<Fun.Tuple2<Boolean, Integer>>();

//1. gender-based index: bind secondary to primary so it contains secondary key
Bind.secondaryKey(primary, genderIndex, new Fun.Function2<Boolean, Integer, Person>() {
  @Override
  public Boolean run(Integer key, Person value) {
    return Boolean.valueOf(value.isMale());
  }
});
4. Use the gender-index to read all male persons
Iterable<Integer> ids = Fun.filter(genderIndex, true);
for(Integer id: ids) {
 System.out.println(primary.get(id));
}

MapdDB offers multiple ways to define indexes on a given collection, It can also be extended to define specific kind of indexes. Follows is an example of implementing the Bitmap index in MapDB:
public static <K, V, K2> void secondaryKey(MapWithModificationListener<K, V> map, final Map<K2, Set<K>> secondary,
      final Fun.Function2<K2, K, V> fun) {
  // fill if empty
  if (secondary.isEmpty()) {
    for (Map.Entry<K, V> e : map.entrySet()) {
      K2 k2 = fun.run(e.getKey(), e.getValue());
      Set<K> set = secondary.get(k2);
      if (set == null) {
        set = new TreeSet<K>();
        secondary.put(k2, set);
      }
      set.add(e.getKey());
    }
  }
  // hook listener
  map.modificationListenerAdd(new MapListener<K, V>() {
    @Override
    public void update(K key, V oldVal, V newVal) {
      if (newVal == null) {
        // removal
        secondary.get(fun.run(key, oldVal)).remove(key);
      } else if (oldVal == null) {
        // insert
        K2 key2 = fun.run(key, newVal);
        Set<K> set = secondary.get(key2);
        if (set == null) {
          set = new TreeSet<K>();
          secondary.put(key2, set);
        }
        set.add(key);
      } else {
        // update, must remove old key and insert new
        K2 oldKey = fun.run(key, oldVal);
        K2 newKey = fun.run(key, newVal);
        if (oldKey == newKey || oldKey.equals(newKey))
          return;
        Set<K> set1 = secondary.get(oldKey);
        if (set1 != null) {
          set1.remove(key);
        }
        Set<K> set2 = secondary.get(newKey);
        if (set2 == null) {
          set2 = new TreeSet<K>();
          secondary.put(newKey, set2);
        }
        set2.add(key);
      }
    }
  });
}
This new index can be used as follows:
final Map<Boolean, Set<Integer>> bitmapIndex = new HashMap<Boolean, Set<Integer>>();
secondaryKey(primary, bitmapIndex, fun);

Continue here

samedi 3 mai 2014

Exploiting Big RAMs

Those are notes from a talk given by Neil Ferguson about how to take benefit of very large amount of memory to improve the performance of server-side applications.

Background
With the increases in the amount of managed data of any enterprise or web application, there is a continuous need for storing more and more of data while providing a real-time access to it. The performance of such applications can be improved by making data available directly from memory and efficiently use the available huge amount of memory that may reach many many terabytes in a near future.

In fact, memory prices is continuously decreasing while the capacity increases to the point where terabytes of RAM will be available for servers in a near future. The cost of a 1MB of RAM was about $0.01 in Jan 2009 while it is $0.005 in 2013, source Memory Prices (1957-2013). In fact, we could by a workstation with 512GB of RAM  for $2299, and new Intell processors (e.g. Xeon) allow up to 144GB of RAM and more (around terabytes) for new generation processors dedicated to server-class machines. However, it still not practical to do anything with such an amount of RAM. Why?

Garbage Collection Limitations
In any Garbage-collected environment (like JVMs), if the object allocation rates overtake the rates at which the GC collect these objects then long GC pauses (time during which the JVM stops applications to just run the garbage collector) may become very frequent. One way to avoid such problem is to leave a plenty of free space in the heap. The thing is when you leave a third of 3GB it's not really a big deal compared to the case when leaving the third of 300GB even if it's the same ratio betwee free space and live data.
The bad news, is that even with large free space there may be some situations where GC pauses are too long typically for memory defragmentation.
You can improve an application performance with -XX:+ExplicitGCInvokesConcurrent as a workaround to avoid long pausses when System.gc() or Runtime.getRuntime().gc() are explicitly called (e.g. Direct ByteBuffer allocation).

Off-Heap storage
To overcome some of these limitations in JVMs or in Garbage-collected environment, allocation of memory off-heap can be a solution. This can be done in different ways:

1. Direct ByteBuffers
The NIO api allows the allocation of off-heap memory (i.e. not part of the process heap and not subject to GC) for storing long-lived data via ByteBuffer.allocateDirect(int capacity). Capacity is limited to what was specified with the JVM option -XX:MaxDirectMemorySize.
The allocation through ByteByffer has implications for GC (long pauses) when it is freed not fast enough and makes it not suitable for short-lived objects, i.e. allocating and freeing a lot of memory frequently.

2. sun.misc.Unsafe 
Direct ByteByffer itself relies on sun.misc.Unsafe.allocateMemory to allocate a big block of memory off-heap and on sun.misc.Unsafe.freeMemory to explicitly free it.
Here is a very sample implementation of a wrapper class based on the Unsafe API for managing off-heap memory:
public class OffHeapObject {
  // fields
  private static Unsafe UNSAFE;

  static {
    try {
      // get instance using reflection
      Field field = sun.misc.Unsafe.class.getDeclaredField("theUnsafe");
      field.setAccessible(true);
      UNSAFE = (sun.misc.Unsafe) field.get(null);
    }catch(Exception e){
      throw new IllegalStateException("Could not access theUnsafe instance field");
    }
  }
  private static final int INT_SIZE = 4;
  // base address for the allocated data
  private long address;
  
  // constructor
  public OffHeapObject(T heapObject) {
    // serialize data
    byte[] data = serialize(heapObject);
    // allocate off-heap memory
    address = UNSAFE.allocateMemory(INT_SIZE + data.length);
    // save the data size in first bytes
    UNSAFE.putInt(address, data.length);
    // Write data byte by byte to the allocated memory
    for(int i=0; i < data.length; i++) {
      UNSAFE.putByte(address + INT_SIZE + i, data[i]);
    }
  }

  public T get() {
    int length = UNSAFE.getInt(address);
    // read data from the memory
    byte[] data = new byte[length];
    for(int i = 0; i < data.length; i++) {
      data[i] = UNSAFE.getByte(address + INT_SIZE + i);
    }
    // return the deserialized data
    return deserialize(data);
  }
  // free allocate space to avoid memory leaks

  public void deallocate() {
    //TODO make sure to not call this more than once
    UNSAFE.freeMemory(address);
  }
}
The OffHeapObject can be used for instance to store values of a cached data, e.g. using Google Guava to store keys-OffHeapObject pairs where the latter wraps data in the off-heap memory. This way GC pauses can be considerably reduced as these objects are just references and do not occupy big block of heap memory. Also, the process size may not grow indefinitely as fragmentation is reduced.

Note that the implementation of the OffHeapObject is very basic, there is a performance impact for using off-heap memory. In fact, everything needs to be serialized on writes to off-heap and de-serialized on read from off-heap memory and these operations has some overhead and reduced throughput compared to on-heap storage.
Furthermore, not every object can be stored in the off-heap memory for instance the OffHeapObject that keep a reference to a block of memory in the off-heap is actually stored in the heap.
The performance of this implementation may be enhanced with techniques like data alignment.

Some existing caches based on off-heap storage

continue from 28:39
Big RAM: How Java Developers Can Fully Exploit Massive Amounts of RAM 

Resources
  • Understanding Java Garbage Collection presentation at JUG Victoria Nov/2013 - Azul Systems
  • Measuring GC pauses with jHiccup - Azul Systems
  • A good documentation of the Unsafe API can be found in this blog post.
  • How Garbage Collection works in Java - blog post.


Random resources related to Docker

General


Management


Continuous Integration

Environment configuration
DevOps

lmctfy

Networking

Ecosystem
  • Atomic project - Deploy and Manage your Docker Containers 
  • GearD - The Intersection of PaaS, Docker and Project Atomic 
  • Classification of the ecosystem of startups based on Docker 
  • Slides from DockerFr Meetup on Docker ecosystem
  • OpenCore a Big Data (Hadoop) as a Service provider


API Client
work in progress

dimanche 13 avril 2014

Managing Docker images and containers

In addition to managing Docker resources (including containers, images, hosts) through the official CLI, there is plenty of solutions available in the community to manage Docker resources in a comprehensive way from a single web-based interface.

DockerUI

Once our containers are running, DockerUI can be use to manage the overall system. It's a simple web app with basic features for:
 - Check the states of the images (running, stopped)
 - Remove images
 - Start, Stop, Kill and Remove containers

DockerUI can be used with the following commands

1. Building the web app from the github repository and tag the build image
$docker build -t crosbymichael/dockerui github.com/crosbymichael/dockerui

2. Launch the built container, make the web app available on the 9000 port and connect to the docker uinx socket to remotely control docker
$docker run -p 9000:9000 -v /var/run/docker.sock:/docker.sock crosbymichael/dockerui -e /docker.sock

Then on the browser, visit localhot:9000 to get something like:

Shipyard

Shipyard is a more advanced Docker management solution based on a client-server architecture where the agents (i.e. clients) collect information on Docker resources and report them to the Shipyard server. It providers in addition to the features available in DockerUI:
 - Authentication
 - Building new images by uploading local Dockerfile or providing URLs to a remote location
 - In the browser terminal emulation for attaching containers
 - Visualizing CPU and memory utilization of the running images
 - ...

1. To use Shipyard, issue to pull the image from the Docker public index:
$docker run -i -t -v /var/run/docker.sock:/docker.sock shipyard/deploy setup

Now, we can register as admin to Shipyard on http://localhost:8000/

2. Install the latest release (e.g. v0.2.5) of Shipyard agent on every hosts to collect the information on Docker resources:
$curl https://github.com/shipyard/shipyard-agent/releases/download/v0.2.5/shipyard-agent -L -o /usr/local/bin/shipyard-agent
$chmod +x /usr/local/bin/shipyard-agent

3. Run the agent and register to the main host where Shipyard is running
$/usr/local/bin/shipyard-agent -url http://localhost:8000 -register

4. On the Shipyard interface, authorize the agents already deployed to enable them.
5. Run the agent with the given key at registration:
$/usr/local/bin/shipyard-agent -url http://localhost:8000 -key agent_key



Troubleshooting, in case you get this message:
Error requesting images from Docker: Get http://127.0.0.1:4243/images/json?all=0
Then stop the Docker service and re-start it while enabling Remote API access for any IP address:
$sudo service docker stop
$docker -H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock -d &

happy dockering

dimanche 6 avril 2014

Automating Docker image builds with Dockerfiles

Hello Dockerfile
This is a continuation of an previous post on Docker with the aim of using specific scripts called dockerfiles in order to automate the steps that we have been issuing to build docker images. When docker parse the script file, it sequentially executes the commands starting from a base image to create a new one after each command.
The syntax of a dockerfile instruction is as simple as :
command argument1 argument2 ... 
or
command ["argument1", "argument2", ...]  only for the entry-point command !!

It's preferable to write the command in uppercase!

Dockerfile instructions
There is a dozen of instructions that can be present in a dockerfile, a detailed list can be found in the official documentation. The most common ones are:
  • FROM all dockerfile should start with this command that specify the name of the image to use as a working or base image;
  • RUN allows to run a command in the current container and commit (automatically) the changes to a new image;
  • MAINTAINER allows to specify information (name, email) on the person responsible for maintain this script;
  • ENTRYPOINT allows to specify what command should be executed at first once the container is started;
  • USER allows to specify with which user account the command inside the container have to be executed with; 
  • EXPOSE allows to specify what port to expose for the running container.
  • ENV to use for setting environment variables
  • ADD to copy files from the build context (it does not work if using stdin to read dockerfile) into a physical directory in the image (e.g. copying a war file into tomcat webapps folder)
Here you can find the official tutorial to experiment with these command.

Parsing dockerfiles
Once finished editing the build script, issue docker build to parse the dockerfile and create a new image. There is different ways to use this command:
  • dockerfile is in current directory docker build .
  • from stdin docker build - < Dockerfile
  • from a github repository docker build github.com/username/repo docker will then clone the repo and parse the files in the repo directory.

Example
Now lets take the instructions from the previous post and gather them into a dockerfile:
# Use ubuntu as a base image
FROM ubuntu

# update package respository
RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list

RUN echo "deb http://archive.ubuntu.com/ubuntu precise-security main universe" > /etc/apt/sources.list
RUN apt-get update

# install java, tomcat7
RUN apt-get install -y default-jdk
RUN apt-get install -y tomcat7

RUN mkdir /usr/share/tomcat7/logs/
RUN mkdir /usr/share/tomcat7/temp/

# set tomcat environment variables
ENV JAVA_HOME=/usr/lib/jvm/default-java
ENV JRE_HOME=/usr/lib/jvm/default-java/jre
ENV CATALINA_HOME=/usr/share/tomcat7/

# copy war files to the webapps/ folder
ADD path/to/war /usr/share/tomcat7/webapps/

# launch tomcat once the container started
#ENTRYPOINT service tomcat7 start
ENTRYPOINT /usr/share/tomcat7/bin/catalina.sh run

# expose the tomcat port number
EXPOSE 8080

Save this script to Dockerfile, build it and tag the image by tomcat7, then launch the container while exposing publicaly the tomcat server port 8080, and finally check if the container is running
$docker build -t tomcat7 - < Dockerfile
$docker run -p 8080 tomcat7
$docker ps

to be continued;

lundi 31 mars 2014

Build your own SaaS with Docker - Part I

Hello Docker
Docker enables sand-boxing of applications and their dependencies in virtual containers to be able to run them in isolated mode. It provides an easy to use API for automating deployment operations that looks very close to Git commands. More introductory information can be found in its Wikipedia page.

Installation
Docker installation on a Ubuntu 64bit (for other OS check official documentation)
$sudo sh -c "curl https://get.docker.io/gpg | apt-key add -" 
$sudo sh -c "echo deb http://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list" 
$sudo apt-get update
$sudo apt-get install lxc-docker

Once docker installed, run a shell from within a container as follow
$sudo docker run -i -t ubuntu /bin/bash

As it is supposed to not find the ubuntu image, docker will pull it from the registry. Once, installed you can prompt:
  • #exit to leave the container
  • $sudo docker images to see all local images.
  • $sudo docker inspect image_name to see detailed information on an image.
  • $sudo docker ps to see the status of the container
  • $sudo docker stop CONTAINER_ID to stop a running image (or container)
  • $sudo docker logs CONTAINER_ID to see all logs if a given container
  • $sudo docker commit CONTAINER_ID image_name to commit changes made to a container

Installing Tomcat within a container
Start a new container using the ubuntu base image:
$sudo docker run -i -t ubuntu /bin/bash

Update the image's system packages
#apt-get update

1. Install the Apache Tomcat application server:
#apt-get install -y tomcat7

Once installed the following directories are created (more details can be found here):
  • /etc/tomcat7 for configuration
  • /usr/share/tomcat7 for runtime, called $CATALINA_HOME
  • /usr/share/tomcat7-root for webapps
2. Install Java DK
#apt-get install -y default-jdk

3. Configure environment variables
#pico ~/.bashrc
export JAVA_HOME=/usr/lib/jvm/default-java
export CATALINA_HOME=~/path/to/tomcat
#. ~/.bashrc to make the changes effective

Now when typing #echo $CATALINA_HOME you should see the exact path set to tomcat7.

4. Start the Tomcat7 server
#$CATALINA_HOME/bin/startup.sh
or
#service tomcat7 start

The start-up may fail with something like "cannot create directory '/usr/share/tomcat7/logs/catalina.out/'". To solve this, you may just have create the logs directory:
#mkdir /usr/share/tomcat7/logs

to check if Tomcat is running issue
#ps -ef | grep tomcat
or
#service tomcat7 status

then check in your browser http://container_ip_address:8080/
to get the IP address of the container issue
#ifconfig

5. Shutdown  Tomcat7
#$CATALINA_HOME/bin/shutdown.sh
or
#service tomcat7 stop

Save the image to index.docker.io
The changes we made on the base image created a new one, we should commit these changes to not lose these changes.

1. Login to index.docker.io
$sudo docker login
Username: your_user_name
Password: your_password
Email: your_email
Login Succeeded

If you don't have an account, sign up here.

2. Commit changes to your repository
$sudo docker commit CONTAINER_ID USERNAME/REPO_NAME

3. Push changes to this repository
$sudo docker push USERNAME/REPO_NAME

4. Start a new container using the image commit to your repository as base image
$sudo docker run -i -t USERNAME/REPO_NAME /bin/bash
#

To run Tomcat in the container
$sudo docker run -i -t USERNAME/REPO_NAME $CATALINA_HOME/bin/startup.sh
or
$sudo docker run -i -t USERNAME/REPO_NAME service tomcat7 start

to cleanup old containers
$sudo docker ps -a -q | xargs sudo docker rm
or
$sudo docker ps -a | awk '{print $1}' | xargs sudo docker rm

to cleanup old and non tagged images
$sudo docker images | grep "^" | awk '{print $3}' | xargs sudo docker rmi -f

Resources
If you are confused with docker terminology (e.g. container, image, etc.) check this official documentation.
General purpose instructions for installing Tomcat7 on a ubuntu machine here.

dimanche 6 octobre 2013

Joyn or RCS

RCS (Rich Communication Services) is a GSMA standard that aims to bring a set of rich communication (that goes beyond SMS and phone calls) yet inter-operable services  across different domains managed by different telecom operators. This telcos standard is marketed under the name of Joyn.
Many operators has already deployed on their networks offering users VoIP and presence services that can be accessed by installing an application from the market store (Google Play, AppStore). In addition, some smarphone manufacturers who have joined the movement already embed the RCS stack into their devices.

The next step of commercializing Joyn is to build an ecosystem by providing APIs and empowering the developers community to create communication-based applications that relies on the platform. Orange through Orange Partner and Deutsch Telekom through the Developer Garden programs are leading these efforts in Europe. For instance, they jointly sponsored the Joyn Hackathon (press release) were the Joyn API  was introduced.

The remaining of this post explains how to use the Android Joyn SDK to build conversational applications. The overall interaction between an application and the Joyn SDK (and behind the RCS platform) is explained in the following figure.
Joyn API call flow

  1. Instantiate Joyn service and establish a connection
   private ChatService mService;
   private JoynServiceListener mListener = new JoynServiceListener() {
      @Override public void onServiceDisconnected(int error) {
         Log.i(TAG, "ChatService disconnected!");
      }
      @Override public void onServiceConnected() {
         Log.i(TAG, "ChatService connected!");
      }
   };
   ...
   // Instanciate API
   mService = new ChatService(getApplicationContext(), mListener); 
   // Connect API
   mService.connect();
  1. When the the connection is successfully established then start calling API methods
   private Chat mChat;
   private ChatListener mChatListener = new ChatListener() {
      @Override public void onReportMessageFailed(String arg0) {}
      @Override public void onReportMessageDisplayed(String arg0) {}
      @Override public void onReportMessageDelivered(String arg0) {}
      @Override public void onNewMessage(ChatMessage arg0) {}
      @Override public void onComposingEvent(boolean arg0) {}
   };
   ...
   @Override public void onServiceConnected() {
      Log.i(TAG, "ChatService connected!");
      if (mService != null && mService.isServiceRegistered()) {
         // Get remote contact
         String contact = getIntent().getStringExtra("contact");
         // Call API Methods
         mChat = mService.openSingleChat(contact, mChatListener); 
         mChat.sendMessage("hello world!");
      }
   }
The API doc of the ChatService can be found on this link.