Installing Cloud Foundry v2 locally on Vagrant

Cloud Foundry (CF)

CloundFoundry (CF) is one of the many PaaS available out there that aims to empower developers to build their applications (e.g. web) without caring about infrastructure details. The PaaS handles the deployment, scaling and management of the apps in the cloud data center, thus boosting the developer productivity.
CF has many advantages over other PaaS solutions as it is open source, it has a fast growing community and many big cloud actors are involved in the development and spreading it adoption. It also can be run anywhere even on a laptop and this what this post is about. So keep reading..

Terminology

- Bosh is an open-source platform that helps deploying/managing systems on cloud infrastructures (AWS, OpenStack/CloudStack, vSphere, vCloud, ect).

- Bosh Lite is a lightweight version of Bosh that can be used to deploy systems locally by using Vagrant instead of cloud infrastructure (e.g. AWS) and Linux Containers (Warden project) for to run your system instead of VMs.
- Stemcell is a template VM that will be used by Bosh to create VMs and deploy them to the cloud. I contains essentially an OS (e.g. CentOS) and a Bosh Agent in order to be controlled.

1. Install Git

$ sudo apt-get install git

2. Install VirtualBox
$ sudo echo "deb http://download.virtualbox.org/virtualbox/debian precise contrib" >> /etc/apt/sources.list
or create a new .list file as described in this thread.
$ wget -q http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc -O- | sudo apt-key add -
$ sudo apt-get update
$ sudo apt-get install virtualbox-4.3
$ sudo apt-get install dkms
$ VBoxManage --version
4.3.10_Ubuntur93012

3. Install Vagrant (the known version to work with bosh-lite is 1.6.3 - link)
$ wget https://dl.bintray.com/mitchellh/vagrant/vagrant_1.6.3_x86_64.deb
$ sudo dpkg -i vagrant_1.6.3_x86_64.deb
$ vagrant --version
Vagrant 1.6.3

Check if vagrant is correctly working with the installed virtual box
$ vagrant init hashicorp/precise32
$ vagrant up

4. Install Ruby(using RVM) + RubyGems + Bundler
4.1. Install rvm
$ curl -sSL https://rvm.io/mpapis.asc | gpg --import -
$ curl -sSL https://get.rvm.io | bash -s stable
$ source /home/{username}/.rvm/scripts/rvm
$ rvm --version

4.2. Install latest ruby version
$ rvm install 1.9.3-p551
$ ruby -v
ruby 1.9.3p551 (2014-11-13 revision 48407) [x86_64-linux]

5. Install Bosh CLI (check the prerequisites for the target OS here)
- Note that Bosh CLI is not suppored on windows - github issue
$ sudo apt-get install build-essential libxml2-dev libsqlite3-dev libxslt1-dev libpq-dev libmysqlclient-dev
$ gem install bosh_cli

6. Install Bosh-Lite
$ git clone https://github.com/cloudfoundry/bosh-lite
$ cd bosh-lite
$ vagrant up --provider=virtualbox

In case the following message is seen The guest machine entered an invalid state while waiting for it to boot, then:

• check if virtualisation (Intel VT-x / AMD-V for 32bits or Intel EPT / AMD RVI for 64bits) is enabled on target system here. If not then enable it from the BIOS, for ESXi check link1 and link2 and add vhv.enable = "TRUE" to the vm configuration file (i.e. vmx) and make sure the VM is of version 9. 
• You may also have to check if USB 2.0 controller is enabled, if it is then disable it.

Target the BOSH Director
$ cd ..
$ bosh target 192.168.50.4 lite
$ bosh login
Your username: admin
Enter password: *****

Logged in as `admin'

Setup a route between the laptop and the VMs running inside Bosh Lite
$ cd bosh-lite
$ ./bin/add-route

7. Deploy Cloud Foundry
Install spiff

$ brew tap xoebus/homebrew-cloudfoundry

$ brew install spiff

$ spiff

To install spiff on linux systems check this issue.

Upload latest stemcell
$ wget http://bosh-jenkins-artifacts.s3.amazonaws.com/bosh-stemcell/warden/latest-bosh-stemcell-warden.tgz
$ bosh upload stemcell latest-bosh-stemcell-warden.tgz
Check the stemcells
$ bosh stemcells

Upload latest CF release
$ git clone https://github.com/cloudfoundry/cf-release
$ export CF_RELEASE_DIR=$PWD/cf-release/
$ bosh upload release cf-release/releases/cf-XXX.yml

Deploy CF releases
$ cd bosh-lite/
$ ./bin/provision_cf
$ bosh target # check the target director
$ bosh vms    # check the installed VMs on the cloud

Manually (to be continued)
Generate a configuration file manifests/cf-manifest.yml
$ mkdir -p go
$ export GOPATH=~/go
$ cd bosh-lite
$ ./bin/make_manifest_spiff

Deploy release
$ bosh deploy

Install CF CLI

Play with CF
$ cf api --skip-ssl-validation https://api.10.244.0.34.xip.io
$ cf login
$ cf create-org ORG_NAME
$ cf orgs

$ cf target -o ORG_NAME

$ cf create-space SPACE_NAME

$ cf target -o ORG_NAME -s SPACE_NAME

To access the VM from the LAN (i.e. another machine):

1. Install an HTTP Proxy (e.g. squid3),
2. Configure CF HTTP_PROXY environment variable, and 
3. Configure the proxy:

       $ sudo nano /etc/squid3/squid.conf 
       acl local_network src 192.168.2.0/24
       http_access allow local_network

Stopping CF
Shooting down bosh-lite VM can be surprisingly tricky. May better stop the VM with:

• vagrant suspend to save current state for next start up, or
• vagrant halt, then next time to start CF use vagrant up followed by bosh cck (documentation).

Troubleshooting
$ bosh ssh # then choose the job to access (password: admin)
bosh_something@something:~$ sudo /var/vcap/bosh/bin/monit summary
Find the Bosh Lite IP address
$ cd bosh-lite/
$ vagrant ssh
vagrant@agent-id-bosh-0:~$ ifconfig
vagrant@agent-id-bosh-0:~$ exit

Complete installation script can be found here.

Resources

• Installing latest versions for virtualbox and vagrant - link
• Installing ruby with rvm - link.
• DIY PaaS (CF v1) running DEA link1, stagging applications link2.
• Deploying CF Playground (a kind of web admin interface) - link
• Installing CF on vagrant - link video
• Installing BOSH lite - github repo, tutorial
• Deploying CF using BOSH lite - github repo, demo
• http://altoros.github.io/2013/using-bosh-lite/
• Installing a new hard drive - link
• xip.io a free internet service providing DNS wildcard - link
• Troubleshooting with Bosh CLI - official doc, app health, monit summary
• Remotely debug a CF application - link
• CloudFoundry manifest.yml generator - link

Test Driven Development

Testing Overview

In Waterfall development process, developers finish code then do some ad-hoc testing. In the contrary with Agile development process, testing is part of every Agile iteration and developers are the responsible for testing own code. Also, testing tools & processes are highly automated.

With Behavior-Driven Design (BDD), one has to develop user stories to describe features, then thanks to Cucumber, user stories become acceptance tests and integration tests.
With Test-driven development (TDD), specify step definitions for new story (this may require new code to be written), write unit & functional tests for that code first, before the code itself. In a nutshell, write tests for the code you wish you had.

How both works together? Use Cucumber to describe behavior via features & scenarios (behavior driven design), and RSpec to test individual modules that contribute to those behaviors (test driven development).
When Cucomber test fails, it calls for RSpec test, if it fails you need to add missing methods. Keep iterating until passing feature implementation test, when features are implemented properly it will pass the Cucomber test then keep going to development.

TDD Getting Started

Unit Test should be FIRST: Fast to run, Independent the order of running tests should no matter, Repeatable if a test fails once then it should fail again if we re-run it again (i.e. always same result), Self-checking which means no human intervention to interpret test result as failed or successful. Timely   can be run on background while developing and bring developers attention when it find bugs, also when code change test should also.
RSpec is a Domain-Specific Language for testing. DSL stands for small programming language that simpifies one task at expense of generality (e.g. migrations, regexes, SQL). RSpec tests are called specs, and inhabit spec directory.
Typing following command rails generate rspec:install will creates this directory structure:

• app/models/*.rb                                    spec/models/*_spec.rb
• app/controllers/*_controller.rb              spec/controllers/*_controller_spec.rb
• app/views/*/*.html.haml                       (use Cucumber!)

Te be continued from slide 13 ...

Configuring Ruby on Rails (RoR)

Here are the steps you need to configure your environment to start building Rails web applications:

• Installing Git
• Installing Ruby
• Installing RubyGems
• Installing Rails

Here are the steps you need to configure your environment to start building Rails web applications.

Installing Git

Rails ecosystem depends tremendously on Git a very powerful version control system. Git must be installed at first as you will see it every time you google any Rails package.
on a Debian-based distribution like Ubuntu, you have to type following instruction to install Git:

$ apt-get install git-core

You may find detailed installation instructions for your platform at the Installing Git section of the book Pro Git.

Installing Ruby

Next thing to do is to install Ruby, for this you need first to install Ruby Version Manager (RVM). RVM primary goal is helping you to install and manage multiple versions of Ruby on the same machine. Following are the steps for installing RVM on Linux, more details can be found in RVM Installation.

You need curl, if you don't have you can installed on Ubuntu by typing:
$ apt-get install curl
Then download RVM
$ curl -L get.rvm.io | bash -s stable
Load RVM
$ source ~/.rvm/scripts/rvm
Check requirements for installing RVM and follow the instructions:
$ rvm requirements
Finally, now you can install Ruby (1.9.3 is latest version)
$ rvm install 1.9.3

For Windows (using Cygwin) check this tutorial.

rbenv is another ruby sandboxing tool lighter and faster than rvm, you can use both in similar way.

Installing RubyGems

RubyGems is a useful package manager for Ruby projects that may save you a lot of time as there are many useful libraries (including Rails) available as Ruby packages, or gems. RubyGems should be automatically installed if you had successfully installed RVM easy once you install Ruby. You may use following command to check its availability:
$ which gem
/Users/mhartl/.rvm/rubies/ruby-1.9.3-p0/bin/gem

Installing Rails

Finally, we are ready for installing Rails framework and star building great web applications. Rails can installed as follows:

$ gem install rails -v 3.2.3

To check Rails installation, run following command to see version number:
$ rails -v
Rails 3.2.3

If you’re running Linux, you might have to install a couple of other packages at this point:

$ sudo apt-get install libxslt-dev libxml2-dev libsqlite3-dev # Linux only


You want to configure Ruby on Rails on a virtual machine based on Bodhi Linux 1.4.0 Stable which is a light distro based on Ubuntu 10.04 LTS. I may be useful to use configure_image.sh which is  configuration file that will install most of the things described earlier and other useful gems.
These are the instruction for configuring your vm and testing a sample web applications:

1. Create a new VM with 256 MB RAM and 4GB HD (VirtualBox > New)
2. Mount bodhi_1.4.0.iso as a live CD (VirtualBox > Settings > Storage)
3. Install Bodhi Linux (VirtualBox > *your_VM* > Start)
4. sudo apt-get update, sudo apt-get upgrade
5. sudo apt-get install git-core (there's no git on Bodhi preinstalled)
6. mkdir -p Documents (there's no Documents directory/folder by default)
7. wget -O configure_image.sh http://pastebin.com/download.php?i=WTXF7g7F
8. chmod +x configure_image.sh
9. ./configure_image.sh (setups the necessary development environment ~ 2h!)
10. cd Documents
11. mkdir -p hw2_rottenpotatoes_2012 (for my fork of the official repo)*
12. git clone git://github.com/saasbook/hw2_rottenpotatoes.git
13. cp -r hw2_rottenpotatoes hw2_rottenpotatoes_2012
14. cd hw2_rottenpotatoes_2012
15. bundle install --without production
16. bundle exec rake db:migrate
17. rails server
18. http://0.0.0.0:3000/movies 

Resources

You may have to look for other useful resources:

• Ruby on Rails site
• Ruby on Rails tutorial is a good book by Michael Hartl if you start with Ruby and/or Rails
• Engineering Long-Lasting Software: An Agile Approach Using SaaS and Cloud Computing by Armando Fox and David Patterson. Here is the link for configuring virtual machine if you want to go through the book example.
• Other useful resources for Ruby in general can be found at RubySource and for Rails at RailsCasts

Introduction to Ruby and Rails

Ruby is an Interpreted and Object-oriented language where Everything is an object and Every operation is a method call on some object.
Ruby is Dynamically typed, i.e. objects have types, but variables don’t. Dynamic means add, modify code at runtime (metaprogramming), ask objects about themselves (reflection), and in a sense all programming is metaprogramming.

Ruby is ...

an Interpreted and Object-oriented language where Everything is an object and Every operation is a method call on some object.
Ruby is Dynamically typed, i.e. objects have types, but variables don’t. Dynamic means add, modify code at runtime (metaprogramming), ask objects about themselves (reflection), and in a sense all programming is metaprogramming.

Naming conventions

Class names use UpperCamelCase, for instance class FriendFinder ...  end
Methods and variables use snake_case, for def learn_conventions  ...  end, def faculty_member?  ...  end, def charge_credit_card!  ...  end.
Constants (scoped) & $GLOBALS (not scoped) are must be in upper case: TEST_MODE = true, $TEST_MODE = true.
Symbols are immutable string whose value is itself, for example:

• favorite_framework = :rails
• :rails.to_s == "rails"
• "rails".to_sym == :rails
• :rails == "rails"  # => false

Variables, Arrays, Hashes

In Ruby, there are no variable declarations, although local variables must be assigned before use and instance and class variables are equal to nil until been assigned. 

It is OK to write:  x = 3; x = 'foo' but it's wrong to write  Integer x=3.

In Ruby, Array and Hash elements can be anyting, including other arrays/hashes, and don't all have to be same type.  As in python and perl, hashes are the swiss army chainsaw of building data structures.

Examples of Array:  x = [1,'two',:three]; x[1] == 'two' ; x.length==3

Examples of Hash:  w = {'a'=>1, :b=>[2, 3]}, w[:b][0] == 2 , w.keys == ['a', :b]

Methods

Everything (except fixnums) is pass-by-reference. Example of methods: 

def foo(x,y)    

  return [x,y+1]

end

def foo(x,y=0)  # y is optional, 0 if omitted

  [x,y+1]       # last exp returned as result

end

def foo(x,y=0) ; [x,y+1] ; end

Methods can be called with:  a,b = foo(x,y) or a,b = foo(x) when optional arg used.

Basic Constructs

In Ruby, statements end with ';' or newline, but can span line if parsing is unambiguous

It is OK to write (as unless cannot end a statement):

raise("Boom!") unless

        (ship_stable)   

But it is wrong to write:

raise("Boom!")

        unless (ship_stable)

Basic Comparisons & Booleans (are just like in all modern programming languges): == != < >  =~  !~  true   false nil  

The usual control flow constructs are:

Strings & Regular Expressions

You should try rubular.com for any regex needs. An example of regex: "string", %Q{string}, 'string', %q{string}

To match a=41 ; the corresponding regex is #{a+1}

Match a string against a regexp (just like with pyhon):

• "fox@berkeley.EDU" =~ /(.*)@(.*)\.edu$/i
• /(.*)@(.*)\.edu$/i =~ "fox@berkeley.EDU"

If no match, returned value is false, in case of  match, value is non-false, and $1...$n capture parenthesized groups ($1 == 'fox', $2 == 'berkeley')

/(.*)$/i  or  %r{(.*)$}i   or  Regexp.new('(.*)$', Regexp::IGNORECASE)

Everything is an object; (almost) everything is a method call

Even lowly integers and nil are true objects, and we can write 57.methods, 57.heinz_varieties, and nil.respond_to?(:to_s) 

Rewrite each of these as calls to send for example:  my_str.length  =>  my_str.send(:length)

• 1 + 2 is equivalent to 1.send(:+, 2) 
• my_array[4]  is equivalent to my_array.send(:[], 4)
• my_array[3] = "foo"   is equivalent to my_array.send(:[]=, 3,"foo")
• if (x == 3)  ....   is equivalent to if (x.send(:==, 3)) ...
• my_func(z) is equivalent to self.send(:my_func, z)

In particular, things like “implicit conversion” on comparison is not in the type system, but in the instance methods.

a.b means: call method b on object a, i.e. a is the receiver to which you send the method call, assuming a will respond to that method. It does not mean:  b is an instance variable of a, and does not mean also:  a is some kind of data structure that has b as a member

Understanding this distinction will save you from much grief and confusion.

SaaS - part II

SaaS Architecture

The Web as a Client-Server System

The Web is a client/server architecture and fundamentally request/response oriented.

Client-Server architecture is a high level architecture where clients and servers are specialized in specific tasks: clients ask questions on behalf of users, servers wait for and respond to questions, serve many clients. 

Client-Server is an architectural pattern, it has another alternative P2P (Peer to Peer) architectures.

A Design patterns capture common structural solutions to recurring problems.

Domain Name System (DNS) is another kind of server that maps names to IP addresses.

Web at 100,000 feet

HTTP (Hypertext Transfer Protocol) is an ASCII-based request/reply protocol used for transferring information on the web 

• HTTP requests include request method (GET, POST, etc.), Uniform Resource Identifier (URI), HTTP protocol version that is understood by the client, headers for transferring extra informations about the request.
• HTTP responses from web server include protocol version, status code (2xx all is well, 3xx resource moved, 4xx access problem, 5xx server error), headers and response body.

Early Web 1.0 problem was how to guide a user through a flow of pages as HTTP is stateless. Many options were chosen: 

• User IP address to identify returning user (problems: public computers, multiple users sharing same IP)
• Embed per-user junk into URI query string (problems: breaks caching)
• Cookies: per-user user state can be used for lots of things like customization (My Yahoo), click/flow tracking, authentication (logged in or not), 

A golden rule: don't trust the client, cookies must be tamper-evident.

Which of the previous things could be implemented on the client side? which ones shouldn't be and why?

3-tier shared-nothing architecture & scaling

Dynamic content generation 

In early days, most web pages were (collection of static pages) plain old files. Later, when e-commerce sites appeared, a program was running to generate pages. Originally, templates with embedded code "snippets". Eventually, code become "tail that wagged the dog" and moved out of the Web server. 

Software as a Service

Sites that are really programs have to deal with many things (frameworks support these common tasks):

• map URI to correct program & function?
• pass arguments between web site pages 
• invoke program on server
• handle persistent storage
• handle cookies
• handle errors
• package output back to user

Sharding vs. Replication

For scaling a Web application is crucial to be able to scale persistence layer. Two techniques are commonly used:

• Sharding: consists of partitioning data across independent "shards" (e.g. user profile table), it scales great, but bad when operations touch more than one table (e.g. when running join queries),
• Replication: consists of replicating all data everywhere, this makes running multi-table queries faster, but hard to scale as writes must propagate to all copies which create a temporary inconsistency in data values.  

Summary

Browser requests web resource (URI) using HTTP which is a simple request-reply protocol that relies on TCP/IP. In SaaS, most URI’s cause a program to be run, rather than a static file to be fetched.

HTML is used to encode content, CSS to style it visually

Cookies allow server to track client (e.g. including a handle to server-side information): browser automatically passes cookie to server on each request, and server may change cookie on each response 

Frameworks make all these abstractions convenient for programmers to use, without sweating the details and help map SaaS to 3-tier, shared-nothing architecture

Model-View-Controller 

The MVC design pattern

Separate organization of data (model) from presentation (view) by introducing controllers that mediate user actions requesting access to data, and present data for rendering by a given view.

Web apps may seem obviously MVC by design, but as an architecture other alternatives are possible. 

• Page Controller where an HTML page is associated in a one to one fashion with a program (e.g. Ruby Sinatra). 
• Front Controller where a single program handle all user requests and render corresponding view (e.g. J2EE servlet)
• Template View where code snippets are inserted directly into the view for customizing rendering based on stored data and user request (e.g. PHP).

Models, Databases, and Active Record

In-Memory vs. In-Storage objects

In-memory object are marshaled/serialized into in-storage objects. The later are unmarshaled/deserialized in the first ones.

How to represent persisted object in storage?

Basic operations on objects are Create, Read, Update, Delete (CRUD).

ActiveRecord gives to every model common mechanisms so that it knows how to CRUD itself.

Rails Models store data into Relational Databases (RDBMS) where each Model gets its own database table. A schema is a collection of all tables and their structure.

• A row is one Model instance, it has a unique value for its primary key, and all rows have similar structure. 
• Each colon stores value of an attribute of the model.

ActiveRecord vs. DataMapper

DataMapper is an alternative to ActiveRecords that associates separate mapper with each Model to abstract underlying storage system and to be able to work with any RBDMS. DataMapper is used by Google App Engine, it scales very well but can't exploit RBDMS features to simplify complex queries (e.g. join) and relationships.

Controllers, Routes, and RESTfulness

Routes

In MVC, each interaction that user can do is handled by a controller action, i.e. method that handle this interaction. A route maps <HTTP method, URI> to controller action.

For instance, Route "GET /movies/3" is mapped to Action "Show info about movie whose ID=3".

Rails Routing subsystem: 

• dispatch <method, URI> to correct controller action, 
• provides helper methods that generate a <method, URI> pair given a controller action, 
• parses query parameters from both URI and form submission into a convenient hash,
• built-in shortcuts to generate all CRUD routes (though most apps will also have other routes)

Example of how Rails manage a given user request GET /movies/3/edit  HTTP 1.0

• Matching route: GET /movies/:id/edit {:action=>"edit", :controller=>"movies"}
• Parse wildcard parameters: params[:id] = "3"
• Dispatch to edit method in movies_controller.rb
• To include a URI in generated view that will submit the form to the update controller action with params[:id]==3, call helper:  update_movie_path(3) # => PUT /movies/3

Representational State Transfer (REST)

The idea behind REST is to create self-contained requests that specify what resource to operate on and what to do to it instead of using cookies, setting them on each user request and do multiple HTTP requests between user browser and Web application.

A service (in the SOA sense) whose operations are like this is a RESTful service: its RESTful URIs name the operations.

Template Views and Haml

Template View pattern

A Template View consists of markup with selected interpolation to happen at runtime, it generates HTML that will be consumed by a human. 

In early days, this was the application, e.g. PHP you start writing views than open <?php to write usually, values of variables      or result of evaluating short bits of code. 

An alternative to this pattern is Transform View to generate JSON/XML instead of HTML, in case you application is called as a service by another application.

Template View vs. Transform View

Don't put code into views as MVC advocates thin views & controllers, also it's awkward to put code into Haml pages. An alternative to Haml is html.erb for embedded Ruby templates (just like PHP).

Helpers are methods that prettify objects for including in views, they have their own place in Rails app. 

Summary & Reflections: SaaS Architecture

2008 "Rails doesn't scale"

• Scalability is an architectural concern that is not confined to language or framework
• The stateless tiers of 3-tier arch do scale, with cloud computing, just worry about constants
• Traditional relational databases do not scale, instead various solutions combining relational and non-relational storage (“NoSQL”) scale much better (DataMapper works well with some of them)
• Intelligent use of caching can greatly improve the constant factors

Architecture is about Alternatives

Summary: Architecture & Rails

Model-view-controller is a well known architectural pattern for structuring apps. Rails codifies SaaS app structure as MVC:

• Views are Haml or embedded Ruby code, transformed to HTML when sent to browser
• Models are stored in tables of a relational database, accessed using ActiveRecord
• Controllers tie views and models together via routes and code in controller methods