Entries in buildpacks (1)

Wednesday
Aug122015

Buildpacks in Lattice

Lattice v0.3.0 just shipped (if you don't know much about Lattice, go to Lattice.cf), and for me the big news was the newly added support for Cloud Foundry Buildpacks within a Lattice cluster.

The goal of Lattice is to get a portion of the larger Cloud Foundry platform repackaged into an easily consumable system that makes getting started with the Cloud Foundry technologies straight forward for individuals and small teams. If you want to understand this a little better, watch James Bayer and Colin Humphreys talk about Lattice at the 2015 CF Summit North America.

In previous releases, Lattice was limited to only running completed Docker images within the cluster. Now that we have Buildpack support, Lattice is really able to give you a much better taste of what it means to use Cloud Foundry. Have some code? Just push it to Lattice and let it do the work to build the container for you.

Here's how to use it:

Setup

For this walkthrough, I'll be using the Vagrant box version of Lattice (using VirtualBox). These instructions assume v0.3.0 of both Lattice and the ltc command line tool. The same process should work for future versions of Lattice, we well as different deployment options (like one of the Terraform configurations for different IaaS environments).

You can grab the ltc cli tool from the bottom of the release notes.

To get the latest 

>  git clone https://github.com/cloudfoundry-incubator/lattice.git
> cd lattice
> git checkout v0.3.0
> vagrant up --provider virtualbox

This should result in a running box with the Lattice system initialized. If it worked correctly, you'll see:

==> default: Lattice is now installed and running.
==> default: You may target it using: ltc target 192.168.11.11.xip.io

By default, the Lattice box environment will be unauthenticated so it's easy to get ltc to target the cluster:

> ltc target 192.168.11.11.xip.io
Blob store is targeted.
Api Location Set

I've run into issues where xip.io doesn't resolve via some DNS servers, so if you have that issue consider using another DNS server like Google's 8.8.8.8 address.

At this point, you should have a functional Lattice system and ltc should be able to talk to it. Give it a test by running 'ltc list'.

> ltc list
------------------------------= Apps =-------------------------------
No apps to display.

------------------------------= Tasks =------------------------------
No tasks to display.

Get a Sample App

Now we're ready to find some source code that we want to run within Lattice. Since Lattice is specifically designed to be a BYOS (bring your own services), my example will not require any database or other backend services. It's a simple Python Flask based web app. Obviously it's a trivial Hello World example, but you can easily extrapolate and see how this would be useful for non-trivial systems.

The sample application we'll be using can be found on GitHub at https://github.com/chipchilders/sample-python. Go ahead and clone the repo now (from a working directory that's not within the lattice project's directory):

> cd ~
> git clone https://github.com/chipchilders/sample-python.git
> cd sample-python

There are only three files in the repo:

 

  • Procfile - the file that helps Flask startup
  • requirements.txt - the list of modules needed to run the app, in this case it's only Flask
  • hello.py - the basic web app that we'll be hosting within Lattice

 

The contents of hello.py are pretty straight forward. The file creates a route for '/' that returns the string "Hello World! Im an instance number X", where X is the Lattice instance number (as retrieved from the app's environment variables).

from flask import Flask
import os

app = Flask(__name__)

cfindex = os.getenv("INSTANCE_INDEX")

@app.route('/')
def hello_world():
    return 'Hello World! I am instance number ' + str(cfindex)

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)

My example assumes that 8080 is a reasonable listening port, since that's the default port that Lattice expects from containers. There's lots of flexibility here, but defaults will do for the example.

Build the Droplet

Lattice uses the same terminology as Cloud Foundry for the container images created via buildpacks: Droplets. Droplets are the result of taking code from the user and running it through the staging process.

Step one is to do a build of the droplet:

> ltc build-droplet sample-python https://github.com/cloudfoundry/python-buildpack

The command tells ltc to build the local directory into a droplet, naming it "sample-python" and using the cloudfoundry/python-buildpack. When you want to do something in other languages, the same process applies. Just be sure to specify an appropriate buildpack.

If everything is working correctly, you'll see Lattice stage the code and build a Droplet image:

Submitted build of sample-python
08/12 14:52:58.93 [BUILD|0] Successfully created container
08/12 14:53:07.60 [BUILD|0] -------> Buildpack version 1.5.0
08/12 14:53:07.67 [BUILD|0] -----> Installing runtime (python-2.7.10)
08/12 14:53:15.09 [BUILD|0] -----> Installing dependencies with pip
08/12 14:53:15.44 [BUILD|0]        Collecting Flask (from -r requirements.txt (line 1))
08/12 14:53:15.60 [BUILD|0]          Downloading Flask-0.10.1.tar.gz (544kB)
08/12 14:53:15.96 [BUILD|0]        Collecting Werkzeug>=0.7 (from Flask->-r requirements.txt (line 1))
08/12 14:53:16.04 [BUILD|0]          Downloading Werkzeug-0.10.4-py2.py3-none-any.whl (293kB)
08/12 14:53:16.11 [BUILD|0]        Collecting Jinja2>=2.4 (from Flask->-r requirements.txt (line 1))
08/12 14:53:16.18 [BUILD|0]          Downloading Jinja2-2.8-py2.py3-none-any.whl (263kB)
08/12 14:53:16.25 [BUILD|0]        Collecting itsdangerous>=0.21 (from Flask->-r requirements.txt (line 1))
08/12 14:53:16.31 [BUILD|0]          Downloading itsdangerous-0.24.tar.gz (46kB)
08/12 14:53:16.46 [BUILD|0]        Collecting MarkupSafe (from Jinja2>=2.4->Flask->-r requirements.txt (line 1))
08/12 14:53:16.53 [BUILD|0]          Downloading MarkupSafe-0.23.tar.gz
08/12 14:53:16.66 [BUILD|0]        Installing collected packages: Werkzeug, MarkupSafe, Jinja2, itsdangerous, Flask
08/12 14:53:16.79 [BUILD|0]          Running setup.py install for MarkupSafe
08/12 14:53:17.50 [BUILD|0]          Running setup.py install for itsdangerous
08/12 14:53:17.66 [BUILD|0]          Running setup.py install for Flask
08/12 14:53:18.00 [BUILD|0]        Successfully installed Flask-0.10.1 Jinja2-2.8 MarkupSafe-0.23 Werkzeug-0.10.4 itsdangerous-0.24
08/12 14:53:37.28 [BUILD|0] Exit status 0
08/12 14:53:37.51 [BUILD|0] Uploaded /tmp/droplet to http://192.168.11.11.xip.io:8444/blobs/sample-python/droplet.tgz.
08/12 14:53:37.52 [BUILD|0] Exit status 0
08/12 14:53:37.55 [BUILD|0] Uploaded /tmp/result.json to http://192.168.11.11.xip.io:8444/blobs/sample-python/result.json.
08/12 14:53:37.56 [BUILD|0] Exit status 0
08/12 14:53:37.60 [BUILD|0] Deleted http://192.168.11.11.xip.io:8444/blobs/sample-python/bits.zip.
08/12 14:53:37.61 [BUILD|0] Exit status 0
Build completed

To confirm that the droplet is registered in the cluster, you can run "ltc list-droplets" (or my favorite shorthand "ltc lsd").

> ltc lsd
Droplet		Created At		Size
sample-python	08/12 18:53:37.00	29.5M

Next, we will launch one instance with the "ltc launch-droplet" comment. You'll need to specify a name for the application (yes, multiple apps can be launched from the same droplet), as well as the name of the droplet to launch from. I used my imagination and named my app "test" in the example below:

> ltc launch-droplet test sample-python
No port specified. Defaulting to 8080.
Creating App: test
.08/12 15:04:55.53 [APP|0] Successfully created container
....08/12 15:05:00.01 [APP|0]  * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)
.08/12 15:05:00.39 [HEALTH|0] healthcheck passed
08/12 15:05:00.39 [HEALTH|0] Exit status 0

test is now running.
App is reachable at:
http://test.192.168.11.11.xip.io

And there we go! You should be able to see the app working by hitting http://test.192.168.11.11.xip.io. You can also now take advantage of the other features of Lattice, like scaling up / down (ltc scale), automatic health management and log aggregation.

Let's play with Lattice

To see the app list:

> ltc list
------------------------------= Apps =-------------------------------
App Name	Instances	DiskMB		MemoryMB	Route
test		1/1		0		128		test.192.168.11.11.xip.io, test-8080.192.168.11.11.xip.io => 8080

------------------------------= Tasks =------------------------------
No tasks to display.

To see the details of the "test" app:

> ltc status test
==========================================================================================
      test
------------------------------------------------------------------------------------------
Instances	1/1
Start Timeout	0
DiskMB		0
MemoryMB	128
CPUWeight	100
Ports		8080
Routes		test.192.168.11.11.xip.io => 8080
		test-8080.192.168.11.11.xip.io => 8080
Annotation	{"droplet_source":{"host":"192.168.11.11.xip.io","port":"8444","droplet_name":"sample-python"}}
------------------------------------------------------------------------------------------
Environment

MEMORY_LIMIT="128M"
PROCESS_GUID="test"
PORT="8080"

==========================================================================================
      Instance 0  [RUNNING]
------------------------------------------------------------------------------------------
InstanceGuid	a4aac9d3-fba5-453b-71ba-ec2004a17138
Cell ID		cell-01
Ip		192.168.11.11
Port Mapping	60004:8080
Uptime		3m57s
Crash Count 	0
CPU 		0.12%
Memory 		13.5M
------------------------------------------------------------------------------------------

Scaling up or down:

> ltc scale test 5
Scaling test to 5 instances
.....................
App Scaled Successfully

Now you should notice when you hit the app's web page that the index will jump around between the 5 instances on each refresh.

Also, the "ltc list" command should now show 5 instances running, and "ltc visualize" should show that there are 5 containers running on your cell (or cells if you deployed a Lattice cluster).

> ltc list
------------------------------= Apps =-------------------------------
App Name	Instances	DiskMB		MemoryMB	Route
test		5/5		0		128		test.192.168.11.11.xip.io, test-8080.192.168.11.11.xip.io => 8080

------------------------------= Tasks =------------------------------
No tasks to display.

> ltc visualize
Distribution
cell-01: •••••

And if you want fancy visualizations of your Lattice environment, check out xray.cf from the kind folks at Pivotal. It will connect to your Lattice environment from your browser, so it even works with your local VirtualBox Lattice instance.

Now let's break something!

Lattice includes both health management (keeping it's promises) as well as great logging visibility into the cluster's operations. To test this out, we can mess with the cluster by killing processes and watch as Lattice heals around our malevolence.

To start, open two terminal windows. In one, we'll use "vagrant ssh" to cause some damage. In the other, we'll watch as Lattice responds.

Get started by running "ltc logs test" in the first terminal to start watching the logs for your test application:

> ltc logs test

08/12 15:24:32.79 [HEALTH|0] healthcheck passed
08/12 15:24:32.79 [HEALTH|0] Exit status 0
...

Now in the other terminal, use "vagrant ssh" to log into the Vagrant box so we can do some damage:

> vagrant ssh
Welcome to Ubuntu 14.04.3 LTS (GNU/Linux 3.16.0-30-generic x86_64)

 * Documentation:  https://help.ubuntu.com/
vagrant@ubuntu-trusty-64:~$ ps -ef | grep iodaemon | grep -v grep | awk '{print $2}'
13469
14531
14580
14636
14665

That's the list of PIDs for the 5 instances of the Lattice app. Go ahead and kill -9 one or more of them!

vagrant@ubuntu-trusty-64:~$ sudo kill -9 13469 14531

If you go over to the terminal window showing logs, you will see that Lattice will notice the issue and quickly restart new app instances within the cluster automatically.

08/12 15:28:33.13 [HEALTH|0] healthcheck passed
08/12 15:28:33.14 [HEALTH|0] Exit status 0
08/12 15:28:38.50 [HEALTH|4] healthcheck passed
08/12 15:28:38.51 [HEALTH|4] Exit status 0
08/12 15:28:40.65 [APP|0] Creating container
08/12 15:28:40.66 [APP|4] Creating container
08/12 15:28:41.68 [APP|0] Successfully created container
08/12 15:28:42.34 [HEALTH|1] healthcheck passed
08/12 15:28:42.34 [HEALTH|1] Exit status 0
08/12 15:28:42.80 [APP|4] Successfully created container
08/12 15:28:45.47 [HEALTH|3] healthcheck passed
08/12 15:28:45.56 [HEALTH|3] Exit status 0
08/12 15:28:45.73 [HEALTH|2] healthcheck passed
08/12 15:28:45.81 [HEALTH|2] Exit status 0
08/12 15:28:48.97 [HEALTH|0] healthcheck failed
08/12 15:28:48.99 [HEALTH|0] Exit status 1
08/12 15:28:49.08 [APP|0]  * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)
08/12 15:28:49.55 [HEALTH|0] healthcheck passed
08/12 15:28:49.58 [HEALTH|0] Exit status 0
08/12 15:28:50.87 [APP|4]  * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)
08/12 15:28:51.03 [HEALTH|4] healthcheck passed
08/12 15:28:51.04 [HEALTH|4] Exit status 0

So there go... All the fun of Lattice, now with buildpacks!