Introducing “The A* Interviews”

Don’t care about the background? Skip to the bottom for the interviews.

Technology is a people business.

You learn that real quick.

Two years ago I left my job as a hedge fund analyst and decided to
join the tech world. Hopped up on three months of reading
TechCrunch, a recent viewing of The Social Network, and a few
breathless stories from colleagues who’d made the leap before me, I
thought that I could just show up and start building.

Hah.

I did teach myself to program, and it’s pretty darn cool to be able to turn an idea into a rough product. But the myth of the lone-ranger technologist is very much a myth. [1]

To get anything non-trivial done in technology, you need other people.

Obviously at some point you’ll need more hands to code. But even that’s not enough. There are a
few people who are just
born great programmers or
great entrepreneurs. I’m
not one of them, and you probably aren’t either. People like you and me need a whole network of friends,
advisors, and mentors to teach us and role-model for us. The world is a complicated place and none of us can even really know how much we don’t know. So we need people in our lives who can help us become our best
self. There are really only two ways to become good enough to survive in this game:

  1. Make painful, expensive mistakes and hope you draw the right lessons from them.
  2. Find people who are already great and learn their secrets.

The second sounds a lot easier, doesn’t it? Just one problem…

The People Who Can Help You Don’t Want To Meet You.

If you’ve already got a cadre of brilliant, dedicated tech world friends from your MIT frat house, then
you’re all set. If you don’t, you’re in trouble. When I left finance I
had no tech world network to speak of. I certainly have brilliant,
loyal friends (hi guys!) but few of them worked in tech and even fewer
had built great technology with their own hands. It immediately became
clear that I needed to “build my network”. But as anyone who has ever
proactively tried “to network” quickly realizes…networking is
pointless
. Unless you’re, like, a boat salesman or something. For
everyone else, trying to network is pointless because of one simple
rule: smart people are super busy. Even if they’re nice enough to
want you to succeed, they’ve got their own goals and they’d rather spend their limited time with people you can help them. Odds are, you can’t do that.

So what are the light-of-rolodex to do?

First, make friends. When you meet someone, don’t think “what can this person do for me?”
Instead, think “do I actually like and respect this person? Is this someone I could build a meaningful friendship
with?” If you can be friends, do. Make the effort and be patient. Some day, your real friends will find
creative ways to help you. Meanwhile your networking “contacts” won’t return
your calls.[2]

Second, find creative ways to help interesting people. Remember, every hour
they spend talking to you is an hour they can’t spend teaching robots
to nuke the moon. So you’ve
got to offer them something to make it worth their time. Not
money
. Interesting people usually have money (and you probably
don’t.) So you’ve got to get a bit clever or crazy or both.

So here’s MY crazy plan.

I want to get better at building stuff. To do that, I need to meet some great technologists, figure out what makes them tick, and then drink their magical think-juice.

Think juice

But really, who cares what I want? The important question is what can I offer? Well, good karma with great leverage of course.

Instead of asking awesome people out for coffee and a brain pickin’,
my plan is to give great technologists a platform to help a lot of
people. How? I’m going to interview them on camera and let them share their wisdom with
the world.
A coffee chat helps one person. A great interview can help thousands. After all, really, really good technologists are very
rare. Many programmers can work for years without every coming into contact with a truly great engineer. I’m hoping this podcast will be the next best
thing. And I’m hoping my quarries will be magnanimous enough share their think-juice at scale.

Either way, let’s find out.

Wait, but why “A*”?

Oh yeah! “A*” is
the name of a highly efficient
graph search algorithm. I’m
generally obssessed with graphs, and my theory is that all
technologists are located somewhere in a connected social graph. So if I traverse that
graph long enough I’m bound to end up identifying and talking to the
actual very best technologists in the world. A* search requires a
“heuristic” that estimates which potential paths are most promising,
and so my heuristic is to ask each interviewee for the names of the two
most competent programmers she knows. I’ll track those people down and repeat the cycle for as long as I can.

I don’t know if this will work (at all), but in the words of Astronaut
Matt Kowalski from the totally freaking rad must-see movie
Gravity, “I’ve got
a bad feeling about this mission.”

Without Further Ado…

#1: Roberto Thais, lead dev at YC-backed HireArt

#2: Carter Schonwald, Haskell guru and consultant at Wellposed

#3: Alexander Young, Math Professor at University of Washington

#4: Gershom Bazerman, organizer of the NYC Haskell meetup group

#5: “Frog Fractions” creator JIm Crawford

#6: C# Hacker Adam Milazzo

#7: “Jamestown” creator Tim Ambrogi

#8 Targo Tennisberg

#9 Former C# Committee Member Eric Lippert

#10 Founder of YC-backed startup RentMetrics and co-creator of HealthSherpa.com, Ning Liang

#11 Chuck Norris of Programming and All-Time StackOverflow champ, Jon Skeet

#12 Creator of noSQL database RavenDB, Ayende Rahein

#13 iPhone Jailbreaker, Agda Wizard, and notorious internet pumpkin – Daniel Peebles

#14 Peripatetic Hacker, Rob Ashton

#15 Creator of Flynn, Co-Founder of Hacker Dojo, Executive Producer of “Indie Game: The Movie”: Jeff Lindsay

#16: StackOverflow’s all-time Python champ, Alex Martelli

#17 Popularizer of the use of Singular Value Decomposition in Recommendation Systems and near-winner of the Netflix Challenge, Simon Funk (Recording Was Inaudible So Link Is To A Kitten) 🙁

#18 Strongly-Typed Emacs Acolyte, Bodil Stokke

#19 Creator of the Twisted Web Framework, Glyph Lefkowitz

Want to know about future interviews? Subscribe to my newsletter for interviews and general tech awesomeness or follow me on Twitter for interviews and Bhutan jokes.

And since I was asked on Twitter, here is a list of the books behind me during the interviews.

[1] Ironically, despite finance’s louche reputation, the myth of the lone-ranger
hedge fund trader is actually true.

[2] Because seriously, who calls people anymore? Use email.

A* Interview #3: Alexander Young

Play Alexander’s game KOMB.

Participate in the Indie Gam Jam.

Follow the zeppelining-about-town of Alexander’s gaming collective The Cloudboat Armada.

Play Frog Fractions.

/* change only these */
var youtubeID = “CQShYy6NRhA”;
var postID = “Alexander Young”;
/* change only these */

// DON’T CHANGE BELOW
$(“.video”).html(“

“);
var tag = document.createElement(‘script’);
tag.src = “http://www.youtube.com/player_api”;
var firstScriptTag = document.getElementsByTagName(‘script’)[0];
firstScriptTag.parentNode.insertBefore(tag, firstScriptTag);

function onPlayerStateChange(event) {
if (event.data ==YT.PlayerState.PLAYING) {
mixpanel.track(“Video played”, {
“youtubeID”: youtubeID,
“postID”: postID

});
}
if (event.data ==YT.PlayerState.ENDED || event.data==0) {
mixpanel.track(“Video ended”, {
“youtubeID”: youtubeID,
“postID”: postID
});
}
}

var player;
function onYouTubePlayerAPIReady() {
player = new YT.Player(‘player’,
{
videoId: youtubeID,
events: {‘onStateChange’: onPlayerStateChange}
});
}

mixpanel.track(“Post viewed”, {
“id”: postID
});

A* Interview #2: Carter Schonwald

Check out Carter’s consultancy Wellposed.

/* change only these */
var youtubeID = “H3JI8NP0D7c”;
var postID = “Carter Schonwald”;
/* change only these */

// DON’T CHANGE BELOW
$(“.video”).html(“

“);
var tag = document.createElement(‘script’);
tag.src = “http://www.youtube.com/player_api”;
var firstScriptTag = document.getElementsByTagName(‘script’)[0];
firstScriptTag.parentNode.insertBefore(tag, firstScriptTag);

function onPlayerStateChange(event) {
if (event.data ==YT.PlayerState.PLAYING) {
mixpanel.track(“Video played”, {
“youtubeID”: youtubeID,
“postID”: postID

});
}
if (event.data ==YT.PlayerState.ENDED || event.data==0) {
mixpanel.track(“Video ended”, {
“youtubeID”: youtubeID,
“postID”: postID
});
}
}

var player;
function onYouTubePlayerAPIReady() {
player = new YT.Player(‘player’,
{
videoId: youtubeID,
events: {‘onStateChange’: onPlayerStateChange}
});
}

mixpanel.track(“Post viewed”, {
“id”: postID
});

A* Interview #1: Roberto Thais from Hire Art

Go use HireArt to get a job or hire your next employee.

/* change only these */
var youtubeID = “tWEP1b4Svi4″;
var postID = ” Roberto Thais”;
/* change only these */

// DON’T CHANGE BELOW
$(“.video”).html(“

“);
var tag = document.createElement(‘script’);
tag.src = “http://www.youtube.com/player_api”;
var firstScriptTag = document.getElementsByTagName(‘script’)[0];
firstScriptTag.parentNode.insertBefore(tag, firstScriptTag);

function onPlayerStateChange(event) {
if (event.data ==YT.PlayerState.PLAYING) {
mixpanel.track(“Video played”, {
“youtubeID”: youtubeID,
“postID”: postID

});
}
if (event.data ==YT.PlayerState.ENDED || event.data==0) {
mixpanel.track(“Video ended”, {
“youtubeID”: youtubeID,
“postID”: postID
});
}
}

var player;
function onYouTubePlayerAPIReady() {
player = new YT.Player(‘player’,
{
videoId: youtubeID,
events: {‘onStateChange’: onPlayerStateChange}
});
}

mixpanel.track(“Post viewed”, {
“id”: postID
});

The Idiomatic Guide to Deploying Django in Production

Idiomatic Django Deployment – The Definitely Definitive Guide

By George London and Adam Burns

Wow, this guide is long. Why not skip it?
George and Adam are both available to freelance.

Or, if you’re kind of excited by how long this guide is, consider following George on Twitter or subscribing to his newsletter.

Overview

By the end of this guide, you should be have a simple but stable actually deployed
Django website accessible at a public URL
. So anyone in the world will be
able to visit “www.yourapp.com” and see a page that says “Hello World!”

You’ll go through the following steps:

  1. Setting up a host server for your webserver and your database.
  2. Installing and configuring the services your site will need.
  3. Automating deployment of your code.
  4. Learning what to do when things go wrong
  5. Next time: Going beyond the basics with caching, monitoring etc.

Updated August 9th, 2015 to use Django 1.8.

Why This Guide Is Needed

A few years ago, George taught himself to program in order to
build his first startup, LinerNotes. He
started out expecting that the hardest part would be getting his head
around the sophisticated algorithmic logic of programming. To his
surprise, he’s actually had to do very little difficult algorithmic work.[1] Instead, the hardest part was getting proficient at using the
many different tools in the programmer’s utility belt. From emacs to
gunicorn, building a real project requires dozens of different
tools. Theoretically, one can a priori reason through a red-black
tree but there’s just no way to learn emacs without the reading the
manual. LinerNotes is actually a lot more complicated under the hood
than it looks on the surface and so he had to read quite a lot of
manuals.

The point of this guide is to save you some of that trouble. Sometimes
trouble is good – struggling to design and implement an API builds
programming acumen. But struggling to configure nginx is just a waste
of time. We’ve found many partial guides to Django deployment but
haven’t found any single, recently updated resource that lays out the
simple, Pythonic way of deploying a Django site in
production
. This post will walk you through creating such a set
up. But it won’t introduce you to basic DevOps 101 concepts. See the bottom for a glossary of acronyms and explanatory footnotes (because Github breaks my intra-page links).[2]

Disclaimer: We’re definitely not the most qualified people to write this post. We’re just the only one dumb enough to try. If you object to anything in this post or get confused or find something broken, help make it better.
Leave a helpful comment (or even better submit a pull request to the Github repo.) The full text of this post is available in the repo and we’ll update this guide as appropriate.

Second disclaimerIf you’re working on a small project, aren’t doing anything unusual or custom with Django, and don’t anticipate needing to handle a large volume of traffic or expanding the scope of your project then you should seriously consider using a PaaS (platform as a service) provider like Heroku or Gondor.io. For a monthly fee, they handle all of the messy configuration (i.e. this guide) for you (as long as your app is structured according to their specifications.) They’re not necessarily easier to get started with than this guide, but they do save you from a lot of down-the-road hassle of administering your own servers (e.g. doing security patches.)

Overview of the Final Architecture

Our example site is just a “hello world” app, but this is going to be the most well-implemented, stable, and scalable
“hello world” application on the whole world wide web. Here’s a diagram of how
your final architecture will look:

Architecture Diagram

Basically, users send HTTP requests to your server, which are intercepted and
routed by the nginx load balancer. Requests for dynamic content will be routed to
your WSGI[3] server (Gunicorn) and requests for static content will be served
directly off the server’s file system. Gunicorn has a few helpers, memcached and celery,
which respectively offer a cache for repetitive tasks and an asynchronous queue
for long-running tasks.

We’ve also got our Postgres database (for all your lovely models) which we run on a
separate EC2 server.[4]

See below for a more detailed description of what each component
actually does.

Set Up Your Host Servers

Set up AWS/EC2

Since this guide is trying to get you to an actual publicly accessible site,
we’re going to go ahead and build our site on the smallest, freest Amazon EC2 instance available, the trusty “micro”. If you don’t want to use
EC2, you can set up a local virtual machine on your laptop using
Vagrant or use your own existing server (you’ll have to tweak my scripts a little). The Docker project has been picking up steam lately but at this point we believe that running Django inside of Docker on EC2 (i.e. running a virtual machine inside a virtual machine) is an unnecessary complication. But don’t be Docker-sad! We will be using Docker to run our deployment tools in an isolated, virtual container on our laptops.

Anyway, we’re going to use EC2 to set up the smallest possible host for our webserver and another
one for our database.

For this tutorial, you’ll need an existing EC2 account. There are many tutorials on setting up an account so I’m not going to walk you through the account setup.

Python has a very nice library called boto for administering AWS
from within code. And another nice tool called Fabric for creating
command-line directives that execute Python code that can itself execute
shell commands on local or remote servers. We’re going to use Fabric
to definite all of our administrative operations, from
creating/bootstrapping servers up to pushing code. I’ve read that Chef (which we’ll use below) also has a plugin to launch EC2 servers but I’m going to prefer boto/Fabric because they give us the option of embedding all our “command” logic into Python and editing it directly as needed.

Start off by cloning the Github repo for this project onto your local machine.

git clone git@github.com:rogueleaderr/definitive_guide_to_django_deployment.git
cd definitive_guide_to_django_deployment

The github repo includes a fabfile.py[7] which provides all the
commandline directives we’ll need. But fabfiles are pretty intuitive
to read so try to follow along with what each command is doing.

First, we need to define AWS settings. In keeping
with the principles of the Twelve Factor App
we store configuration either in environment variables or in config files which
are not tracked by VCS. You can find your AWS access and secret keys on your
AWS security page.

echo '
AWS_ACCESS_KEY_ID=<your key here>
AWS_SECRET_ACCESS_KEY=<your secret key here>
AWS_DEFAULT_REGION=<your region here e.g. us-east-1>
AWS_SECURITY_GROUP_NAME=<your security group name>
AWS_INSTANCE_TYPE=t1.micro
AWS_AMI_ID=<see http:></see>
AWS_SSH_KEY_NAME=<your ec2 key name>
AWS_SSH_PORT=22' > deploy/environment
chmod 600 deploy/environment

Make sure you fill out the values between angle brackets with .

These settings will be exported as enviornment variables in the docker
container where both fabric and the AWS CLI will read them. We recommend using
an AMI for a “free-tier” eligible Ubuntu 14.04 LTS image.

If you already have an EC2 SSH key pair that you want to use, make sure you copy it to the deploy folder (otherwise skip this step and we’ll create one for you automatically):

cp -p <path to your ec2 key> deploy/ssh

We’re also going to create a settings file that contains all the configuration for our actual app.

echo '{}' > deploy/settings.json
chmod 600 deploy/settings.json

Now it’s time for Docker. Follow one of the guides for installing Docker and its pre-requisites on your development machine.

Docker runs a “container”, i.e. a lightweight virtual machine running (in this case) Ubuntu in an isolated environment on your development machine. So we can safely install all the tools we need for this guide inside the container without needing to worry about gumming up your development machine (or other things on your machine causing incompatibilities with these tools.)

If you’re on OSX then as of this writing the two ways to start a Docker container are using Boot2Docker or Kitematic. For either of those, you’ll need to open the app inside your Applications folder. This will start (or give you the opportunity to start) the VM that Docker uses to create its container VM’s. It may also push you into a new terminal, so be sure to

cd <your guide directory>

…if you’re not already there.

Now we can build and run the container for our deploy tools:

docker build -t django_deployment .
docker run --env-file=deploy/environment -tiv $(pwd):/project/django_deployment django_deployment /bin/bash

This step may take a while to download everything, so why not watch a PyCon video while you wait?

Once the process finishes, you’ll be at a bash shell in a container with all the tools installed. The
project’s root directory has been mounted inside the container so un-tracked
files like settings will be stored on your workstation but will still be
available to fab and other tools.

If Docker is giving you errors about being unable to connect and you’re on OSX, make sure that you’re running these commands in a terminal tab that was opened by Boot2Docker or Kitematic.

Warning: If you are running boot2docker on OS X you may need to restart
the boot2docker daemon on your host when you move to new networks (e.g. from
home to a coffee shop).

Now we’re going to use a Fabric directive to setup our AWS account [6] by:

  1. Configuring a keypair ssh key that will let us log in to our servers
  2. Setting up a security group that defines access rules to our servers

To use our first fabric directive and setup our AWS account, go to the directory where our fabfile lives and
do

fab setup_aws_account

Launch Some EC2 Servers

We’re going to launch two Ubuntu 14.04 LTS servers, one for our web host
and one for our database. We’re using Ubuntu because it it seems to be
the most popular linux distro right now, and 14.04 because it’s a (L)ong (T)erm (S)upport
version, meaning we have the longest period before it’s officially
deprecated and we’re forced to deal with an OS upgrade. Depending on what AMI you choose in your settings, you may end up with a different version. (If in doubt, choose a i386 ebs-ssd LTS version.)

With boto and Fabric, launching a new instance is very easy:

fab create_instance:webserver
fab create_instance:database

These commands tell Fabric to use boto to create a new “micro”
(i.e. free for the first year) instance on EC2, with the name you
provide. You can also provide a lot more configuration options to this
directive at the command line but the defaults are sensible for now.

It may be a few minutes before new instances are fully started. EC2 reports
them online when the virtual hardware is up but Linux takes some time to boot
after that.

Now you can ssh into a server:

fab ssh:webserver

If you create an instance by mistake, you can terminate it with

fab terminate_instance:webserver

Install and Configure Your Services

We need a project to deploy. Your easiest option is to use the sample project I’ve created. The deployment will clone it onto webservers automatically, but you might want your own local clone so you can follow along:

git clone git@github.com:rogueleaderr/django_deployment_example_project.git

If you don’t do that, then you need to…

Make sure your project is set up correctly:

This guide assumes a standard Django 1.5+ project layout with a few small tweaks:

  • Your settings should be comprised of three files:

    app
    --settings
        --__init__.py # tells python how to import your settings
         --base.py # your default settings
         --local_settings.py # settings that will be dynamically generated from your settings.json. don't track in git
    

    And your __init__.py should consist of:

    # application_python cookbook expects manage.py in a top level
    # instead of app level dir, so the relative import can fail
    try:
        from .<project_name>.<project_name>.settings.base import *
    except ImportError:
        from <project_name>.settings.base import *
    
    try:
        from local_settings import *
    except ImportError:
        pass
    

    Our bootstrapping process will create a local_settings.py but to develop locally you’ll need to make one manually with your database info etc. (Don’t check it into git.)

  • We’re serving static files with dj-static. To use dj-static, you need a couple project tweaks:

    In base.py, set

    STATIC_ROOT = 'staticfiles'
    STATIC_URL = '/static/'
    

    Modify your wsgi.py:

    from django.core.wsgi import get_wsgi_application
    from dj_static import Cling
    
    application = Cling(get_wsgi_application())
    
  • Your installed apps must contain your project app and also djcelery.

Understand the services

Our stack is made up of a number of services that run
semi-independently:

Gunicorn: Our WSGI webserver. Gunicorn receives HTTP requests fowarded to it from nginx, executes
our Django code to produce a response, and returns the response which nginx transmits back to the client.

Nginx: Our
load balancer (a.k.a. “reverse proxy server”). Nginx takes requests from the open internet and decides
whether they should be passed to Gunicorn, served a static file,
served a “Gunicorn is down” error page, or even blocked (e.g. to prevent denial-of-service
requests.) If you want to spread your requests across several Gunicorn nodes, it just takes a tiny change to your nginx configuration.

Memcached: A simple in-memory key/value caching system. Can save
Gunicorn a lot of effort regenerating rarely-changed pages or objects.

Celery: An async task system for Python. Can take longer-running
bits of code and process them outside of Gunicorn without jamming up
the webserver. Can also be used for “poor man’s” concurrency in Django.

RabbitMQ: A queue/message broker that passes asynchronous tasks
between Gunicorn and Celery.

Supervisor: A process manager that attempts to make sure that all key services stay
alive and are automatically restarted if they die for any reason.

Postgres: The main database server (“cluster” in Postgres
parlance). Contains one or more “logical” databases containing our
application data / model data.

Install the services

We could install and configure each service individually, but instead
we’re going to use a “configuration automation” tool called
Chef. Chef lets us write simple Ruby
programs (sorry Python monogamists!) called Cookbooks that automatically
install and configure services.

Chef can be a bit intimidating. It provides an entire Ruby-based
DSL for expressing configuration. And it
also provides a whole system (Chef server) for controlling the
configuration of remote servers (a.k.a. “nodes”) from a central location. The DSL is
unavoidable but we can make things a bit simpler by using “Chef Solo”, a stripped down version of Chef that does away with the whole central server and leaves us with
just a single script that we run on our remote servers to bootstrap our
configuration.

(Hat tip to several authors for blog posts about using Chef for Django[8])

Wait, a complicated ruby tool? Really?

Yes, really. Despite being in Ruby[9], Chef has some great advantages that make it worth learning (at least enough to follow this guide.)

  1. It lets us fully automate our deployment. We only need to edit one configuration file and run two commands and our entire stack configures itself automatically. And if your servers all die you can redeploy from scratch with the same two commands (assuming you backed up your database).
  2. It lets us lock the versions for all of our dependencies. Every package installed by this process has its version explicitly specified. So this guide/process may become dated but it should continue to at least basically work for a long time.
  3. It lets us stand on the shoulders of giants. Opscode (the creators of Chef) and some great OSS people have put a lot of time into creating ready-to-use Chef “cookbooks” for nearly all our needs. Remember, DRW-EWTMOTWWRS (Don’t Re-Invent the Wheel, Especially When the Maker of the Wheel Was Really Smart).

Okay, buckle up. We’re going to need to talk a little about how Chef works. But it’ll be worth it.

At the root, Chef is made up of small Ruby scripts called recipes that express
configuration. Chef declares configuration rather than executing a
series of steps (the way Fabric does). A recipe is supposed to describe all the resources that
are available on a server (rather than just invoking installation
commands.) If a resource is missing when a recipe is run, Chef will
try to figure out how to install that resource. If a configuration file has the wrong information, Chef will fix(/brutally overwrite) it. Recipes are
(supposed to be) idempotent, meaning that if you run a recipe and
then run it again then the second run will have no effects.

But which recipes to run? Chef uses cookbooks that
group together recipes for deploying a specific tool (e.g. “the git
cookbook”). And Chef has a concept called “roles” that let you specify
which cookbooks should be used on a given server. So for example, we
can define a “webserver” role and tell Chef to use the “git”, “nginx”
and “django” cookbooks. Opscode (the makers of Chef) provide a bunch
of pre-packaged and (usually well maintained) cookbooks for common
tools like git. And although Chef cookbooks can get quite complicated, they are just code and so they can be version controlled with git.

Chef, make me a server-which.

We use some tools that simplify working with Chef:

We’re going to have two nodes, a webserver and a database. We’ll have four roles:

  1. base.rb (common configuration that both will need, like apt and git)
  2. application_server.rb (webserver configuration)
  3. database.rb (database configuration)
  4. deploy.rb (to save time, a stripped down role that just deploys app code to the app server)

The role definitions live in chef_files/roles. Now we just need to tell Chef which roles apply to which nodes, and we do that in our chef_files/nodes folder in files named “{node name}_node.json”. If you use names other than “webserver” and “database” for your nodes, you must rename these node files.

Any production Django installation is going to have some sensitive
values (e.g. database passwords). Chef has a construct called data
bags
for isolating and storing sensitive information. And these bags
can even be encrypted so they can be stored in a VCS. Knife solo lets us create a databag and encrypt
it. Fabric will automatically upload our databags to the server where
they’ll be accessible to our Chef solo recipe.

Start by loading the values we need into our settings.json file. Be sure to
update settings anytime you create new servers.

APP_NAME should be the name of the project in your repo, as it would appear
in a Python import path.

echo '{
"id": "config_1",
"POSTGRES_PASS": "<your db password>",
"DEBUG": "False",
"DOMAIN": "<your domain name>",
"APP_NAME": "<name of python package inside your repo: e.g. deployment_example_project>",
"DATABASE_NAME": "<your database name>",
"REPO": "<your github repo name: e.g. django_deployment_example_project>",
"GITHUB_USER": "<your github username: e.g. rogueleaderr>",
"DATABASE_IP": "DB_IP_SLUG",
"EC2_DNS": "WEB_IP_SLUG"
}' 
| sed -e s/DB_IP_SLUG/`cat deploy/fab_hosts/database.txt`/ 
| sed -e s/WEB_IP_SLUG/`cat deploy/fab_hosts/webserver.txt`/ 
> deploy/settings.json

Now we need an encryption key (which we will NOT store in Github):

cd chef_files
openssl rand -base64 512 > data_bag_key
cd ..
# if you aren't using my repo's .gitingore, add the key
echo "chef_files/data_bag_key" >> .gitignore

Now we can use the knife-solo to create an encrypted data bag from our settings.json file:

cd chef_files
knife solo data bag create config config_1 --json-file ../deploy/settings.json
cd ..

Our chef_files repo contains a file Berksfile that lists all the cookbooks we are going to install on our server, along with specific versions. Knife solo will install all of these with a tool called Berkshelf, which I honestly assume is named after this. If a cookbook becomes dated, just upgrade the version number in chef_files/Berksfile.
Now we’re going to use Fabric to tell Chef to first bootstrap our database and then bootstrap our webserver. Do:

fab bootstrap:database
fab bootstrap:webserver

This will:

  1. Install Chef
  2. Tell Chef to configure the server

A lot of stuff is going to happen so this may take a while. Don’t worry if the process seems to pause for a while. But if it exits with an error please create an issue on Github describing what went wrong (or better yet, leave a pull a request to fix it.)

What is this magic?

Chef actually does so much that you might be reluctant to trust it. You may(/should) want to understand the details of your deployment. Or you may just distrust Ruby-scented magic. So here’s a rough walk-through of everything the bootstrap scripts do.

Database

The database role first installs the essential base packages specified in base.rb, i.e. apt, gcc, etc, and sets up our ubuntu admin user with passwordless sudo.

Then we run our custom database recipe that:

  1. Installs Postgres on the server

  2. Modifies the default postgresql.conf settings to take full advantage of the node’s resources (dynamically calculated using a cookbook called ohai.) Changes the Linux shmmax/shmall paramaters as necessary.

  3. Tells Postgres to listen on all ports and accept password authenticated connections from all IP’s (which is okay because we use Amazon’s firewall to block any connection to the Postgres node from outside our security group.)

  4. Creates our application database using a password and database name read in from our settings.json file.

  5. Restarts Postgres to pick up the configuration changes.

Webserver

Again, install base packages per base.rb.

Then runs our main application server setup recipe:

  1. Runs Opscode cookbooks to install basic packages, including: git, nginx, python-dev, rabbit-mq, memcached, and postgres-client.

  2. Reads configuration variables from our encrypted data bag (made from settings.json)

  3. Updates Ubuntu and installs the bash-completion package. Creates a .bashrc from a template.

  4. Creates an nginx configuration for our site from a template, load it into nginx’s configuration folder and restart nginx.

  5. Deploy our Django app, which consists of:

  • Create a folder called /srv/<app_name> that will hold our whole deployment

  • Create a folder called /srv/<app_name>/shared with will hold our virtualenv and some key configuration files

  • Download our latest copy of our Github repo to /srv/<app_name>/shared/cached-copy

  • Create a local_settings.py file from a template and include information to connect to the database we created above (details loaded from our data bag.)

  • “Migrate” (i.e. sync) the database with manage.py syncdb. The sync command can be overwritten if you want to use South.

  • Install all our Python packages with pip from requirements/requirements.txt.

  • Run manage.py collectstatic to copy our static files (css, js, images) into a single static folder

  • Install gunicorn and create a configuration file from a template. The config file lives at /srv/<app_name>/shared/gunicorn_config.py.

  • Bind gunicorn to talk over a unix socket named after our app

  • Install celery, along with celery’s built-in helpers celerycam and celerybeat. Create a configuration file from a template. The config lives at /srv/<app_name>/shared/celery_settings.py.

  • Create “supervisor” stubs that tell supervisor to manage our gunicorn and celery processes.

  • Copy the ‘cached-copy’ into a /srv/<app_name>/releases/<sha1_hash_of_release> folder’. Then symlink the latest release into /srv/<app_name>/current which is where where the live app ultimately lives.

  1. Create a “project.settings” file that contains the sensitive variables (e.g. database password) for our Django app.

Hopefully this list makes it a bit more clear why we’re using Chef. You certainly could do each of these steps by hand but it would be much more time consuming and error-prone.

Make it public.

If Chef runs all the way through without error (as it should) you’ll now have a ‘Hello World’ site accessible by opening your browser and visiting the “public DNS” of your site (which you can find from the EC2 management console or by doing cat deploy/fab_hosts/webserver.txt). But you probably don’t want visitors to have to type in “103.32.blah-blah.ec2.blah-blah”. You want them to just visit “myapp.com” and to do that you’ll need to visit your domain registrar (e.g. GoDaddy or Netfirms) and change your A-Record to point to the IP of your webserver (which can also be gotten from the EC2 console or by doing):

ec2-describe-instances | fgrep `cat deploy/fab_hosts/webserver.txt` | cut -f17

Domain registrars vary greatly on how to change the A-record so check your registrar’s instructions.

By default, EC2 provides a “public DNS” listing for instances which looks like ec2-54-166-68-245.compute-1.amazonaws.com

Any time you stop and then start an EC2 instance, the DNS address gets re-assigned. If you’re putting your website on one of these nodes, that’s not ideal
because you would have to update your A-Record every time you need to stop the instance for any reason. (And it’s even worse because A-Records updates can take 24-48 hours to
propogate through the internet so your site may be unreachable for a while.)

To avoid that problem, Amazon lets you associate an “Elastic IP” with a given server. An Elastic IP is a fixed IP address that will
stay with the the server even if it’s stopped.

To associate an elastic IP with your webserver, you can do

 fab associate_ip:webserver

Note: AWS has a soft limit of 5 elastic IP’s per account. So if you already have elastic IP’s allocated, you may need to delete or reassign them in your AWS management console.

After you change the IP, you’ll also need to change your ALLOWED_HOSTS django setting. The fab command updates the host settings in your config files in the local guide directory you’ve been working in,
but to push those changes to your server you’ll need to update your secret databag:

cd chef_files
knife solo data bag create config config_1 --json-file ../deploy/settings.json
cd ..

And re-bootstrap the server to update the settings:

fab bootstrap:webserver

Automatically Deploy Your Code

Well, this is simple. Just commit your repo and do

git push origin master

Then back in the deployment guide folder, do:

fab deploy:webserver

Debugging:

Deploy

  1. If you edit deploy/settings.json, remember to regenerate the chef data bag.
  2. If you terminate and re-launch an instance, remember to update the IP and DNS fields in deploy/settings.json.
  3. The vars defined in deploy/environment are read when the docker container starts. If you edit them, exit and re-run the docker container so they’re re-read.
  4. The cache in chef_files/cookbooks can get outdated. If you’re seeing Chef errors, try deleting the contents of that directory and starting over.
  5. By default Chef will try to roll back failed launches, but that can make it hard to figure out why the launch failed. To disable rollback, add rollback_on_error false to chef_files/site-cookbooks/django_application_server/recipes/default.rb in the same place as its repository and revision options.

Nginx

Nginx is the proxy server that routes HTTP traffic. It has never once gone
down for me. It should start automatically if the EC2 server restarts.

If you need to start/restart nginx, log in to the webserver and do:

sudo service nginx restart

If nginx misbehaves, logs are at:

/var/log/nginx/

If, for some reason, you need to look at the nginx conf it’s at:

sudo emacs /etc/nginx/sites-available/<app_name>.conf

If you need to edit it, avoid making changes to any conf files on the server, instead change:

chef_files/site-cookbooks/django_application_server/templates/default/nginx-conf.erb

And rerun the Chef bootstrap process. It’s idempotent so it won’t change anything else (unless you’ve been tinkering directly with the server in which case your changes will be “reeducated”).

RabbitMQ

Another service that’s started automatically. I have literally never had to interact
directly with it. But if can also be restarted by

sudo service restart rabbitmq

Memcached

Memcached is also a service and starts automatically if the EC2 node restarts. Unless you’ve designed a monstrosity, your site
should also continue to function if it dies (just be slow). Caching issues can sometimes
cause weird page content, so if something seems unusually bizarre try flushing the cache
by restarting memcached:

sudo service restart memcached

Memcached is pretty fire and forget…since it’s in memory it’s theoretically possible it
could fill up and exhaust the memory on the webserver (I don’t have a size cap and I make my TTL’s
very long) but that has never happened so far. If it does, just reset memcached and it
will clear itself out.

To start Gunicorn/Celery:

Gunicorn and the Celery Workers are controlled by Supervisor, which is a Linux process runner/controller. Supervisor will automatically restart them if
they’re terminated abnormally.

The Supervisor configuration is located at:

/etc/supervisor/conf.d/<app_name>.conf

To restart gunicorn and celery together, simply do:

fab restart

To restart manually, you can use a Supervisor utility called supervisorctl that lets you check the status of and
restart processes. So if you need to restart gunicorn or celery, you can do:

sudo supervisorctl restart <app_name>
sudo supervisorctl restart <app_name>-celeryd

Or to check process status, just do

sudo supervisorctl status

Supervisor routes all log output from managed processes to /var/log/supervisor/<process-name-and-long-character-string>. So if your server is behaving badly you can start there.

I keep a GNU screen active in the log directory so
I can get there quickly if I need to. You can get there with

screen -r <my screen>

Postgres

Postgres is a very stable program but can be a bit touchy on a small server under heavy load. It’s probably
the most likely component to give you trouble (and sadly basically your whole site becomes totally
non-operational if it goes down.)

Postgres runs as a service so if you need to restart it (try not to need to do this) you
can do:

sudo service postgresql restart

The disk can also fill (especially if something gets weird with the logging.) To check disk space:

df -h

If a disk is 99% full, find big files using

find / -type f -size +10M -exec ls -l {} ;

EC2 instances (larger than our micro type) all have “instance store” disks on /mnt, so you can copy obviously
suspicious files onto the instance store and let me sort it out later.

If that’s not enough, check the logs for the service at /var/log/postgresql/

Future Crimes of Configuration

This guide has gotten long enough for now, so I’m going to see how it’s recieved before delving into advanced topics. But here are a few quick suggestions:

Set Up Monitoring

There are a bunch of open and closed source solutions. They’re all a bit more complicated than I’d like to set up. But here’s what I personally use:

Datadog Monitoring

Datadog makes pretty metric dashboards. It will automatically monitor server CPU/memory/etc status. Datadog can send an
alert if there’s no CPU activity from the webserver or the database (probably meaning the
EC2 servers are down.) And it can also hook into a custom statsd library and lets you emit/graph whatever metrics you want from anywhere in your app. You just have to decorate your code by hand.

Notifications / PagerDuty

PagerDuty is a website that will call or email you if something goes wrong with a
server. I’ve configured it to email/SMS if anything goes wrong with my site.

Django by default automatically emits error emails, which I:

  1. route to PagerDuty so it automatically sets up an “incident” and SMS’s me
  2. sends an email to me with the details of the error

Occasionally these emails are for non-serious issues but there’s no easy way to
filter. It can be a bit chatty if you haven’t chased down all the random non-critical errors in your app, but it helps save you from being unaware your site was down for 12 hours.

Connection pooling

As of Django 1.5, Django opens a new Postgres connection for every request, which requires a ~200ms SSL renegotiation. Skip that overhead by using a connection pooler like django-postgrespool. You can also use PgBouncer on your Postgres server to make sure you don’t get overwhelmed with incoming connections.

Apparently Django 1.6 includes a built-in connection pooler.

Cache Settings

A lot of what Django does from request to request is redundant. You can hugely increase responsiveness and decrease server load by caching aggressively. Django has built in settings to cache views (but you have to enable caching yourself.) You can also use cache-machine to cache your models and significantly reduce your database load.

Backup

The nice thing about this Chef setup is that if anything goes wrong with your webserver, it might actually be faster to deploy a new one from scratch and fail over than to try to restore your broken server. But you’ve still got to back up your database. Nothing can help you with deleted data. Postgres has a number of options, including streaming replication and (w)rite(a)head (l)og WAL shipping to S3.

South migrations

Just use them. Also apparently baked into Django 1.6.

Gevent

In my experience, by far the biggest cause of slowness in Django is I/O or network requests (e.g. calling an external API to supply some data for a widget.) By default, Python blocks the thread making the call until it’s done. Gunicorn gives you “workers” which run in separate threads, but if you have four workers and they all block waiting for a long database query then your whole site will just hang until a worker is free (or the request times out.)

You can make things way faster by using “green threading” via gevent. Green threading is conceptually complicated (and occasionally buggy) but the basic idea is that one thread can contain many “green” threads. One one green thread runs at a time, but if it needs to wait for I/O it cedes control to another green thread. So your server can accomodate way more requests by never blocking the gunicorn worker threads.

Wrap up

And there you go. You’ve got a production-ready Django website. It’s reasonable secure, easy to update, cheap to run, fully customizable and should be able to easily handle the kind of traffic a new site is likely to get. If you need more power, just shutdown your EC2 instances and upgrade to a larger instance type. Or get fancy by spinning up more webservers and using the load balancer to route requests among them.

Anyway, thanks for making it this far! If you’ve got any suggestions for how to do anything in this guide better, please leave a comment or a pull request! And if you build any custom Chef code for your own deployment, please consider contributing it back to this guide or to the official application_python cookbook.

And if you enjoy this kind of material, consider following me on Twitter or subscribing to my newsletter.

Notes

[1] And Python has existing libraries that implement nearly any algorithm better than I could anyway.

[2] I’ll
try to be gentle but won’t simplify where doing so would hurt the
quality of the ultimate deployment. If you
don’t know what a load balancer or an SSH key is, you’re going to have
a hard time following along. But Google can help you with that. Don’t worry, I’ll be here when you get back.

[3] You can run Postgres on the same VM, but putting it on a
separate box will avoid resource contention and make your app more scalable. You also can run nginx and celery on their own VM’s which will make your site super scalable. But if you need this guide then you’re probably not seeing enough traffic to make that worth the added complexity.

[4] More about WSGI

[6] For development I enjoy VirtualenvWrapper which makes switching between venv’s easy. But it installs venvs by default in a ~/Envs home directory and for deployment we want to keep as much as possible inside of one main project directory (to make everything easy to find.)

[7] Hat tip to garnaat for
his AWS recipe to setup an account with boto

[8] Hat tip to Martha Kelly for her post on using Fabric/Boto to deploy EC2

[9] Chef/Django posts:

[*] Yes, there are other configuration automation tools. Puppet is widely used, but I find it slightly more confuing and it seems less popular in the Django community. There is also a tool called Salt that’s even in Python. But Salt seems substantially less mature than Chef at this point.

Glossary

AMI

– An "AMI” is an Amazon Machine Image, i.e. a re-loadable snapshot of a configured system.

CLI

– command line interface. Amazon gives us a set of new command line “verbs” to control AWS.

EC2

– Elastic Compute Cloud, Amazon’s virtual server farm.

AWS

– Amazon Web Services, the umbrella of the many individual cloud services Amazon offers

DSL

– Domain specific language. Aka a crazy mangled version of Ruby customized to describe service configuration.

VCS

– Version control system, e.g. git or SVN or mercurial

Bibliography

Randall Degges rants on deployment

Rob Golding on deploying Django

Aqiliq on deploying Django on Docker

How to use Knife-Solo and Knife-Solo_data_bags

Tumblr Eats a Pelican: The Problem With Static Website Generators

Tl;dr: Don’t bother switching. Tumblr is better.

Guys, my blog is failing.

I’ve been blogging (via Tumblr) at http://www.rogueleaderr.com for two years and the Google analytics data speaks for itself:

Except for one random post about subverting Megabus wifi, my posts hardly ever see more than 200 hits (few of whom actually read to the end). I’m grateful for the readers I have, but I’m only getting marginally more amplification than shouting from a literal soapbox.

So I’ve decided to step up my blogging game

I’ve got some ideas I want to spread. And it’s time to find out whether no one else cares about those ideas or whether I’m just doing a bad job of broadcasting.

So I’m taking a few steps to step things up:

  1. Creating an email newsletter. (You can sign up here)
  2. Getting serious about promoting my blog posts.
  3. Redesigning my blog.

1 & 2 are topics for a different day. This post is about #3, my decision to switch from trusty Tumblr to a static website generator.

Spoiler: I will be wrecked by “what you know you know that just ain’t so…”

I should have listened to Mark Twain. That’s the main moral of this story. The longer story is that I let bad assumptions waste quite a lot of my time.

If you spent any time on Hacker News et al., you’re probably familiar with the classic “hacker profile page”, as best exemplified by Tom Preston-Werner or Kenneth Reitz or Paul Graham. These inimitably classy and minimalist sites generally contain a short tagline-style bio of the hacker, a list of projects, a list of blog posts, and maybe a contact form or easter egg.

I may not have a tenth the skill of any of those chaps, but I know how to steal a good idea when I see one. I wanted my blog to look like theirs and so I assumed (falsely!) that I needed to build my blog the way they built theirs. That meant a static website generator since, after all, Tom Preston-Werner popularized the very idea of static website generators with his tool Jekyll.

If you’re not familiar, static website generators (hereafter “SWG’s”) let you store all the components of your website (including each individual blog post) as files in a directory (often in a markup language like Markdown). You run a script and presto everything is compiled to interlinked HTML files which can be uploaded and served directly from a webserver or even for free from Github.

Static is the move.

As I started researching SWG’s, I quickly realized that there are a lot of options. The most popular seems to be the Ruby-based Jekyll, the original heavy weight. I read Ruby at a 3rd grade level and SWG’s in general seem to require a lot of tweaking, so I was inclined to prefer a Python-based option.

There are plenty of those as well, but I found a glowing post about one called Pelican so, in the absence of any clear compelling differences among the options, I decided Pelican was the way to go. It’s pretty simple to setup a basic Pelican site and even to deploy it publicly on Github Pages (probably 30 minutes work.) But a few minutes later, the problems started.

First, I wanted to create an “About Me” page. That’s quite easy, if you already know how to do it. If you don’t the Pelican docs (and limited tutorials available online) require some pretty close study.

Next, I wanted to make my blog pretty. Pelican ships with some off the shelf themes. IMHO, they’re all somewhere between ugly and “meh.” So I hoped over to CreativeMarket (with a big discount from AppSumo burning a whole in my pocket) and looked for a theme. There are hundreds of WordPress and Tumblr themes for sale, but nothing for Pelican and only a few for “generic HTML site”. Before accepting the cost of customizing one of those to fit Pelican, I googled and found a generic Svbtle ripoff theme for Pelican.

Sure it’s not responsive, but only 7% of my traffic is mobile. So…goodbye forever iPhone losers!

I need to build a link to the past

Next problem: my blog gets a small but non-trivial amount of inbound search traffic. Simply shutting down the old site would kill all traffic to those archives. So I needed to import my posts from Tumblr. But as I’ve learned the hard way on my current project LinerNotes, Googlebot ain’t too bright. Even if I redirect the rogueleaderr.com domain to my pretty new static site, any change in the URL’s of my old posts will at least break any external links to my posts and at worst will cause Google to penalize my posts as both broken links and duplicate content. Plus, cool URI’s don’t change.

A site deployed behind a webserver like nginx could use URL rewrites to redirect incoming URL’s (assuming the URL slugs could be preserved). But…Github pages doesn’t allow redirects (except perhaps via Javascript; I never got that far.)

And I’m back to Tumblr

While googling for how to import from Tumblr to Pelican, I stumbled upon a Tumblr manual page and had a massive facepalm moment.

It turns out that Tumblr has a feature called “Pages” that lets you include and link to static pages right inside your Tumblog.

All you have to do is press a button and drop in a little HTML and you’ve got your own “About Me” page. In other words, basically all the customization power I was hoping to get from a static site is already available in Tumblr with none of the hassle of learning how to use a generator, managing the migration, and maintaining a separate site.

With about 45 minutes of work and the help of the Tumblr Svblte Theme, I was able to create a site that looked and worked better than my Pelican site and preserved all my lovely links and Tumblr followers.

And it’s even responsive.

Moral of the story

Tumblr is a lot more powerful than I realized (as probably is WordPress, which I haven’t used). And Tumblr gives you an awful lot of bang for the setup buck. I’m now struggling to think of a use case that SWG’s allow and Tumblr doesn’t. (Maybe if you want lots of highly customized static pages, or if you want to run a webserver with complicated URL trickery.) If you’re publishing a lot of varied types of content, you’ll probably be better served by a fullblown CMS. SWG’s are only marginally less complicated to learn.

I’ve read worries about “what if Tumblr dies off like Posterous” – well, Tumblr allows export of your posts via an API, so you can deal with learning Pelican as soon as Yahoo writes off its $1bln investment.

One good argument I’ve heard is that storing a blog in a directory allows version control. That’s true, but I basically never change old posts. And it’s easy enough to save a copy of the HTML of a Tumblr theme before changing it.

All in all, the hassles of a Static Website Generator don’t outweight the costs.

Think I’m crazy? Let’s talk it out in the comments.

Did you like this post?

Then upvote it on Hacker News. Follow me on Twitter. Or subscribe to my newsletter.