NaNoWriMo 2016

Ever since Marie wanted to learn to program in 2009, I’ve wanted to write a book to help her. But I never could get started.

The National Novel Writing Month is November every year for just this purpose: to motivate people to put 50,000 words on paper (about the size of the novel, Slaughterhouse-Five), editor be damned.

I first heard about it in 2007, when I started using Scrivener, but dismissed it because the requirement that a novel be fiction. I only just found about NaNo Rebels, which allows you to customize the “50,000 words” into nearly any other creative exercise, including non-fiction. So yesterday, this was born:

NaNoWriMo 2016 Participant Badge

Nowadays, I use Ulysses. I simply created a group in the software and set a 50,000 word goal and started typing away!

NaNoWriMo 2016 Day 2

Goal setting on my iPad.

I don’t know if I can finish since it’s about a good sized blog article each and every day. We’ll see how it goes. So far it’s been a bit strange writing a book. For instance, I can’t use my WordPress shortcode macros lest I ruin the word count.

Periodically, I’ll dump the output to my blog, which you can track here. Wish me luck!

If you want to buddy up, I’m “tychay” there.

I come to bury PHP, not to praise it?

(I received an answer request from someone on Quora for this question. At the time of my writing, there were only two answers, both that boiled down to “use Python.” While I feel I was asked to balance out that with some PHP cheerleading, I can’t bring myself to do that. Nor, can I hop on the bandwagon.)

Between PHP and Python which is better for a startup?

My answer reads like a Zen koan: it depends on the nature of the problem; it doesn’t really matter. It becomes the nature for future problems; that makes it matter.

Continue reading an to see that the koan does actually make sense after the jump

Why do Python programmers hate PHP?

Why do Python programmers hate PHP?

I saw a colleague answer this question and thought I’d give a go:

Are sure it’s just Python programmers? I think that relatively speaking Ruby developers and Java developers hate on PHP more (and Perl and Javascript developers less).

From a software language design perspective, the PHP language is an ugly hack. One of the creators of Java said that when he thinks of PHP his heart goes dark (showing a complete black slide in his presentation for emphasis). The more structured the language (and Java and Python certainly are), the more offensive this ugliness is. (Ruby gets this view not from the language but for being the home of a lot of programmers who were trained in Java in school and for Ruby on Rails being a popular framework in the webspace already crowded out by PHP.)

However, there are t-shirts out there of me saying back in 2006, “PHP is a ball of nails, but when you throw it at a wall, it sticks” and that still applies today—it’s ugly, true; but it works. If anything, the language is far more ubiquitous on the web now than then, though that’s mostly due to the popularity of applications written in PHP such as WordPress and Facebook rather than any success in the language itself. The language has proved remarkably successful due to its adaptability and limited focus on what problems it tries to solve.

I imagine that this popularity-in-spite-of-aesthetic-beauty is offensive to those working in the web who think themselves above the direct, simple, and crass. That reality may sit so ill with some people that they feel the need to disparage PHP which becomes an object of their derision and the villain in their story. It’s only natural for a cohort in the minority to feel that way toward a market leader: Apple Macintosh users used to act like that toward Windows PCs, Windows Phone users act this way toward iOS and Android, etc.

The irony is that PHP is a market leader in only one tiny aspect of the programming market: the server-side, non-event-driven web. It has no footprint in the client-side web, none in application space or even the mobile app space, no recognized products in non-web tools and applications; and is just a rounding error in the DevOps space (to pick just one area where Python seems dominant/ascendant). It even makes no pretension to be a programming language to facilitate learning programming (another area Python is better suited to). Since all these areas are growing as the niche PHP represents is static, I find it ironic people still wage language-religious jihad against PHP.

That’s like hating on a horseshoe monopoly after the automobile has been invented.

And I say this as someone who has programmed PHP for 16 years and gives keynote speeches in the language.

The Apostles Query

Saw this on my feed the other day:

Beautiful queries

“Just wrote the most complex SQL statement I have ever written. It won’t scale, but it’s so beautiful. :)”

Cal’s query brought tears to my eyes. When a finch landed on it, I saw it pivot. It was deeply religious…


I believe in Codd, the Father Almighty,

and in MySQL, His PRIMARY KEY, our INDEX:
Which was established by the DB driver,
born of the Open Source;
suffered under Larry Ellison,
was TRUNCATE’d, DELETE’d and was DROP’d.
It RAISE()ed into non-TEMP, RIGHT JOIN’d with Codd the Father Almighty;
from thence It shall come to SELECT FROM the relational and non-relational.

I believe in the DB driver,
the Fourth Normal Form,
the relations of tables,
the ROLLBACK of failed transactions,
the RESTORE from logical backups,
and the persistence of storage.


Next up: Ave MariaDB.

On being a beginner (again)

Compiled from three separate discussions on IRC, twitter, and in person:

“Whatcha do with your time these days? Learning Rails? ?”

I did pick up Objective-C again after an aborted attempt at learning Swift. Mostly I’m trying to catch up on the javascript frameworks that have come out since I stopped coding. Right now it’s AngularJS—I figure I can jury-rig React into it if performance becomes an issue.

On the non-programming side, I’ve been messing with Ansible because I just got tired of doing things by hand—and I never needed to learn this because I’ve always had operations engineers working with me.

The ripping on Rails thing is over with me because there’s no point in arguing over how to solve a solved problem—today, the web problem is the easy part. What I find strange is people still feel the need to defend Ruby on Rails. I mean who the fuck cares what your middle layer code is written in when everything is an API to something written in Javascript?

“I don’t like that everything is an API to something in Javascript. As a user, the Web feels slower and flakier than it used to.”

I don’t like that everything on the front-end is pushed toward a single-page application. The reason for this is that the DOM-based model of front-end javascript (e.g. jQuery) gets so taxing when the application gets big because you’re bolting feature-on-feature, library-on-library to get it to work as smoothly as you envision. At a certain point, a true MV(VM) javascript framework (e.g. AngularJS) gives you much more because it abstracts all that in a consistent manner.

As soon as you buy into one of these, you’re invested into a huge initial javascript payload which causes you to not want the user to leave the page to unload anything, which then forces you into an API-based model with HTML partials and a client-side route/sitemap and more crap in the payload until you have a single-page application.

And then pretty soon your website is like Flickr where I swear every tenth click I’ve got to reload the page because the UI became non-responsive and I’m deciding to open the app in my iPhone just to do something without that frustration. How fucked up is that?

But then I look at Bootstrap and I figure, I’d rather have a SPA than everything looking like it was designed by some Apple-loving hipster (and this coming from a person who has used and loved Apple products longer than they’ve been alive).

“I’ve always enjoyed your talks and lamented that you didn’t remain on the PHP speaking circuit.”

Maybe I’ll start speaking when I have something to say. Like I’ve said before, PHP solves the “web problem” very well, but the web-problem is not a hard problem anymore.

Remember, it’s been four years since I’ve done any UI programming so everything is new to me. Basically, I’m a newbie, and I don’t think anyone wants to hear from someone who doesn’t know what they’re talking about.

But I did notice this from managing engineers: the worst problem a coder can get into is fear of having to start over. You get good at what you’re good at and when things pass you by, you feel the need to protect what you have because its what you know.

That’s how I feel about Ruby on Rails and that’s how I feel about me and PHP.

So, I’m a beginner again.

Why “every” developer should be using ansible

AnsibleFest starts tomorrow so I thought I’d make a case why you should be using ansible.

First, in order to prevent a ton of hate-comments which usually follow such pronouncements, I should define who you are.

  • You are a developer focused on the web or some web-based tool.
  • You are not already very proficient in operations (e.g. in DevOps).
  • You are responsible in some degree for a system that is not your own (e.g. live server upkeep, deploys, other development or testing machines)
  • You are not in an organization large enough to already have operations/system administration as a distinct and separate organizational unit.
  • You are working on a project or startup that has the potential to grow beyond its current state.
  • You are not currently coding the project/startup mostly in Ruby.
If the above are **all** true for you, then I recommend you should learn and use ansible.

What is ansible?

According to Wikipedia, Ansible is a free software platform for configuring and managing computers.

As a developer, this statement means nothing to me. Here is how I saw all development:

  1. Develop.
  2. ???
  3. Profit!

But when I stop to think about the process in the real world, it went something like this:

  1. Join some startup.
  2. Make friends with someone in operations and have them hook me up with a development environment.
  3. Develop (step 1 before).
  4. Commit code.
  5. Convince someone to deploy my code or make a command that I can type to deploy my code.
  6. Go to step 3.

If step 2 or 5 ever were blockers, someone got bitched out until 2 or 5 ceased to be blockers.

But in the last nine years, startups no longer can necessarily afford resources to have the people responsible those steps be someone distinct from you, the developer. In fact, being a sysadmin is often a hat you as a developer who “knows that computer stuff” has to wear at an organization.

So putting on your “operations” hat, here is the process of web development from scratch:

  1. Get a working development machine.
  2. Do a system update on that machine.
  3. Get the software you need running on your computer.
  4. Get the software configured so that you have a working directory where you’ll start your project.
  5. Link that working directory with your development environment
  6. Commit the working directory to a code revision system.
  7. Start developing.
  8. Commit Code
  9. Repeat 1 on a live machine
  10. Repeat 2 on a live machine
  11. Repeat 3 on a live machine
  12. Repeat 4 on a live machine
  13. Purchase domain names.
  14. Point domain names to live machine.
  15. Deploy committed code through some process.
  16. Repeat step 6, 7, and 15 as needed.
  17. When on boarding someone repeat steps 1-5 as needed for them.
  18. Make any changes on both systems as needed.

So what does ansible do when it says “for configuring and managing computers?” It is software that you code to automate steps 2, 3, 4, 9, 10, 11, 12, 14, and 15 for you as well as help you do step 18. This covers everything that you don’t already know how to do except step 1 and 5.

Other choices

Of course, Ansible is nothing special. There are a lot of other choices out there for configuring and managing computers. As a class those choices are known as configuration management software. What I’ve hoped you learned is that whatever does configuration management can also do machine creation in cloud environments, deploys, and simple ad hoc changes.

Ansible is just one such choice.

So what are your other choices?

In order of decreasing popularity they are:

  1. Yourself, by hand, writing shell scripts as needed for automation.
  2. Puppet
  3. Chef
  4. (Ansible)
  5. Salt
  6. …others…

Why not yourself, by hand?

This is the default choice. It’s what most people are doing right now, every day, and it’s important to remember this because the argument against hand-crafting your system administration is not an argument for ansible, but rather an argument for any configuration manager at all.

The biggest advantage of doing things by hand is that any other option has a sunk cost. Anything you would do in any configuration management system requires you have done this at some level first. This is a huge argument for the default and should not be discounted.

Learning and using a configuration manager may justify its sunk cost at some point. That point depends on a number of outcomes unrelated to the choice of configuration managers.

If the project you are working on becomes successful then you will be hiring more developers (who need virtual or cloud machines, or development environment setup), deploying on more machines, or adding more steps to your process such as testing, or blue-green deploys.

For each of those machines the work of configuration management is repeated, after a set number of repetitions, you would probably start automating things through ad hoc shell scripts, which you’ve also already done for deployments. At some point that work becomes much greater than the savings generated by not bothering to learn a configuration manager.

Often before that point, your organization may become big enough to warrant a person (later a team) to specialize in operations, system administration, or DevOps. In addition, your organization has already been paying extra money in the form of some backup service because of snowflake servers. So while you personally wouldn’t have to directly pay the debt accrued by avoiding using a configuration management system, your organization will have in both backup services and work for your ops team to port an undocumented live infrastructure. A configuration management system avoids this cost because it acts as both a backup-restore system and a complete and consistent set of documentation of your machines’ states.

This is true even if the operations team chooses a configuration management system that’s different from the one you chose, though obviously if they’re the same, that’s even better.

Even if the project fails it is still possible to have positive returns to using a configuration management system.

  • A configuration management is not tied to your cloud provider. So you are free to switch cloud providers from say Amazon Web Services to DigitalOcean with minimal hassle.
  • Developer environment setup is manageable using the same system. At some point you may have gotten a return here even if you are the sole developer, if you are rebuilding your environment from scratch enough times. These scripts can be made identical to your production provision and deploy scripts.
  • Deployment and testing can be automated through configuration management. So there is some return from avoiding writing custom scripts for this process.
  • This is a skill needed in any internet-based startup. You have a skill-transfer directly to your next (hopefully successful this time) project.
  • You may find configuration management interesting and find a career as an expert in DevOps as a specialty instead of as a general developer.

Why not something other than Ansible?

I’ve built out a simple LAMP stack in Puppet Apply, Chef Solo, and Ansible. Overall, I don’t find that large a difference between them for the things I’d need to do at this scale.

When deciding which configuration management system to chose as a developer, I weighed two criteria:

  1. I wanted the learning curve (necessary to be proficient enough to replace direct command-line operations on a machine) to be as small as possible.
  2. I wanted to mitigate the amount of lock-in created on a future operations team by this choice.

Puppet, being the oldest and most popular of the four, has the advantage of being the most likely to already be known by an experienced future DevOps hire in most scenarios.

However Puppet’s execution model is derived via dependencies. While I understand the reasoning, this adds a lot to the learning curve because manifests and modules are not like typing in the command line. Those who say that dependency-based management is easier to learn and use than sequential-ordering are the same idiots that a decade and a half ago told me SAX parsers were more intuitive than DOM ones: go fuck yourself. Furthermore, other CM systems have a sequential ordering so migrating away from Puppet to something like Chef is more costly than it would be from Ansible to Chef.

Finally, I find Puppet manifests and modules very intimidating to someone jumping in media res, which simply isn’t true of other configuration managers. This is a big deal because I, as a developer, I plan on pawning of site operations responsibility onto someone as soon as I can con them into thinking machine maintenance is fun.

I remember watching a core developer of Puppet scratch his head to try to figure why anyone would choose Chef over Puppet. Once, After someone shat on me for liking Ansible, I stared incredulously as they recommended Chef—as every argument just used they for Chef over Ansible could be used for using Puppet over Chef. Having said that, Chef does have two advantages over Puppet. The first is that Chef recipes and cookbooks has ordered execution like you’d expect, and the second is that those are written in Ruby. However, Ansible already does the first, and I only care to know enough Ruby to edit a Vagrantfile. Gaining proficiency in another programming language just to execute commands on a machine when I already know shell scripting is a waste. Also, I’d think having a programming language at the ready instead of a markup language is a recipe (pardon the pun) for encouraging a developer to rely on it instead of learning CM best practices. In the process an adept programmer/beginner DevOps is likely to break important features of good configuration management like idempotence.

However, if you are a Ruby developer, I’d recommend Chef because you already know the markup language and because it’s highly likely the DevOps you hire (or the Ruby shop you work at next) will be using Chef as their configuration management system. Sometimes life in the bubble is good, why burst it?

With Ansible you will have to learn YAML to write tasks and playbooks and Jinja2 is the templating system. Both are trivial. Tasks will resemble, almost exactly, a command you would type on the command line so the learning curve is quick. Once in a while, a command you type might be actually two separate tasks. Oh, the humanity! The sky is falling! It connects to the machines without a client just as you would if you were the ssh into a box. In fact, the biggest difficulties I’ve found were not with finding which ansible module task emulated the command line I wanted, but in navigating the idiosyncrasies of SSH.

Salt has a core focus on scalability and performance at the cost of an agentless model, which I’ve heard rumor does actually exist in Salt. Unfortunately, nearly every tutorial steered me away from this toward the server-client one because the magic pixie dust that powers Salt—it goes by the name ZeroMQ. This is a non-starter in terms of learning curve as a developer who suffers through, rather than enjoys, DevOps. Other than that, I think they’ve made better architectural choices at every turn than all the other CM systems, including Ansible.

The biggest B.S. argument against Ansible.

I remember over dinner one day, an ops engineer scoffed when I mentioned I was using ansible for my personal projects. When I asked him what I should use instead, he said, “puppet.” When I asked him why, he said, “Ansible doesn’t scale.” For about half of my career, Puppet didn’t exist, so I told him, “ I guess no website built before 2006 ever scaled.”

Cal Henderson is fond of saying, all programming languages scale, it’s architectures that do not. In mathematical terms, this is because all programming languages are Turing complete which means that they’re mathematically homomorphic (or equivalent) to each other.

In the same way, all configuration management systems are interchangeable—at the end of the day, they’re executing just commands on the target machine. Worst case scenario, you could make an ansible playbook whose only task is to make sure the puppet agent is up to pull from its master. That’ll scale just as well as any puppet client configuration.

If someone tells you that some sort of configuration management system doesn’t scale, what they’re really saying is, “I couldn’t get that configuration management system to scale.” Which really is more an indictment on their competence, rather than the software they’re disparaging.

How do I get started

Do a web search for “ansible”, “your favorite language”, “your favorite server”, and “your favorite flavor of operating system.”

As for me, I started with Phansible and before that, this tutorial on setting up vagrant to run an ansible playbook.

One last argument

So there you have it, my argument to encourage you to learn and use Ansible if you are an internet developer who does not program in Ruby or have a preferred CM system already but has to “wear an sysadmin hat” as part of their job which involves some sort of project that might reach scale.

Yes, that’s a pretty tight set of requirements. Change any one of those and there’s just not enough of a difference between CM systems to justify one as being definitive.

But if you are still going to shit on me for recommending ansible, because insert your favorite CM system is so much better, let’s take a moment to consider that the developer I’m addressing is probably not going to use your CM system, but rather none at all. And that person might be the person who hires you someday, to that future you I say, “Would you rather port some ansible playbooks to your clearly superior configuration manager or a bunch of random shells scripts and snowflake servers?”

You can thank me for making the future you’s life significantly less miserable. 🙂

Getting Ansible to work with DigitalOcean API v2

Today I finally decided to solve something I’ve been putting off for a while. I recently migrated to reasonably-priced cloud hosting solution DigitalOcean. Version 1 of their API will stop working on November 9. The tool I use to automate my servers and developer installs is Ansible currently uses version 1 will not support version 2 of the DigitalOcean API until Version 2.0, which was supposed to have been released by now. My guess is that since it is almost November, the ansible team decided to wait until AnsibleFest to release 2.0. Unfortunately, that’s on November 19th.

So there’s a 10 day window + developer time where any playbooks I’m using will no longer work on the live site. Not cool.

So I decided to start up a project to test a working fix for this.

The first thing I needed to do was install Ansible 2.0 from source using the instructions on the website.

$ git clone git:// --recursive
$ cd ansible
$ source ./hacking/env-setup
$ sudo pip install paramiko PyYAML Jinja2 httplib2 six
$ make
$ sudo make install

This installed Ansible into my /usr/local/bin which I verified was correct by typing ansible --version

Next I installed the python DigitalOcean API, dopy:

$ sudo pip install dopy
view raw 2_install_dopy hosted with ❤ by GitHub

Log in to DigitalOcean and get a v2 token (there is no client_id in v2) and add the below to your .profile replacing the api_token with the one you generated.

$ export DO_API_VERSION='2'
$ export DO_API_TOKEN='api_token'

Generate a RSA key pair and upload it to DigitalOcean if you haven’t already done so and copy the following provision script into provision.yml (example modified from Jeff Geerling’s Ansible for DevOps chapter 7.) remembering to modify the ssh key signature.

Then run the script with the command:

$ ansible-playbook provision.yml
view raw 5_run_playbook hosted with ❤ by GitHub

This will create an ubuntu droplet and log in and run a install on it all automated like. You’ll notice that unlike Jeff’s script, you can refer to regions, images, sizes, and keys by name instead of looking up esoteric numbers for them.

Good luck and have fun!

Rituals and Religion

From an e-mail I received a year ago:

What does that actually mean? It means facilitating the “rituals” that
are part of an agile team’s work (e.g. the daily stand-ups, the
sprint planning meetings, retrospectives, etc.) and continually facilitating
the team’s discovery of improving the way they work.

What is the difference between a software process and a religion? Nothing.

I’m cool with software process, just like I believe in God.

I hate named software process because, like organized religion, it’s full of theology removed from reality, practice without the empiricism, theory without the application. When you show them empirical evidence on the consequence (or outright failure) of one of their particular rituals, they’re quick to maneuver with the words, “That’s ‘big A Agile.’ I’m not talking about that, I’m ‘little A Agile.’” (Whatever the fuck that is.)1

You can’t pin them down because they actually stand for nothing—there is no “there” there. It is the natural result of adapting a process that originated to allow sub-1000 page software consulting contracts with Fortune 500 multinational industrials in the 80’s and 90’s and blindly applying it to a shoe-string funded startups over a full decade after the dotCom crash in non-enterprise consumer-facing Internet whose entire business is software. Two different worlds; two different failure costs; one would assume that there would be two different names for two different software processes.

Instead there are hundreds of different processes all under the “little A agile” banner. And they look not alike at all. To watch the rhetorical hoops these agile adherents go through to call it “all agile” would be amusing if it wasn’t so unnecessary.

When I was a kid at evangelical summer camp, there was a parable I heard the counselor’s tell:

A man gets the opportunity to visit Heaven and Hell. He visits Hell first and meets Satan and asks, “Do you have any Catholics here?”

Satan responds, “Oh yes, we have a lot of them.”




“Them too!”

And so on, listing every denomination and finding them well-represented in Hell.

Depressed he goes to Heaven and chats with Saint Peter. “Do you have Catholics in Heaven?”

“No,” Peter says.


“No, none of those.”



And so on. Exasperated, the man asks Peter, “Well then what do you have in Heaven?”

“Christians.” Peter responds.

Catholicism is Scrum; Presbyterian is Extreme Programming; Baptists is Kanban. I suppose Hell is the dead-pool, Heaven is getting a getting funded or IPO2, and Martin Fowler is Saint Peter.3 It reads the same.

“Little A Agile”: the “non-denominational Christians” of the software process religions. If it works, it’s “Agile.” And if it fails to get you to Internet Heaven?4

Oh, that shit is “big A Agile.”

  1. If your process is defined by the outcome alone, then it is useless in a business setting. 
  2. A successful exit, in life or startups. :-) 
  3. Does that make me Satan? 
  4. The goal is to have an lucrative exit so you can blog about how hard it is to be a Founder and tweet about your First World problems. 

Pork Tocino

If you live in The Richmond, you know that B*Star Bar is like eating at Burma Superstar but without the line.

Breakfast plate with pork tocino, garlic fried rice topped with two eggs cooked over easy. Spoon in foreground
Pork Tocino with Garlic Fried Rice
B*Star Bar, The Richmond, San Francisco, California, United States

Sony DSC-RX1
0.013 sec (1/80) @ f/4.5, iso400, 35 mm

My favorite brunch dish there is the Pork Tocino. Grilled jerk pork over a bed of garlic fried rice and cherry tomatoes, topped with scallions and balsamic vinegar.

Since Marie loves their Huevos Racheros, I end up ordering this dish a lot. The only times I don’t is when we bring a guest, then I suggest they get it and I order something else.

Continue reading about some WordPress plugin notes after the jump