Running Python scripts on a schedule in AWS

Answered in Quora:

Q: What is the easiest/fastest way to set up AWS to repeatedly run a small Python or PHP script (24×7)?

Probably the easiest way is to use Lambda. It has a built-in scheduler and you only pay for the time it is running. (Note that in PHP you have to create a lambda package using this tutorial or via node Serverless here.)

If you mean continuously, and not repeatedly, then the caveat for PHP at least, it is a bad idea to have it repeat forever in the process because it was designed to be set up and torn down and might leak memory (with the proper setting to set_time_limit and ignore_user_abort, have it run forever). In those cases you would have something else constantly call the php script via command line and restart.

Other than that, whether PHP or Python or whatever, it is better to create a non-burstable EC2 instance and run it, since it will always have some load. You can write something simple to the crontab that will make sure the process is always running or, better yet, use forever.

What’s something very few people know about PHP?

Answered in Quora:

Q: What’s something very few people know about PHP?

It is mind-bogglingly popular for web development. That popularity hasn’t diminished even though conventional wisdom says otherwise…

Over a decade ago, I said about 40% of the top 100 websites use PHP — a number I pulled out of my ass — but nobody (not even the Ruby on Rails developers I pissed off) argued with that spurious claim. In 2009, Matt Mullenweg, the creator of WordPress became curious with my claim and did a survey of Quantcast’s top 100 sites — he got almost exactly 40. Even today, among the 10 most important websites, four use PHP as their language of choice — 40% again.

Overall, almost 80% of the internet is powered by PHP, and that has held steady for years! Newer web languages such as Ruby or NodeJS have only grown at the expense of other languages such as ASP, Java, or Perl.

Just one single application written in PHP, WordPress, is used by over 30% of all websites on the entirety of the internet. That’s more than double the market share since back when I last worked at Automattic/WordPress in 2011! It grew until it saturated its entire market — over 60% of all CMSs. In the CMS market as a whole, PHP-based CMSs occupy positions 1, 2, 3, 6, 7, and 8 in the top 10. The most popular non-PHP-based CMS is both closed source and sitting at only a 2.5% share.

It was estimated back in 2009 that there were 5 million PHP developers worldwide. It’s difficult to make this estimate today, but it’s obvious that that number has also held steady or grown.

These last few years, I’ve been commercially working in Ruby (on Rails), GoLang, NodeJS (for static servers), and Python (Django), but PHP is still also my love in that love/hate relationship.

Come see my talk in February 2019 at SunshinePHP in Florida!

What don’t you like about PHP?

Answered in Quora:

Q: What don’t you like about PHP?

It’s a ball of nails with almost no logic to it. From the perspective of software language design — as Tim Bray, co-creator of XML and at the time Directory of Sun Microsystems’s Web Technologies division, once said — “PHP is soul destroying.”

To take one of the most notorious examples: strpos and in_array.

In one, the order of variables is $needle, $haystack in the other it is $haystack, $needle. I challenge even the best PHP developers to remember which it is without a reference. Also, even though they both serve the same purpose they return different things and don’t obey a consistent naming convention: they should be named strpos/arrayin, posinstr/inarray or `str_pos`/`array_in`. That’s three terrible design choices in one single example!

Another example is the namespace separator in PHP is what is an escape sequence everywhere else \\. Why? Because by the time PHP added namespaces, they had run out of ASCII symbols for the lexer.

I could go on and on, but I won’t.

I love the language sometimes because I hate it so much. 😉

Come see my talk in February 2019 at SunshinePHP in Florida!

Which has better packages, Python or PHP?

Answered in Quora

Q: Which has better packages, Python or PHP?

It depends on the target utility. In the Python world, the most common package installer is pip; the PHP world didn’t settle on a dominant format/installation for packages until composer, and that was relatively recently (last 4 years).

The reason for this is because PHP is a language almost exclusively designed for websites, and the economy of the open-source world has found that a dedicated application, instead of a framework or library could serve the bulk of most commercial needs. For instance, a single application, WordPress, four years ago (2014) accounted for nearly 20% of all websites and 50% of all blogs. There is little need package management when a single download and install (or a single click during ISP/hosting account creation or visit to site like Bitnami) can set up everything a typical website might need, fully under your control, with its own package infrastructure (plugin and theme ecosystem) where the underlying language (PHP) is pretty much irrelevant to the user.

Other languages like Ruby (with gems) or Go (import) or Javascript (with npm) which matured later and are more general purpose had a more robust package management and installation system. PHP Composer, in fact, is most modeled on npm.

Python, like Perl, predates these languages, but its general purpose demanded package management early in its development. However, because of this, pip takes less advantage of web conventions that npm does. Python has the added headache of version compatibility with its core runtime. Still pip is perfectly workable and relatively seamless and easy to use.

So which has better packages? The answer is it depends on the domain. In nearly any language you can find an adequate package for any of your needs, but overall you will find the packages are higher quality, more up-to-date, and sometimes just better overall in the domain the language seems to target well.

In my opinion, packages for data science/AI/ML applications, Python packages are second to none. As a PHP expert, I wouldn’t even bother looking in Composer for them and would instead find a PHP extension that binded to a dedicated engine optimized for the problem area or write a dedicated solution/web service in Python/flask and call it from PHP. If, for some crazy reason, I needed to do this entirely within PHP, I would search for the best solution in Python and see if someone has a port for that in PHP.

I would expect the same in DevOps for Python, but right now that space, like blogging (WordPress), seems to have dedicated solutions where a domain-specific language has abstracted one from the language-specific package management solution. For instance, in the configuration management space the top four applications are: Puppet, Chef, Ansible, and SaltStack. Of those four, I believe only one (Chef) exposes the underlying language (Ruby), and it does this to the DevOps engineer no more than WordPress exposes PHP to a blogger.

For server-side web-based packages, I feel PHP and Composer are going to have a solution for any problem PHP itself is capable of solving. That is a restricted set, however, as more and more work for web applications is being done on the client-side in Javascript and there is already an adequate toolchain for that (as well as a package management system in npm). Other web-based languages will probably have adequate solutions in this space with decreasing order/quality: Ruby (on Rails), Javascript (NodeJS), Python, Go, Perl, Java… The web problem is now pretty discovered at this point so there is less a need to adopt the bleeding edge in the web problem.

On being a beginner (again)

Compiled from three separate discussions on IRC, twitter, and in person:

“Whatcha do with your time these days? Learning Rails? 😈”

I did pick up Objective-C again after an aborted attempt at learning Swift. Mostly I’m trying to catch up on the javascript frameworks that have come out since I stopped coding. Right now it’s AngularJS—I figure I can jury-rig React into it if performance becomes an issue.

On the non-programming side, I’ve been messing with Ansible because I just got tired of doing things by hand—and I never needed to learn this because I’ve always had operations engineers working with me.

The ripping on Rails thing is over with me because there’s no point in arguing over how to solve a solved problem—today, the web problem is the easy part. What I find strange is people still feel the need to defend Ruby on Rails. I mean who the fuck cares what your middle layer code is written in when everything is an API to something written in Javascript?

“I don’t like that everything is an API to something in Javascript. As a user, the Web feels slower and flakier than it used to.”

I don’t like that everything on the front-end is pushed toward a single-page application. The reason for this is that the DOM-based model of front-end javascript (e.g. jQuery) gets so taxing when the application gets big because you’re bolting feature-on-feature, library-on-library to get it to work as smoothly as you envision. At a certain point, a true MV(VM) javascript framework (e.g. AngularJS) gives you much more because it abstracts all that in a consistent manner.

As soon as you buy into one of these, you’re invested into a huge initial javascript payload which causes you to not want the user to leave the page to unload anything, which then forces you into an API-based model with HTML partials and a client-side route/sitemap and more crap in the payload until you have a single-page application.

And then pretty soon your website is like Flickr where I swear every tenth click I’ve got to reload the page because the UI became non-responsive and I’m deciding to open the app in my iPhone just to do something without that frustration. How fucked up is that?

But then I look at Bootstrap and I figure, I’d rather have a SPA than everything looking like it was designed by some Apple-loving hipster (and this coming from a person who has used and loved Apple products longer than they’ve been alive).

“I’ve always enjoyed your talks and lamented that you didn’t remain on the PHP speaking circuit.”

Maybe I’ll start speaking when I have something to say. Like I’ve said before, PHP solves the “web problem” very well, but the web-problem is not a hard problem anymore.

Remember, it’s been four years since I’ve done any UI programming so everything is new to me. Basically, I’m a newbie, and I don’t think anyone wants to hear from someone who doesn’t know what they’re talking about.

But I did notice this from managing engineers: the worst problem a coder can get into is fear of having to start over. You get good at what you’re good at and when things pass you by, you feel the need to protect what you have because its what you know.

That’s how I feel about Ruby on Rails and that’s how I feel about me and PHP.

So, I’m a beginner again.

You lost me there, bub

Reading [this blog post ranting on PHP][phphorrors]

> No corporation supports PHP’s growth & maturity like Sun & Google do for Java, Google (Guido van Rossum) for Python (jnc Django framework), Ruby (inc RoR) by 37 signals etc…

37Signals & Ruby? Thank his noodly appendage PHP’s support isn’t as terrible as that company on that language.

You lost me there, bub.

When it comes to engineering choice, programming language is not even in the top 10 of important choices a software architect has to make. If you’re worried about the language, you’re worried about the wrong thing. (I’m also a little amused that the author holds Python as a language with great unicode support.)

[phphorrors]: “PHP: coding horrors & poor excuses—The Road to Silicon Valley”

It's just a tool

Previously: [Part 1][programming1], [Part 2][programming2], [Part 3][programming3], and [Part 4][programming4].

I received an interesting question the other day from a recruiter:

> Would you be open to moving away from PHP?

My answer:

First, by analogy. I’m a photographer, most of the cameras I own are manufactured by Nikon. This is like asking if I’m open to using a Canon.

I happen to be very adept with using a Nikon. However, I don’t think [the photons bouncing around care much what logo sits on front of the camera][camera purchasing] they are in. I’d be a poor photographer if what I did was care about the brand of camera I carry instead of the picture I am taking.

PHP is designed to solve one problem: the web problem. It solves it very well—as evidenced by marketshare and its continuing durability. At nearly all other stuff, PHP happens to suck at.

However it (the web problem) [isn’t that difficult] of a problem (having been solved repeatedly by languages such as Ruby, Python, C, C++, Java, Javascript, etc.). The real difficult parts are invariably outside the “web problem.” (Example: Twitter’s web problem was solved in Ruby using Rails, but their real problems were not web problems and of those problems, the most prominent was solved in Scala.)

I happen to be very adept at PHP. That pretty much means knowing that there is a better tool than to use PHP to solve the problem at hand at hand, and using that instead of PHP: MySQL, Apache, memcached, MongoDB, CouchDB, Scala, Erlang, Puppet, CruiseControl, AWS, etc.—none of which, to my knowledge anyway, written in PHP.

I’d be a poor programmer indeed if I thought that the problems I solved cared much the language I was adept in.

I’m not a poor programmer.

The Short answer: “Yes. Depends on the problem and the existing environment.”

(Recruiter was amused by my answer because he was owns a Canon. I suggested we “get on either side of the street and start making threatening dance moves at each other like [the Jets and the Sharks][west side story].”)

Another question I got:

> Which do you think is better [some programming language] or [another programming language]?

My answer:

They’re just tools. That’s like asking a handyman which they think is better: a hammer or a screwdriver.

(That elicited a laugh.)

Using a programming language is simply [a choice][making choices]. There are many things that go into making a correct choice that are far beyond what language one prefers for a problem that happens to be not that hard to solve in the first place.

[programming1]: “Learning Programming Part 1: 5 million”
[programming2]: “Learning Programming Part 2: Programming Frameworks”
[programming3]: “Learning Programming Part 3: C/C++ superiority”
[programming4]: “Learning Programming Part 4: “Programming is Hard””
[camera purchasing]: “Camera purchasing advice”
: “Is Ruby the dog and PHP the dogfood?”
[making choices]: “Simple prescriptions and making choices”
[west side story]: “West Side Story—Wikipedia”

Super Tuesday

When I first moved to San Francisco, the PHP meetup group hadn’t had a meeting in a year. That was before [Touge][touge] took it up, and, along with [Mariano][mariano], does the hard work of actually scheduling people to come shoot the shit.

Apparently, it’s time for my shit to be shot.

[Tomorrow, I’m giving a talk][talk] at [SFPHP][sfphp] on [DevOps][devops] for PHP developers. I’ve giving this talk before as the [closing keynote][lineman phpcomcon] at [PHP Community Conference][phpcomcon] and [to sysadmins][lineman oscon] at OSCON.

> **Living without Your Linemen: The Programmer Becomes System Operator in the Cloud**
> If a website architect is the quarterback, then site operations is the offensive line—overworked, under-appreciated, and only noticed when it fails. They make you look good. However, four years ago cloud computing networks like Amazon Web Services and Slicehost have appeared. While deficiencies in frameworks in other languages have forced those worlds to adopt Infrastructure-as-a-Service, the PHP world—with it’s ultra-cheap shared-hosting (on one end) and tradition of dominance on some of the most trafficked websites (on the other)—has been slow to move. But as the technology continues to disrupt, modern web engineers will be expected to use their programming skills to not only build, but also provision and maintain fast, scalable websites.
> The efficiencies of a web-based language and experience in scalable website architecture offer a unique opportunity for programmers to transfer their skills when wearing a sysop hat. Not to mention some of the best libraries for programming them are written in PHP! When going from a small pet project to a go-live site, maybe we can learn to live without our linemen.

Trust me, you’ll like it.

[Please come][talk]!

Also, If you are an American citizen, go vote! 🙂

[talk]: “Living without Your Linemen: The Programmer Becomes System Operator in the Cloud—SFPHP”
[sfphp]: “The SF PHP Meetup Group—SFPHP”
[touge]: “Grep My Mind”
[devops]: “DevOps—Wikipedia”
[lineman oscon]: “2011 07 Living without your Linemen—OSCON”
[lineman phpcomcon]: “Living Without Linemen—PHP Community Conference 2011”
[phpcomcon]: “PHP Community Cpnference”

PHP Community Conference Closing Keynote

I submitted a couple talks for [PHP Community Conference]( last month, of which one was accepted.

Unfortunately, it was the one I hadn’t prepared at all. The title was “living without your linemen” and was supposed to be about cloud services. A bit later, they asked if I could make it the closing keynote for the conference. This allowed me to write it from scratch and actually finish the talk (which I did about 30 minutes before I had to present it).

I’m told that they’ll eventually have a video archive of the talk at [](, but in the meantime, I audio-recorded it and synced it to the slides on slideshare. (Apologies for the sound quality being poor, I recorded it from my Mac Book Air).

Continue reading about PHP Community Conference after the jump.

My beef with Quora

Last year back when Quora was beta, someone pointed this Quora entry out. I explained why this guy was mistaken and let it lie. But, since a friend sent it to me recently, I guess people are actually using Quora (or something) and this deserves a response

The page being discussed is part of a larger article I wrote (and never finished) here. In the page linked, there are almost none of my opinions, but rather a summary of what was provided by Haiping during a briefing at Facebook. The outline of the page is as follows:

  • PHP has some inherent advantages as a programming language for web development.
  • PHP has some disadvantages (for Facebook). The biggest are:
    1. High CPU
    2. High Memory usage
    3. PHP components are not easy to integrate from outside
    4. Extensions writing is not the same as PHP coding
  • There were multiple attempts at Facebook to migrate from PHP but they failed: Mainly because an re-architecture team cannot keep up with the new code that is being written by the rest of Facebook—mostly writing new PHP code. The year before the presentation alone had 4 attempts at internal migrations
  • Improving the PHP core was done at Facebook and, in fact, received a lot of mileage, but this was not felt to be sustainable vs. HipHop solution.

Continue reading about my beef with Quora after the jump