Hack This: Scripting Deeper, Better Hacks in Python

Part of the point in presenting small, hack-minded tutorials a la "Edit an Image in Python" or "Send an Email from Python" is that some subset of readers who actually follow the steps will wind up interested enough in the subject(s) to go deeper on their own. I can present a detailed recipe, but learning to cook is somewhat left to the reader. And given that there are actually many readers with many varying levels of interest and existing ability, this is potentially some very tricky business.

So, let's take a moment to regroup. The past couple of "Hack This" columns have involved using some very powerful Python libraries in some very simple ways. For one thing, they're based on entering commands directly into the Python interpreter one or a few at a time. This is a pretty bare-bones, limiting, and ultimately misleading way to go about things, as the power of the Python language and any other language is in putting all of these pieces together into more powerful wholes.

These are what we'd usually just call programs, but Python is a bit strange in that it's a programming language that is also a scripting language. I talked about the distinction in some detail in this column on Unix shell scripting from last fall, but let's take a moment to recap.

When most of us think about a computer "program" we probably think about the things we install on our computers and then run by clicking on an executable file. The program whirrs to life and now has some amount of computer resources dedicated to its operation, at least until we close it. Programs like this are compiled. That is, they are written in some language using that language's syntax and are then fed into another program, which translates the code into machine-readable code. This is the thing you install and it's what gets invoked every time the program is run or executed.

Scripting is a bit different than programming. You know in those earlier tutorials how we were able to just tap line by line commands into the Python interpreter? Each one of those don't individually get compiled into machine instructions and then run as an executable program one by one. This is because that compilation has already occurred. The machine instructions for those commands are already generated and so it's more just a matter of matching commands to instructions on the fly. It's sort of like building with Lego blocks bought from a store rather than molten plastic. The pieces are pre-built and ready to be snapped together.

So, a Python script is really just an arrangement of the same commands we could tap into the interpreter but instead packaged together into one file. That package of commands, which together may wind up behaving in much the same way as a regular old computer program, can be invoked by running the Python file with the "python" command. All we have to do is tap in:

python

This is where hacking becomes programming.

This post is mostly about what a script is and can be, and how to start putting things together into more complex and much more useful wholes. It isn't an intro to Python or intro to programming thing because the universe sure doesn't need that right now. It should serve to give some grounding to the Python-based hacks/recipes you're likely to see here, e.g. how those things connect back into the larger world of code and become way powerful.

Before going on, you'll obviously need to have Python installed. It can be downloaded from here. You'll also need a text editor for writing and editing scripts. I use Sublime Text, which is great and not free. It's worth the money. Atom is its open-source analog. Any text editor will do though: Notepad++, Vim, Emacs, TextWrangler. Python also comes with its own editor called Idle, which I've never actually used and don't know anything about. Note that full-on word processors like Word will not work, however, as they tend to include all kinds of invisible characters that will mess things up.

To demonstrate the whole script vs. interpreter thing, let's do something really, really basic. Make a file in your editor and save it as hello.py. Add just a single line to the file:

print "hello, world"

And save it. Go to your system's terminal or command line window and type in this command:

python hello.py

You should see "hello, world," obviously.

All you did was tell your system to run Python on the given file. If you got an error about Python not being found or something similar, it means you either don't have Python installed or your system doesn't know where to find the Python program. (Python is a program that runs other programs.) This can get tricky, so just Google whatever error you have and you'll get an answer pretty fast.

If you get an error about the script file not being found or somesuch, note that you need to be running Python from the same location as wherever the script is saved.

OK, now just type "python" onto your system's command line. This will launch the Python interpreter and you can tap in the "print …" command above just to verify that it does give you the same output whether entered at the Python interpreter or when invoked via a Python script.

In Python, "print" is a reserved word interpreted as a command telling the interpreter to output to the screen (unless otherwise specified) the following string, which can either be some text stuck between a pair of quotation marks, or it can be a variable containing a string. You should take a few minutes to the look at the Learn Python the Hard Way section on variables and names, and also the section on strings and text.

Let's try something a little different. Open up your hello.py script and change it to the following:

print "Hello What?", some_input = raw_input() print "Hello, %s" % some_input

Save the script and run it again as before. It should print "Hello What?" and wait for you to type something. Tap in a word or two and press enter. The script will print back "Hello " followed by whatever text you gave it. The second print statement is a bit funky-looking, but the idea is that within the quotes, the placeholder "%s" is telling the command to look at whatever is entered in following the quotes and another "%." You could have more than one placeholder "%s" in there, but then you would have to do this (again: you should look at this section at LPHW):

print "Hello, %s, Goodbye, %s" % (some_input, other_input)

Now that we know the bare minimum about Python scripting, let's do something useful, relatively speaking. One of the first project ideas I had circa CS 161 Intro to Computer Science at Portland Community College was to make a webpage that displayed all of the weather forecasts for a user's area from all of the different sources, e.g. the Weather Channel, Wunderground, NOAA, etc. on the same page—sort of a Kayak of weather forecasts—along with a running estimate of each source's accuracy.

This is pretty easy to do because all of these sources have their own APIs, e.g. standardized sets of protocols that we as programmers can use to interact with the sites' raw content via our programs

So, let's do that. Not the whole thing, I mean—let's just make a script that will give us the weather forecast for whatever ZIP code we feed it. The code here is partially based on a tutorial at Stray Marcs.

Two things to note. First, you'll notice we're importing two modules into our script. Modules are basically extensions to the Python library that you can install and then import into your code. Read more about that here. Second? What the hell is JSON? Good question. JSON is a data format used all over the internet. A JSON object is like a bit list of values and key pairs, where we can extract data by referencing the corresponding key. In the example, we use one function to open up a URL from Wunderground, and then another to give us its contents in the form of a JSON object.

import urllib import json

Once we have said object, we can get stuff from it. What all is available can be found in the API's documentation, but we'll just get a couple of things to keep it simple.

First, let's have our script ask the user for their zip code. We'll use the raw_input function again, but a bit differently. It turns out that we can make the same function print the prompt itself rather than having to do it in a separate "print" statement.

zip = raw_input("where are you?")

Now, we have the script make a URL and then ask that URL for our JSON object. Note here that we'll need a key from the Wunderground API to make this happen. It takes like a second to sign up for it here. API keys allow places to control how much traffic is accessing their resources from various API users, for one thing. The basic free Wunderground account only allows a certain smallish number of API calls, but if you were building something big and relatively popular, you'd have to pay for more.

We construct the URL like this.

url = 'http://api.wunderground.com/api/' + key + '/geolookup/conditions/q/PA/' + zip + '.json'

Which is pretty easy.

So we open the constructed URL (which uses values stored in variables named "zip" and "key") and then read the contents as a JSON object, which is then parsed into something usable.

f = urllib2.urlopen(url) json_string = f.read() parsed_json = json.loads(json_string)

All we have to do know is tell it what we want, which we do by referencing the various fields that the API makes available to us via this JSON data-wad. You can view the data-wad in its native form just by pasting the URL constructed above into a browser. This might help make sense of the idea and how we're getting at the various pieces of information stored within it.

So, finally, let's get some stuff from this data-wad. I'm getting both a general one-word description of the weather along with the temperature in degrees Fahrenheit. I then print those two things.

weather = parsed_json['current_observation']['weather'] temp = parsed_json['current_observation']['temp_f'] print 'How about this ' + weather.lower() + '?' print str(temp) + ' degrees??? That is bullshit.

In the code above, I'm able to take different strings and paste them together just using the plus sign, which is handy and much easier than it would be in another language. The catch is that the data in the temp_f field from the JSON data-wad is an integer number and not a string. Nothing says this anywhere because we didn't create the "temp" variable with a type. It just took on whatever data type the stuff on the right side of the equals sign happened to have, which happened to be a number.

In any case, we use str() to convert temp to a string type, which allows us to stitch it to other strings. Here's the whole script:

import urllib2 import json key = 'yourkeygoeshere' zip = raw_input('For which ZIP code would you like to see the weather? ') url = 'http://api.wunderground.com/api/' + key + '/geolookup/conditions/q/PA/' + zip + '.json' f = urllib2.urlopen(url) json_string = f.read() parsed_json = json.loads(json_string) weather = parsed_json['current_observation']['weather'] temp = parsed_json['current_observation']['temp_f'] print 'How about this ' + weather.lower() + '?' print str(temp) + ' degrees??? That is bullshit.' f.close()

And here is some output. Indeed, it is bullshit.

Getting some weather information from an API isn't very interesting, but we'll get there. The point here is just to demonstrate how simple Python commands can be used together in a script. On the other side of this threshold lies everything from hacked together Twitter bots to entire software frameworks. We'll start with the Twitter bots.