Today I was working away from my home file server which runs the BitTorrent server Deluge.  My Deluge Web UI stopped working for some reason (and it never worked that well in the first place) and I really needed to start a torrent at home.  Deluge has an “Auto Managed” feature that’s enabled by default, but since I seed a lot of torrents, it actually prevents a new torrent from starting to download automatically, so I have to disable that and the resume the torrent after adding it.  Thankfully Deluge has a Python client interface we can use to write our own scripts.

Here’s an example script that adds a URL from an argument with auto_managed turned off:

#!/usr/bin/env python
# encoding: utf-8
"""
DelugeEasyAdd

Starts a new Deluge torrent from a URL.

Created by Ted Hayes.

"""

import urllib, os, sys, re
# Import the client module
from deluge.ui.client import client
# Import the reactor module from Twisted - this is for our mainloop
from twisted.internet import reactor

# Set up the logger to print out errors
from deluge.log import setupLogger
setupLogger()

# Connect to a daemon running on the localhost
# We get a Deferred object from this method and we use this to know if and when
# the connection succeeded or failed.
d = client.connect()

PRIO_NORMAL = 1
PRIO_MAX = 5

torrentURL = sys.argv[1]
# btjunkie URLs use 'download.torrent' as the filename, get a better one
if re.search("btjunkie", torrentURL):
    filename = re.findall("torrent\/(.*?)\/", torrentURL)[0]
else:
    filename = os.path.basename(torrentURL)

print "Using filename: %s" % filename

def addTorrent():
    client.core.add_torrent_url(torrentURL, '/home/daleth/Downloads/'+filename, {'auto_managed':False}).addCallback(added)

def added(s):
    print "Added status: ",s
    #client.core.resume_torrent()
    disconnect()

# We create a callback function to be called upon a successful connection
def on_connect_success(result):
    print "Connection was successful!"
    addTorrent()

def disconnect():
    print "disconnecting"
    client.disconnect()
    reactor.stop()

# We add the callback to the Deferred object we got from connect()
d.addCallback(on_connect_success)

# We create another callback function to be called when an error is encountered
def on_connect_fail(result):
    print "Connection failed!"
    print "result:", result

# We add the callback (in this case it's an errback, for error)
d.addErrback(on_connect_fail)

# Run the twisted main loop to make everything go
reactor.run()

I still have to figure out how to then automatically resume the torrent.

Here are some helpful links:

UIClient Documentation

UIClient Example 

Today I was listening to a great Soundcloud artist with 22 tracks, all available to freely download.  Since I didn’t see an option to download them all en masse, I wrote up a nifty script to do it, then added a bunch of features to make it shiny.

Download the SoundcloudScraper zip file here:  soundcloudScraper_v1  To use, you’ll need Python; I looked into making it a standalone binary but I’m afraid that’ll have to wait for another time.  Open up a terminal and go to the directory the script is in, and type:

./soundcloudScraper.py artistname

This will automatically download all the available tracks by that artist, meaning tracks with a ‘Download’ button.  You can also specify multiple artists:

./soundcloudScraper.py artist1 artist2 artist3

This script uses the fantastic Animated Terminal Progress Bar module by Nadia Alramli!  If you find any bugs or get some use out of the script, leave a comment! Enjoy!

Tired of seeing this?

<type 'exceptions.UnicodeDecodeError'> at /such/and/such
'ascii' codec can't decode byte 0xf8 in position 12: ordinal not in range(128)

So am I.  I had a bytestring with extended characters in it:

data = '\xfcnic\xf8de test'

And no matter what I tried:

data.decode('utf-8')

…I kept getting errors. Turns out those extended characters simply aren’t in the utf-8 codec, so I tried latin-1:

data.decode('latin-1')

And it works! At long last!!

I’m going to be using OpenID as a universal login for a new webapp I’m working on. As luck would have it Web.py has a module just for this, and the API is super simple!  Now that I’ve got this working I’ve got to figure out how to actually integrate it into the app; this link looks helpful towards that.

1. Install web.py:

   sudo easy_install web.py

   or

   sudo apt-get install python-webpy

2. sudo easy_install python-openid

   or

   sudo apt-get install python-openid

3. main.py:

   import web, web.webopenid
   
   urls = (
       r'/openid', 'web.webopenid.host',
       r'/', 'Index'
   )
   
   app = web.application(urls, globals())
   
   class Index:
       def GET(self):
           body = '''
           <html><head><title>Web.py OpenID Test</title></head>
           <body>
               %s
           </body>
           </html>
           ''' % (web.webopenid.form('/openid'))
   
           return body
   
   if __name__ == "__main__": app.run()

4. Run

   python main.py

5. Go to http://localhost:8080
6. Voilà!

I’ve also added this to the Web.py cookbook.  Please let me know if you find any bugs or mistakes!

Usually if you want to kill some processes they have to have unique names or you have to know their process IDs.  Lately I’ve had a group of python daemon scripts that need to be restarted occasionally, so instead of looking at a list, finding the pids and killing them manually, here’s an  approach that uses PID files to keep note of the process’ ID, so you can just reference the file.  There’s a Bash script and a snippet that should be at the beginning of the python scripts.  Right now this is in conjunction with monit, which will start the scripts if they’re not running.  Monit can restart them for you, too, but apparently I was feeling masochistic this day.  This is adapted from original code by Brian House (thanks Brian!)

restart.sh:

#!/bin/bash

PROCESS_PATH="run/"
PROCESS_LIST=( "render_city.pid" "render_cards.pid" "email_sender.pid" "sms_sender.pid" )

function killProcess {
    echo "killProcess $1"
    PIDFILE=$PROCESS_PATH$1
    PID=0
    if [ -e $PIDFILE ]; then
        PID=`cat $PIDFILE`
        if [ "x" == "x$PID" ]; then
            PID=0
        fi
    fi
    if [ "$PID" != "0" ]; then
        echo "--> Killing current process $PID..."
        sudo kill $PID
    else
        echo '--> Nothing to kill.'
    fi
}

if [ ! -z "$1" ]; then
    if [ $1 == "all" ]; then
        echo "*** Killing all"
        chmod +x daemons/*  # since svn strips permissions, always make sure they're executable

        ELEMENTS=${#PROCESS_LIST[@]}
        for (( i=0; i<$ELEMENTS; i++)); do
            killProcess ${PROCESS_LIST[${i}]}
        done
    fi
fi

Here’s a python function that manages the PID file (written by Brian, goes in a util.py module):

def confirm_pid(run_folder):
    import sys, os, signal, __main__
    name = prefix('.', os.path.basename(__main__.__file__))
    log.info("Attempting to launch daemon %s..." % name)
    pid = str(os.getpid())
    pidfile = "%s%s.pid" % (run_folder, name)
    if os.path.isfile(pidfile):
        old_pid = open(pidfile).read()
        log.warning("--> pidfile already exists for %s, attempting to kill process..." % old_pid)
        try:
            result = os.kill(int(old_pid), signal.SIGKILL)
        except OSError, e:
            if e.args[0] == 3:
                log.warning("--> no process with pid %s" % old_pid)
            else:
                log.error(e)
                exit()
        else:
            log.info("--> killed process %s" % old_pid)
        try:
            os.unlink(pidfile)
        except OSError, e:
            log.error("--> could not remove pidfile, %s" % pidfile)
            exit()
    open(pidfile, 'w').write(pid)
    log.info("--> launched with pid %s" % pid)

…and the snippet that goes at the top of the daemon:

import sys, os
sys.path.append(os.path.dirname(__file__) + "/../")
util.confirm_pid(os.path.dirname(__file__) + "/../run/")

Say you’ve got a time-intensive script and you want to benchmark how long it takes—here’s a really simple and effective snippet.

from time import * #not best practice but... meh
startTime = time()

# super-complicated, time-consuming code

elapsed = time() - startTime
print "Finished. Total time: " + strftime('%H:%M:%S', gmtime(elapsed))

Simple!

…turns out to be really straightforward using the web.data() method (thanks to Brian for the heads-up!):

Flash code—for this example you’ll need the AS3CoreLib JPGEncoder class:

_video = new Video(640, 480);
_camera = Camera.getCamera();
_camera.setMode(640, 480, 30);
_video.attachCamera(_camera);
addChild(_video);

_jpegEncoder = new JPGEncoder(60);

_urlReq = new URLRequest();
_urlReq.url = "http://localhost:8080/addImage";
_urlReq.method = URLRequestMethod.POST;

_urlLoader = new URLLoader();

_snapShot = new BitmapData(_camera.width, _camera.height);
_snapShot.draw(_video);
_snapShotImage = new Bitmap(_snapShot);
var jpgStream:ByteArray = _jpegEncoder.encode(_snapShot);
_urlReq.data = jpgStream;

try {
	_urlLoader.load(_urlReq);
} catch (e:Error) {
	trace("Remote.request caught error: "+e);
}

And the Python (using web.py):

import web

urls = (
  '/addImage', 'addImage'
)

app = web.application(urls, globals())

class addImage:
    def POST(self):
        data = web.data()
        f = open("test.jpg", 'wb')
        f.write(data)

if __name__ == "__main__": app.run()

That’s it!

Want to interpret Python scripts in Apache on your OSX installation?  Look no further!

After about 4 hours of repeated head-to-wall collisions, I finally managed to get mod_python running with Apache 2.2.14 on my Snow Leopard machine.  Here’s what I had to do:

1. Use the MacPorts Apache instead of MAMP.  I absolutely could not get this to work for MAMP, which might be because MAMP uses version 2.062 of Apache.  All the following instructions are for MacPorts installs.  Once you install MacPorts, you can install Apache by doing

   sudo port install apache

2. Now you need mod_python, which is a module for Apache that allows the HTTP server to use the python interpreter “inside” the server.  With MacPorts, just do

   sudo port install mod_python

   If for some reason this doesn’t work, I’ve packaged it up along with the mod_python python package described next.  Put it in

   /opt/local/apache2/modules

3. Make sure you have the mod_python Python package (yeah, it’s got the exact same name as the Apache module but is a completely different thing).  Get into your Python interpreter and try

   import mod_python

   If it works, rejoice.  If not, I’ve included mine in a zip because I could not for the life of me figure out where to get it from, but I had it in one of my other Python installs. It goes here:

   /Library/Python/2.6/site-packages

4. Edit your httpd.conf file by adding the following:

   LoadModule python_module modules/mod_python.so
   
   SetHandler mod_python
   PythonHandler mod_python.publisher
   PythonDebug On

   This first tells Apache to load the module, which is a shared object file (.so).  Technical note for the curious: A shared object is a binary that contains compiled functions that can be directly addressed in memory, so you can call functions in it given a header file.  It’s similar to a Windows .dll.

   Anyway, the rest tells Apache to use mod_python’s VERY handy “publisher” function, described here.  This is a REALLY fast and easy way to start writing Python programs for the web.  Follow the examples there and you should be good to go!  Enjoy!

Using blog corpora from The Blog Authorship Corpus and the Political Blog Corpora, I have begun developing a simple blog-post generator for Enpersonator, the identity-creating software part of Automenon, an artwork in progress with Sofy Yuditskaya.

See Enpersonator in action here: http://soniayuditskaya.tumblr.com/

Enpersonator is in development here: http://github.com/virgildisgr4ce/Enpersonator.  It requires the Python Tumblr library, which requires Django for its utilities.  The Markov Generator is modified from Adam Parrish’s.

The current version uses all 5 of the blogs in the Political Blog Corpora, comprising 133,953 tokens.  Not too shabby, but still not enough to avoid lots of wholesale reproductions.