But the underlying data must be extracted from several spreadsheets from the census site.

So here’s a simple Python list of the annual data (1950-2010):

[(1950, 10677000),
 (1951, 11134000),
 (1952, 11635000),
 (1953, 12251000),
 (1954, 12746000),
 (1955, 13133000),
 (1956, 13713000),
 (1957, 14264000),
 (1958, 14880000),
 (1959, 15467000),
 (1960, 15870000),
 (1961, 16497000),
 (1962, 17072000),
 (1963, 17668000),
 (1964, 18151000),
 (1965, 18585000),
 (1966, 18858000),
 (1967, 19176000),
 (1968, 19394000),
 (1969, 19711000),
 (1970, 19971069),
 (1971, 20345939),
 (1972, 20585469),
 (1973, 20868728),
 (1974, 21173865),
 (1975, 21537849),
 (1976, 21935909),
 (1977, 22352396),
 (1978, 22835958),
 (1979, 23256880),
 (1980, 23667902),
 (1981, 24285933),
 (1982, 24820009),
 (1983, 25360026),
 (1984, 25844393),
 (1985, 26441109),
 (1986, 27102237),
 (1987, 27777158),
 (1988, 28464249),
 (1989, 29218164),
 (1990, 29950111),
 (1991, 30414114),
 (1992, 30875920),
 (1993, 31147208),
 (1994, 31317179),
 (1995, 31493525),
 (1996, 31780829),
 (1997, 32217708),
 (1998, 32682794),
 (1999, 33145121),
 (2000, 33987977),
 (2001, 34479458),
 (2002, 34871843),
 (2003, 35253159),
 (2004, 35574576),
 (2005, 35827943),
 (2006, 36021202),
 (2007, 36250311),
 (2008, 36604337),
 (2009, 36961229),
 (2010, 37349363)]

And here’s a list of the raw data and its sources:

# raw data: (year range, raw numbers, unit/multiplier, delim, source)
    raw_data = (
        ( range(1950,1955),
          '10,677   11,134   11,635  12,251   12,746',
          1000,
          '\s+',
          'http://www.census.gov/popest/archives/1980s/st5060ts.txt' ),
        ( range(1955,1960),
          '13,133  13,713   14,264   14,880  15,467',
          1000,
          '\s+',
          'http://www.census.gov/popest/archives/1980s/st5060ts.txt' ),
        ( range(1960,1965),
          '15,870   16,497   17,072   17,668   18,151',
          1000,
          '\s+',
          'http://www.census.gov/popest/archives/1980s/st6070ts.txt' ),
        ( range(1965,1970),
          '18,585   18,858   19,176   19,394   19,711',
          1000,
          '\s+',
          'http://www.census.gov/popest/archives/1980s/st6070ts.txt' ),
        ( range(1970,1976),
          '19971069  20345939  20585469  20868728  21173865  21537849',
          1,
          '\s+',
          'http://www.census.gov/popest/archives/1980s/st7080ts.txt' ),
        ( range(1976,1980),
          '21935909  22352396  22835958  23256880',
          1,
          '\s+',
          'http://www.census.gov/popest/archives/1980s/st7080ts.txt' ),
        ( range(1980,1985),
          '23667902  24285933  24820009  25360026  25844393',
          1,
          '\s+',
          'http://www.census.gov/popest/archives/1980s/st8090ts.txt' ),
        ( range(1985,1990),
          '26441109  27102237  27777158  28464249  29218164',
          1,
          '\s+',
          'http://www.census.gov/popest/archives/1980s/st8090ts.txt' ),
        ( range(1999,1989,-1),
          '33145121  32682794  32217708  31780829  31493525  31317179  31147208  30875920  30414114  29950111',
          1,
          '\s+',
          'http://www.census.gov/popest/archives/1990s/ST-99-07.txt' ),
        ( range(2000,2011),
         '"33,987,977","34,479,458","34,871,843","35,253,159","35,574,576","35,827,943","36,021,202","36,250,311","36,604,337","36,961,229","37,349,363"',
          1,
          ',',
          'http://www.census.gov/popest/intercensal/state/ST-EST00INT-01.csv' ),
    )

Since performance is often a consideration with statistical computing, my testing module includes a basic timing function called timeit. The Python standard library includes a timeit module, but I don’t find it very friendly, as it creates a new process with its own separate environment. I just want a timing function that can time comparable pieces of code in the current environment. Here’s my timeit function:

def timeit(f, *args, **kw):
    """time a function over an n number of trails:

    f1 = lambda a: a * a
    def f2(a, b):
        f1(a) + f1(b)
    def f3():
        f2(10, 20)

    USAGE:
        t1 = timeit(f1, a, n=1000)
        t2 = timeit(f2, a, b, n=1000)
        t3 = timeit(f3, n=1000)
        pprint([t1, t2, t3])
    """
    n = kw.get('n', 100)
    print 'timing %s over %s trials' % (f, n)
    t0 = time.time()
    for i in range(n):
        f(*args)

    total_time = time.time() - t0
    per_trial = total_time / n
    return '%.2f (%s trials at %.6f per trial)' % (total_time, n, per_trial)

It takes a function as it’s first argument, a list of arguments, and an n keyword to specify how many times to run it.

The pattern I like best is to enclose the operation you wish to test in a function, then just pass that with the n keyword:

import pprint

num_trials = 100000
f1 = lambda: Percentile(scores, 50)
f2 = lambda: iPercentile(scores, 50)
t1 = timeit(f1, n=num_trials)
t2 = timeit(f2, n=num_trials)
pprint.pprint([t1, t2])

# output
timing <function <lambda> at 0x8950bc4> over 100000 trials
timing <function <lambda> at 0x8950bfc> over 100000 trials
['0.86 (100000 trials at 0.000009 per trial)',
 '0.38 (100000 trials at 0.000004 per trial)']

I have an App Engine app that keeps tracks of some tweets (“status” in the API). I decided I wanted to store the time the message was originally tweeted. So I need to update the schema of one of my models, TweetDigest, to add the new property (field). But then I also would like to update any existing records to include the value.

Solution

First things first. Let’s update the model. This is easy enough. Just add the new property to the existing model. The relevant code:

from project.models.twitter import TweetDigest

class TweetDigest(db.Model):
    tweet_id        = db.StringProperty(required=True, indexed=True)
    user_id         = db.StringProperty(required=True)
    screen_name     = db.StringProperty(required=True)
    text            = db.StringProperty(required=True, multiline=True)
    stored_at       = db.DateTimeProperty(auto_now_add=True)

    # new property
    tweeted_at      = db.DateTimeProperty(required=True)

That’s the easy part. Any new records will include that property. Existing records, however, will not have the property, at all. Let me emphasize that: it’s not that the new field is set to null for existing records. Existing records do not have the field at all. A couple queries in the interactive console illustrate:

from datetime import datetime
from google.appengine.ext import db
from pprint import pprint

print datetime.now()
count = TweetDigest.all().filter('user_image', None).count()
print count

record = TweetDigest.all().get()
pprint(record.__dict__['_entity'])

Output:

2011-09-05 00:11:25.657055
0
{u'screen_name': u'klenwell',
 u'stored_at': datetime.datetime(2011, 7, 6, 9, 26, 56, 757862),
 u'text': u'Premature optimization is the root of all evil.',
 u'tweet_id': u'1948390000',
 u'user_id': u'1820900'}

So how to fix this? Well, in this case, I have to retrieve the created_at value for each status using the Twitter API then update each record. I set up an action in a special controller to do this. It queries the datastore to fetch all the existing records, the creates a task for each that will query the Twitter API, get the created_at value, and update the record in the datastore.

Here’s the controller code that creates the tasks:

def add_tweet_created_at_field(self):
    # queue settings
    queue_name = 'tweetdigest-schema-change'
    queue_url_f = '/backend/queue/store_tweet_created_at_value/%s'
    queue_params = {}

    # purge queue
    self.purge_queue(queue_name)
    logging.info('purged queue: %s' % (queue_name))

    # select all TweetDigest records without image_url and add to queue
    queue_count = 0
    query = TweetDigest.all()
    for digest in query:
        if digest.tweeted_at is None:
            queue_url = queue_url_f % (digest.tweet_id)
            added_task = self.queue_task(queue_url, queue_params, queue_name)
            queue_count += 1

    logging.info('queued %s TweetDigest records for update' % (queue_count))

    # output
    response = {
        'queued tasks'          : queue_count,
    }
    self.set('data', pformat(response))
    self.render(self.default_view)

This action is runs within the appswell framework. The task code is left as an exercise for the reader.

References

http://code.google.com/appengine/articles/update_schema.html
http://stackoverflow.com/questions/7037269/check-if-a-field-is-present-in-an-entity
http://appswell.appspot.com/

If you attempt to store an object more than approximately 1 MB in size using memcache in the Google App Engine, it will give a ValueError, something like this:

ValueError: Values may not be more than 1000000 bytes in length; received 1088171 bytes

Solution

I’ve added a library to my Appswell framework that allows you to get around this limit by serializing an object into multiple strings and storing these along with an index object that stores the key.

Usage Example:

import multicache as memcache

# cache params
cache_data = some_large_nested_dict
cache_key = 'test_multicache'
cache_len = 60

# save data
memcache.set(cache_key, cache_data, cache_len)

# retrieve data
retrieved_data = memcache.get(cache_key)

The module can be easily extracted from the framework. See these links for additional details:

source code: http://code.google.com/p/appswell/source/browse/appspot/lib/multicache.py
wiki page: http://klenwell.com/is/AppengineMulticache

from datetime import datetime
julianday = '%d%03d' % (datetime.now().timetuple().tm_year,
    datetime.now().timetuple().tm_yday)

#
# MAIN
#
if __name__ == "__main__":
    print sys.modules[globals()['__name__']].__doc__    # print module doc
    suite = unittest.TestLoader().loadTestsFromTestCase(YourTestClassHere)
    unittest.TextTestRunner(verbosity=2).run(suite)

$ sudo crontab -e

Or link a cron file to /etc/cron.d:

$ sudo ln -sv /home/klenwell/crons/my.cron /etc/cron.d/mycron

I like option two because it provides a more modular and organized way to enable cron tasks for various projects. The only problem with this method is that, when things go wrong, it’s been a huge headache trying to figure out what the problem is and how to get the tasks in the linked cron file running again. The cron documentation and Google are frustratingly tight-lipped on the subject of /etc/cron.d. But I think I have figured out a reliable way to configure and trouble /etc/cron.d.

Some rules that will help in using /etc/cron.d:

1. You must specify a user in your cron file.
2. The script being linked must be owned by root (giving root permission alone doesn’t seem enough)
3. To force /etc/cron.d to reload, touch the /etc/cron.d dir: sudo touch /etc/cron.d/

I was able to confirm these rules by linking a cron file to /etc/cron.d, then changing the owner to myself (klenwell), then checking the /var/log/syslog file.

Cron checks /etc/cron.d minutely, so you have to give it at least 60 seconds to register any changes. I set up my test cron file (/home/klenwell/crons/my.cron) to touch a file in /tmp (/tmp/crontest) every minute.

When the cron file is owned by my user (klenwell), I see this message in the log:

Nov 26 02:55:01 myserver cron[1924]: (*system*mycron) WRONG FILE OWNER (/etc/cron.d/mycron)

The linked cron file will not run again until the owner is corrected and the directory touched:

$ sudo chown -v root /home/klenwell/crons/my.cron
$ sudo touch /etc/cron.d/

After this, I see this in /var/log/syslog:

Nov 26 02:57:01 myserver CRON[21170]: (klenwell) CMD (   touch /tmp/crontest)

Took me a lot of trial and error to figure this out. Hope it saves your an extra headache or two.

>>> d = {'a':1, 'b':2, 'c':3}
>>> d.get('a')
1
>>> d.get('d', 'not found')
'not found'

Sometimes, I find something like this would be useful for lists, using the list’s index in place of the dict’s key:

>>> l = [0,1,2,3]
>>> l.get(2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'list' object has no attribute 'get'

Here’s a quick lambda that accomplishes that:

list_get = lambda l,i,a=False: (len(l) <= i and a) or (len(l) > i and l[i])

Usage:

>>> list_get = lambda l,i,a=False: (len(l) <= i and a) or (len(l) > i and l[i])
>>> my_list = [0,False,None,3]
>>> list_get(my_list,0)
0
>>> list_get(my_list,1)
False
>>> list_get(my_list,2)  # returns None
>>> list_get(my_list,3)
3
>>> list_get(my_list,4)
False
>>> list_get(my_list,4,0)  # Note: returns False instead of 0
False
>>> list_get(my_list,4,1)
1

It’s not perfect. Be careful with False-equivalent values in the alt parameters as noted above. But otherwise it should work for most practical cases were such usage is desired.

from google import WebApp, Predict, Users, Maps, Html5, NsaProxy

user = Users.lookup_by_existing_cookies_and_ip()
address = Predict.get_address('Enter the address where you grew up:')
data_raper = NsaProxy.init(user, address)

try:
    map = Maps.load_data(address)

    if map.context == 'urban':
        kid = Html5.load_urban_kid()
    else:
        kid = Html5.load_suburban_kid()

    video = WebApp.play(Html5.load_videos(kid, map))

    while video.playing():
        data_raper.rapes_your_personal_data()

except Html5.MovieOver:
    WebApp.redirect_to_launchpage()
except:
    WebApp.redirect_to_chrome_required_page()

Great album, by the way.

The it’s-all-a-dream interpretation is the only one that makes sense from a science-fiction point-of-view, however unsatisfying it is from a critical perspective. The slightly wobbling top at the end points, rather overtly, to the real answer: it’s meant to be ambiguous.

That said: WTF? We’ve got to bring down a company on the verge of being The Most Powerful Corporation in the World. So let’s use this highly radical, unproven and improbable, voodoo-science inception approach. Oh, and by the way, we have to plant a mole within the highest ranks of the multinational beast within the next, say, two weeks, so we can get valuable information that is necessary to penetrate and manipulate the fragile psyche of our target.

Cut-scene: it’s done. This is logic that only makes sense within the context of a dream. The sort of dream Coleridge, or Rube Goldberg, would have.

Science generally favors the application of Occam’s Razor: buy the airline and crash the damn plane with the kid on it. Or just send in the ninjas. The ensuing infighting among the various executive factions vying for the top spot after junior is out of the picture will ensure that any plans Acme Energy Inc had for world domination are put off course indefinitely.

Inception was entertaining. But Memento was a more interesting film intellectually. That’s a film that truly merited this kind of philosophical scrutiny. Remember Sammy Jankis.

This was in support of the basic premise: Inception isn’t science fiction