If you attempt to store an object more than approximately 1 MB in size using memcache in the Google App Engine, it will give a ValueError, something like this:

ValueError: Values may not be more than 1000000 bytes in length; received 1088171 bytes

Solution

I’ve added a library to my Appswell framework that allows you to get around this limit by serializing an object into multiple strings and storing these along with an index object that stores the key.

Usage Example:

import multicache as memcache

# cache params
cache_data = some_large_nested_dict
cache_key = 'test_multicache'
cache_len = 60

# save data
memcache.set(cache_key, cache_data, cache_len)

# retrieve data
retrieved_data = memcache.get(cache_key)

The module can be easily extracted from the framework. See these links for additional details:

source code: http://code.google.com/p/appswell/source/browse/appspot/lib/multicache.py
wiki page: http://klenwell.com/is/AppengineMulticache

from google import WebApp, Predict, Users, Maps, Html5, NsaProxy

user = Users.lookup_by_existing_cookies_and_ip()
address = Predict.get_address('Enter the address where you grew up:')
data_raper = NsaProxy.init(user, address)

try:
    map = Maps.load_data(address)

    if map.context == 'urban':
        kid = Html5.load_urban_kid()
    else:
        kid = Html5.load_suburban_kid()

    video = WebApp.play(Html5.load_videos(kid, map))

    while video.playing():
        data_raper.rapes_your_personal_data()

except Html5.MovieOver:
    WebApp.redirect_to_launchpage()
except:
    WebApp.redirect_to_chrome_required_page()

Great album, by the way.

This is the official FormEncode explanation of state:

All the validators receive a magic, somewhat meaningless state argument (which defaults to None). It’s used for very little in the validation system as distributed, but is primarily intended to be an object you can use to hook your validator into the context of the larger system.

For instance, imagine a validator that checks that a user is permitted access to some resource. How will the validator know which user is logged in? State! Imagine you are localizing it, how will the validator know the locale? State! Whatever else you need to pass in, just put it in the state object as an attribute, then look for that attribute in your validator.

Hmmm. Maybe an example would help. Here’s the best example of its usage I could find on the Pylons site:

  this_schema = TheAppropriateSchema()
  try:
    state = dict() # this won't actually work. you'll need an object that formencode can hang things on.
    state["useful_state_information"] = something_useful
    form_result = this_schema.to_python(postvars, state)

    # validation is successful - form_result contains good data for you to consume

    redirect_to("somewhere")

  except Invalid, e:
    defaults = request.params
    errors = e.error_dict
    unfilled_html = a_function_that_draws_your_form_page_html()
    filled_html = htmlfill.render(unfilled_html, defaults, errors)
    return filled_html

Notice the comment: # this won’t actually work. Thanks for the warning.

So what is state and why would you want to use it?

Let’s answer the second question first, because at work, where we’re using the Pylons framework, I came up with an excellent situation which helped me figure out what it is and why I would want to use it.

The situation, briefly: our application has two separate forms. One is a form where a user can add new records in a multi-row table form. Each row is an individual record. The second is a review form, which looks just like the new record form, but is filled in from data uploaded in csv file (from another form).

Since DRY is a guiding principle of development in my office, the goal is to use the same underlying code for the form, controller, validation, etc. But the problem is that data originate in two different formats: one is as POST data from the add form, the second as the contents of a csv file. So how can we normalize these two data formats so that we can use the same underlying code? All together now: State! Now you’re talking!

If we could just tell our FormEncode validator where the data was coming from, then we could create two separate normalization methods in our validation class that would transform the data into a form that the validator and template could deal with. Anyway, enough verbiage. Find below a representative generalized version of the FormEncode validator I created and the controller it’s used in.

One more note before I finish. Notice the FormencodeState object comment. That explains why the example from the Pylons wiki does not work (in most cases). My first impulse would be to use a dict, too, as the state object. But the internals of the FormEncode validator class require an object. This is what gets passed in the validator method state argument and it is what passes info from the general framework environment to the validator.

Pylons Controller Code (with State Object)

class FormencodeState(object):
    """
    State class for formencode
    Although NOT well documented, to use the state argument in the to_python
    method in the context of schema that does complex, multistep validation,
    the state argument must be an object that formencode can hang additional
    attributes from, else you get errors like:
    Module formencode.schema:114 in _to_python
    >>  state.full_dict = value_dict
    <type 'exceptions.AttributeError'>: 'dict' object has no attribute 'full_dict'
    """
    pass

# ... within actual Controller class method
Validator = SourceDataPreValidator()
ControllerState = FormencodeState()
ControllerState.source = 'csv'
Validator.to_python(DataValue, ControllerState)     # <- state object in action!

FormEncode Validator Code

class SourceDataPreValidator(formencode.validators.FormValidator):
    """normalize data from either a form submission or a csv file upload and validate"""
    validate_partial_form = True
    def _to_python(self, value, state):
        """normalize csv input"""
        if state.source == 'csv':
            value['normal_data'] = self._normalize_csv_data(value.get('csv_import_file'), state)
        elif state.source == 'form':
            pass
        return value

    def _normalize_import_data(self, csv_file_string, state):
        NormalDataList = []
        ImportedLines = csv_file_string.split('\n')
        for i in range(len(ImportedLines)):
            ColValues = ImportedLines[i].split(',')
            fpre = 'csv_import-' + str(i)
            NormalDataList.append((fpre + '.id', str(ColValues[1])))
            NormalDataList.append((fpre + '.amount', str(ColValues[2])))
            NormalDataList.append((fpre + '.date',
                self._importdate_to_python_date(ColValues[4])))
        return NormalDataList

    def _importdate_to_python_date(self, datestr):
        """some code to convert a date string to object"""
        pass

I hope that sheds a little light on this powerful mystery. Questions or comments welcome.

What if I need to compare really big numbers in PHP? Like comparing 2^66 > 3^53?

Overview

This will work:

print (int) (pow(2,66) > pow(3,53));

PHP will convert the integers to scientific notation. But this script illustrates the limitation of normal operational syntax (i.e.: pow(2,66) > pow(3,53) vs. bccomp(bcpow(2,66), bcpow(3,53),1) > 0):

$max = 1000;
foreach ( range(1,$max) as $x )
{
    if (
        ! ( pow(2,$x)+1 > pow(2,$x) ) ||
        ! ( pow(2,$x)-1 < pow(2,$x) ) ||
        ! ( pow(2,$x-1)*2 == pow(2,$x) )
    ) break;
}

$maxpow = $x-1;
$maxint = bcpow(2, $maxpow);
$Result['op'] = "max supported int: 2^$maxpow or $maxint\n";

foreach ( range(1,$max) as $x )
{
    if (
        ! ( bccomp(bcadd(bcpow(2,$x),1), bcpow(2,$x)) == 1 ) ||
        ! ( bccomp(bcsub(bcpow(2,$x),1), bcpow(2,$x)) == -1 ) ||
        ! ( bccomp(bcmul(bcpow(2,$x-1),2), bcpow(2,$x)) == 0 )
    ) break;
}

$maxpow = $x-1;
$maxint = bcpow(2, $maxpow);
$Result['bc'] = "max supported int >= 2^$maxpow or $maxint\n";

printf("<pre>%s</pre>", print_r($Result,1));

Solution

Use bccomp. The script above can be found on the klenwell code site:

http://code.google.com/p/klenwell/source/browse/trunk/pastebin/bc_demo.php

Results will vary depending on the processing capacity of the system. Here’s the results on my machine:

In CakePhp, I want to invalidate a field submitted in a form and set the error message dynamically. If the error message is set by a validation parameter within the model, the new message should override that error message and be displayed by the form in the view.

Overview

Consider the following three cases as a developer:

1. You have a textarea field which requires user input of at least 12 words. When the user inputs less than 12 words, you’d like to display a message that states, “This field requires a response of at least 12 words. Your response was $x words long” where $x is the actual number of words in the user’s input.

2. You have a form with three input fields. None of them is required in and of itself. However, for the form to be valid, 2 of the 3 must be filled in.

3. You have an application in which one user submits a form which is then reviewed and approved by a second. The form has a “summary” field which the first user may fill in but does not have to. However, the second user must fill it in if it is empty.

Each of these requirements describes a somewhat complex validation case that CakePhp’s normal method for configuring data validation within the model doesn’t adequately address.

Solution

Rather than using the static form of validation described in the documentation, or even a custom validation rule which, although it may suffice, is still oriented around a single field with a static method, I recommend validating the field dynamically from within the controller or a custom model method using the model’s invalidate field.

The following commented code demonstrates a technique which could be used to satisfy each of the three cases above:

<?php

class MyModel extends AppModel
{
    var $name = 'MyModel';
    var $my_textarea_min_word_req = 12;
    var $ValidCodeList = array(
        0 => 'pass',
        1 => 'fail',
        2 => 'requires further review'
    );

    // snipped

    // essentially a wrapper for save with some extra circumstantial validation
    function update_record($FormData)
    {
        $UpdateData = $FormData['MyModel'];

        // situation-specific validation: this code is only required in this
        // instance so we don't set a rule within the model's validation parameters
        // above.  note: this could also be given its own separate method
        $textarea_wordlen = str_word_count($UpdateData['my_textarea']);
        if ( $textarea_wordlen < $this->my_textarea_min_word_req )
        {
            $s_ = ( $textarea_wordlen === 1 ) ? '' : 's';   // for the grammar nazis

            // note that it is here we're dynamically setting the error message
            $this->invalidate('my_textarea',
                "This field requires at least {$this->my_textarea_min_word_req} words.  Your response was {$textarea_wordlen} word{$s_}.");
        }

        // in update only, an internal code is required
        if ( !isset($this->ValidCodeList[$UpdateData['internal_code']]) )
            $this->invalidate('internal_code', 'please select a valid review code');

        // if the other fields are valid, the model will be saved
        if ( $this->validates($UpdateData) )
        {
            $this->create();
            $this->id = $UpdateData['id'];        // set id to update record
            return  $this->save($UpdateData);
        }
        // return false (which will lead back to the form)
        else
        {
            return 0;
        }
    }
}

?>

In the end, it’s pretty simple and straightforward. Still, I could not find it clearly spelled out in any one place in the CakePhp documentation or elsewhere on the web.

One caveat to note that will further explain the inclination to forego usage of the model’s validate parameter above the I came across here: http://groups.google.com/group/cake-php/msg/095804b352539b97?hl=en. If you do have a validation rule set for that field within the model, the message will get overridden by the default error message there if the field also fails the validation set there.

I need to validate input from a textarea field. I want to allow a few tags, like a, i, b, etc. But everything else needs to be filtered out. And the input should be checked to see that its is nested properly.

Overview

User input sanitization and validation is one of those things that just needs to be done. Dealing with a textarea is more complicated and validating html for re-display on a web page is less trivial than it seems at a glance. It’s a pain in the ass so I like it when I can find someone who has already solved the issue. I found a new solution here: HTML Purifier.

The author, ezyang, offers a detailed study of the issue as well as a short history of the PHP HTML validators that preceded his and where they fall short. It’s well-done, an impressive work of engineering and scholarship, but I hesitate to use the class because it’s just so damned large. Also, the dependencies are a little bit confusing to me, though I am sure they could be sorted out easily enough by testing the class. On the plus side, the API is simple enough. But I don’t need the kind of comprehensive solution that’s offered here.

The Process

Since the solution I am looking for involves two parts: (1) sanitizing the input and (2) validating the markup, I figured I’d take the Reese’s approach and just try to combine two existing classes that will taste great together. The data sanitization component is based on the Input Filter class that the ezyang critiques, justifiably, as inadequate. It does the job sanitizing input. Where if falls short is in validating the markup to make sure it’s properly formed. That’s where the second source comes in: Simon Willison’s Safe HTML Checker. This relies on Php’s native XML parser and is nice and short.

I’m calling my class Input Baffle, where baffle means “A device used to restrain or regulate.” It basically acts as a wrapper (is this the composite pattern?) for these two subordinate classes.

Even with the hard work done for me, the problem still proved thorny and took me several hours to work out all the significant wrinkles. The major complications I encountered included:

1. Parser Iteration in Safe HTML Checker

I set up my class to create a single instance of the SafeHtmlChecker class as a member. SafeHtmlChecker is essentially a wrapper for Php’s native XML Parser. The problem with this is that there isn’t a way – at least, an obvious one – to reuse the parser after it has detected an error. This is a problem, as in my unit test, where my class might be validating more than one field in a form. The problem is identified here. The solution was to have each call to validate the submitted markup create, and destroy, a separate parser.

2. Recursion Overflow in InputFilter

Try running this <<<>>> through InputFilter. Memory overflow. To solve that problem, I added a preprocessor to my class that uses regex to eliminate this and a few other hobgoblins that InputFilter misses.

3. InputFilter vs. the rsnake XSS Cheat Sheet

Discovered the XSS cheat sheet on the HTML Purifier site. An excellent test set, I added methods (based on HTML Purifier’s smoke test), to run the test. HTML Purifier aces the test. InputFilter does not. InputFilter, in conjunction with SafeHtmlChecker catches all the exploits except one. So I just added that one to the preprocessor.

Solution

You can find my code, with unit tests, on my Google Code site:

http://klenwell.googlecode.com/svn/trunk/projects/php/kwoss/input_baffle/

My goal ultimately is to convert this to a CakePhp behavior.

I already knew the stakes involved with input filtering and validation. And I knew it was not a trivial problem. Working through this, and referring to ezyang’s more fastidious solution, gave me a much deeper appreciation for how complex it really is.

As the ticket notes:

You can only paginate LEFT JOIN tables by default. hasMany relationships are performed as a separate query, and thus cannot have conditions in the primary paginate() call. If you wish to do this, you need to implement paginate() and paginateCount() in your model.

So that’s what I did. Only it wasn’t quite as simple as I thought it would be.

The CakePhp manual does provide some guidance: 4.9.4 Custom Query Pagination. This comes from the example originally posted here (littlehart.net), where there are additional comments to both confuse and clarify a bit.

One thing that I didn’t see ever made explicit is how exactly you integrate the custom model methods you create, paginate and paginateCount, with the paginator syntax in your controller or helper. Well, it turns out, as you would probably hope, that it’s fairly seamless. You just use the standard pagination markup , and the custom methods override Cake’s default methods (this changeset illustrates).

With hasMany associations, as is the case in the example above from Cake’s website, you are probably going to need to use GROUP BY syntax. For my particular case, this required a custom SQL query, which meant using Cake’s Model class’s query. (I’m a little surprised and concerned that it doesn’t support bind parameters.)

I’ve posted the relevant code to my wiki: http://www.klenwell.com/is/Paste20090117

I hope it helps steers someone to the answer they’re looking for.

One way to make a radio list more efficient is to break it up into columns. I’ve written a CakePhp helper that accomplishes this. An illustration:

radio group without (l) and with (r) columns

It turns out that CakePhp Form helper’s input method gives me all the tools I needed it to adapt my code pretty easily to it.

I’ve pasted the code over on my wiki: http://www.klenwell.com/is/Paste20090110

The part of the code I think is most interesting is the code I use to divvy up the options array into the columns:

// get option list
$OptionList = explode($this->separator_marker, $middle);
$num_options = count($OptionList);

// snip

// divvy options into columns (wrote this code a while ago and seems to work)
for( $i = 1; $i <= $num_cols; $i++ )
{
    $COL[$i] = ceil( $num_options/($num_cols+1-$i) );
    $num_options = $num_options - $COL[$i];
}

I look at it now and I know the gist of what it’s doing, but still have to really stop and think about it to fully understand it. I think it’s interesting because it shows how my mind works better procedurally than it does mathematically.

Here’s an illustration showing again the columnized radio list along with the class and id markup that can be used for styling:

radio columns with id/class markup

I’ll eventually add this the cakewell app in my google code repository.

One caveat: this works with the Form helper’s input method (type set to ‘radio’), but does not seem to work with the Form helper’s radio method.