Getting Data From a Bound but Unvalidated Form

I have a form with dynamic fields, meaning the answers available for field B depend on the answer for field A.  Therefore, when the form is submitted I get the value from field B, but when it’s redisplayed I have to read the value from field A and populate the choices for field B or else field B is blanked out in the browser.

I looked for a convenient way to read the bound data from a specified form field and was surprised that I couldn’t find one.  I could just read the POST, but this form is in a formset with a prefix, so I’d have to assemble a complicated key name like “foo_bar_form-1-fieldname”.  The closest I saw to an interface to get the data was the _raw_value(fieldname) method which I couldn’t get to work and is suboptimal since it’s an internal call and could change in the future.

I ended up having to go to the POST data anyway, but I saw a helper function in _raw_value(fieldname) that made it easier.  What I ended up doing was this:

field_value = self.request.POST[form.add_prefix('fieldname')]

add_prefix(fieldname) is a public method to take care of the prefix hassle for me, so reading the post value is doable.  The only thing to watch out for is that you have to handle potential invalid values when redisplaying a form with errors.

Vim Is Wrong

I just saw this post on Stack Overflow about some clever vim commands and as a vim user  I was glad to learn some new tricks I can use.  Then I read the reddit comments about the story and found this (screencapped in case that link changes):

When writing software the code is written once but probably read dozens of times.  Therefore, readable code is very important.  This is one of the central tenets of Python.  These commenters are debating whether readability is equally as important with Vim commands as it is in programming languages.

Previously I would have sided with the advocates of Vim’s terse command syntax without much thought.  Short commands are easier to type and can be more effective.  However, this thread made me reconsider.  Because of Vim’s terse commands, they are harder to learn and to remember.  And once a command is learned, it may be learned as a magical incantation and not a sequence of instructions, so the components won’t be learned and reused for other needs.  So unless a Vim user spends a considerable amount of time learning commands, they’ll be using the editor inefficiently and getting less done than they could because of Vim’s terse commands and steep learning curve.  I’m no longer sure that the efficiency gained by having short, easy to type commands is worth the efficiency lost by having a large set of hard to remember commands in your brain, and an equally large set that you aren’t using because you don’t know them.

Python has a relatively small syntax.  Most things are explicit so there aren’t a bunch of implicit things you have to remember to know what your code is doing.  The advantage that gives a developer is that they can write code more quickly without having to look up syntax rules or function calls all the time.  Also, a new developer can come to the code and will understand the program more quickly with fewer things to learn.  As I thought about Vim’s command complexity, it occurred to me that we already have a command set that’s easy to write, read, and remember, and it’s quick and easy to write a short script like the one in that Stack Overflow comment: Python!  Why can’t Vim use Python to edit the working document?  The Python scripting support is a good start for complex add-ons, but why not make it even more integrated with things like :find(‘query’) instead of /query?  I’m very curious if a readily accessible, full blown python interpreter would make my vim use more effective.

P.S.  Emacs? </blasphemy>

Get The Display Name Of A ModelChoiceField’s Selected Value in Django

I just had a formset where each form had a pre-loaded ModelChoiceField, and I needed to display the name of the selected choice in the template.  On a model you would just use the get_FOO_display() method, but a form has no such convenience.  I found a number of almost solutions on Stack Overflow, and after poring over them I concluded the best solution for my problem was to add the following method to the form:

class QuestionLanguageForm(forms.Form):
    ''' basic QuestionLanguage form '''
    language = forms.ModelChoiceField(queryset=Language.objects.all())
    short = forms.CharField()
    place_holder = forms.CharField()
    long = forms.CharField(widget=forms.Textarea(attrs={'rows':None, 'cols':None}))

def get_language_name(self):
    ''' returns the name of the selected language '''
    try:
       return Language.objects.get(id=self.initial['language']).name
    except:
       return None

This way I can use

{{ form.get_language_name }}

in my template and it prints the human readable name of the language set in my initial data.  This code is not robust; I catch all exceptions and ignore them because I’m only using it in a circumstance where this code won’t raise an exception (and in case it does, at least it won’t fail catastrophically).  However proper error handling could be added pretty simply.  #ExerciseForTheReader

P.S.  The title says “selected” value but I’m actually using a pre-loaded value, not anything user selected.  If you need to get a user selected value you could use self.data or self.cleaned_data instead of self.initial.

Convert Django Model Instances To Dictionaries

I just searched for a method to convert a model instance to a dictionary in Django and the top few results were a bunch of custom methods in Django Snippets and Stack Overflow.  I was about to use one of those when I clicked on one last link that showed me a better solution.  There is a method already built into Django that does exactly what I want: django.forms.models.model_to_dict.

As I pointed that out to a coworker, he showed me that that’s basically the same output as the .values() method in the queryset api.  If you don’t specify a value to retrieve, it will retrieve the whole model instance as a dictionary.

Here’s the output of these two pieces of code:

>>> Q.objects.filter(id=6).values()
[{'short': u'oui', 'deleted': False, 'language_id': 3L, 'long': u'oui', 'place_holder': u'oui', 'id': 6L, 'question_id': 8L}]
>>> model_to_dict(Q.objects.filter(id=6)[0])
{'short': u'oui', 'language': 3L, 'deleted': False, 'question': 8L, 'long': u'oui', 'place_holder': u'oui'}

There are two differences here, and one of them is pretty important.  First, .values() turns a queryset into an iterable that yields dictionaries, and model_to_dict turns a model instance into a single dictionary.  As long as you know about it this is pretty trivial to overcome.  Second, .values() converts foreign keys to {‘<field_name>_id’: #} and model_to_dict converts them to {‘<field_name>’: #}.  Since my purpose in using this was to populate a form I had to convert the id’s into model instances and update the dictionary.  Not a big problem, but a bigger annoyance to solve than the first difference.

IOError: request data read error

I couldn’t find a good simple explanation of what this error means on the web, so to future googlers, you’re welcome.

This most likely means that the user made a web reqeuest, probably through ajax, on your site and then cut off the connection before the ajax request completed.  The connection can be cut off by clicking on another link, closing the browser, or things in the middle like firewalls and whatnot.

There isn’t a good solution to this issue that I could find, the error is just something that happens from time to time.  If it happens regularly and you don’t think it’s an accidental connection cut off, do some more exploring (particularly with firewalls and other potential connection problems), otherwise it can be ignored.

An experiment in statistical decryption of simple substitution ciphers

When people put up an encrypted text puzzle online it is frequently a simple substitution cipher whereby each letter stands for a different letter and the plaintext is revealed by discovering the correct replacements for each letter. I always wondered if it was possible to automatically decrypt these puzzles by analyzing the frequency with which each character appears and compare that to the frequency of each character in the english language. (Obviously I’m only decrypting english text, otherwise how would I know when it’s right? Regardless, this code will work equally well for any language.) Recently I decided to test the theory. The code is after the jump.

Continue reading

That’s the last time I trust Ubuntu to upgrade correctly

I host Barcamp Chicago’s website.  It’s a custom Django site on Ubuntu.  I recently upgraded the server to the latest LTS and later discovered my Postgresql database was gone.  Postgresql had gone from 8.3 to 8.4 in the upgrade, but since it didn’t warn me about needing to migrate the data I assumed it took care of that.  That was a mistake.  I had three databases on Postgresql 8.3 and none of them were present anymore.  I read on a forum that I could reinstall 8.3, but that person was working from Karmic not Lucid like I had.  I ultimately had to:

  1. add the Karmic repositories to apt,
  2. shut down 8.4,
  3. install 8.3,
  4. go through the normal data dump procedure for upgrading Postgresql manually,
  5. uninstall 8.3,
  6. start 8.4,
  7. and load all the data again.

After that the CLI showed my databases were present so I relaunched the barcamp site, but it still wasn’t connecting.  A little more googling revealed that Postgresql likes to increment the port number it listens on when there are two versions installed on the same machine.  That was indeed the problem, so I changed the port number back, restarted it and Apache, and finally I’m back to where I started.

I should have known better than to trust Ubuntu to migrate the data, but even if I did that myself I’d never expect a minor version upgrade to listen on a different port when that upgrade disables the old version anyway.

Testing Email in Django The Easy Way

Today a coworker showed me a very easy way to test django code that sends emails.  It’s straight from the documentation:

Another approach is to use a “dumb” SMTP server that receives the e-mails locally and displays them to the terminal, but does not actually send anything. Python has a built-in way to accomplish this with a single command:

python -m smtpd -n -c DebuggingServer localhost:1025

This command will start a simple SMTP server listening on port 1025 of localhost. This server simply prints to standard output all e-mail headers and the e-mail body. You then only need to set the EMAIL_HOST and EMAIL_PORT accordingly, and you are set.

inner join

Today I had a sql query joining a bunch of tables that looked to be hanging.  I ran it and went to lunch, and when I got back it still wasn’t done.  So I killed all my connection threads, restarted the server, tried to find every way I could to kill any errant table locks that were hanging around, but nothing would make it complete.  Finally after going through the syntax again I noticed that I left off the condition joining one of the tables, so it was joining 500,000 rows * 100,000 rows and it just hadn’t finished in the hour I left it to work.

I was more likely to have this problem because of the joining syntax I was taught:

select *
from table1, table2
where table1.id = table2.id

As you add more tables the where clause gets longer, and combined with the rest of the query’s constraints it gets easy to lose track of the ones that are necessary to do the joins properly. My boss never has this problem because he uses a different syntax:

select *
from table1
inner join table2 on table1.id = table2.id
where ...

This way, you can easily see what constraints are joining the tables together and would know if there was something missing. Also, this syntax is more like what you would use for left and right joins, so your queries will be more consistent. So from now on I’m going to use this syntax to try and avoid wasting time figuring out a problem that’s so easily preventable.

Making Things Work or Doing Them Right

When I’m trying to figure out how to get my CSS styles to render properly there are usually a number of possible solutions that “work”. I sometimes have trouble deciding which one is the best: the one with the most concise CSS? The one with the least entanglement with the HTML? It can be hard to know which way is “right”.

While I was working on the portfolio on this site, I had a problem like this. I wanted to float the screen shot to the left so the text would wrap around it, but I had to get the container for each screenshot/description to actually encapsulate them so its external spacing would line up properly. The usual way I knew to do it was to put another element in there that cleared the float so the container would be forced down where I want it. However this seems hackish and entangles the display with the content more than I like, so I looked for a better way.

I found a page on the subject of clearing floats and immediately I could see that the solution described there was the “right” way to do it. It’s simple, unentangling, and above all it works. This is one I’ll have to remember for next time.