Categories: , ,
Posted by: bjb

I found this very helpful. It worked great with Django 1.3, PostgreSQL 8.3 in 2011/06:

And the penultimate step (filling in the primary key in existing rows) took around half a minute for almost 200,000 records on a not-particularly well-endowed laptop.

Summary:

CREATE SEQUENCE rcvh7_id_seq;
ALTER TABLE rcvh7 ADD id INT UNIQUE;
ALTER TABLE rcvh7 ALTER COLUMN id SET DEFAULT NEXTVAL('rcvh7_id_seq');
UPDATE rcvh7 SET id = NEXTVAL('rcvh7_id_seq');
ALTER TABLE rcvh7 ALTER COLUMN id SET NOT NULL;
Categories: ,
Posted by: bjb

I was looking at django sessions and was a bit confused until I read the source code.

django/contrib/sessions/models.py
django/contrib/sessions/backends/base.py
django/contrib/sessions/backends/db.py
django/contrib/sessions/backends/cache.py
django/contrib/sessions/backends/cached_db.py
django/contrib/sessions/backends/file.py

You might think the interesting file to look at is django/contrib/sessions/models.py, but really the “toplevel” session object is defined in base.py. The object in base.py SessionBase. It is a base class for the various session implementations.

If you’re using a database-backed SessionStore, then you’ll be using base.py, db.py and session.py. db.py uses session.py for the database model and database interaction. In a view, the session object in the request object that is supplied by the session middleware is actually a SessionStore object. That is the object that has the methods get_expiry_age, etc.

I wanted to get some info out of every session in a batch mode including expiry age, so I needed to traverse SessionStore.objects.all(), not Session.objects.all().

Categories: ,
Posted by: bjb

When learning about formsets, it was never clear to me how many submit buttons there would be:

  • one per form
  • one for the whole formset.

It turns out, there is one submit button for the whole formset.

Categories: ,
Posted by: bjb

South migration names cannot have hyphens in them. grrr.

In fact they cannot have anything but alphanumeric characters and underscore.

Categories: ,
Posted by: bjb

My app wasn’t running in the virtualenv I had prepared for it … I activated a virtualenv by sourcing /usr/local/pythonenv/DJANGO-1-1/bin/activate, and then ran ./manage.py runserver. However, according to the debug stuff I put in the template (django.get_version() and python.VERSION), the app wasn’t using the right django.

It turns out that manage.py has #!/usr/bin/python at the top — it wasn’t running the python in the path but a hardcoded path to the system python. To use the virtualenv’s python, change that top line to:

#!/usr/bin/env python

This will use the python that can be found in the path of the calling process — which is what you want when using virtualenv.

It may be that newer django’s will generate the /usr/bin/env line, but if you created your project with an older django, you might need this step.

Categories: , ,
Posted by: bjb

If you are using pip install somepackage, and it is not going into your virtualenv, perhaps you are tripping over the same thing I did.

I had activated the virtualenv with /usr/local/pythonenv/DJANGO-1-1/bin/activate, and (DJANGO-1-1) was appearing in the shell prompt. which pip showed /usr/local/pythonenv/DJANGO-1-1/bin/pip. python followed by

import sys
for p in sys.path:
    print p

showed that the system site-packages directories were not being included. So why was the python package being installed to /usr/local/python2.6/site-packages???

Well, the pythonenvs had been installed as root. I was actually running sudo pip install somepackage and that meant I was losing the virtualenv environment (which sets some environment variables that don’t survive the sudo to root transition).

So the solution was to run [UPDATED 2011/01/04]

sudo /usr/local/pythonenv/DJANGO-1-1/bin/pip install -E /usr/local/pythonenv/DJANGO-1-1/ somepackage

and indeed I threw in the --download-cache option to cut down on the download time (although I subscribe to DSL, my download times are closer to dial-up speeds) for subsequent installs into the DJANGO-1-0 and DJANGO-1-2 virtual envs.

Categories: , ,
Posted by: bjb

I really like django, but one thing it lacks is db schema migration. I’m trying out a django package called “south” that does that. (Debian package python-django-south).

South keeps track of migrations applied by adding a table to your database called south_migrationhistory. Of course, this table has to be added before south will work … you add it using syncdb. Fortunately, south keeps track of what it does and what syncdb does.

So you start using south by installing south, and then running ./manage.py syncdb. Now you have some new ./manage.py commands, called startmigration and migrate (also datamigration, schemamigration and graphmigrations, maybe I’ll look at those another time). To snapshot your initial db state or to detect changes to your schema, you use the startmigration command. To apply migrations to your database, you use the migrate command.

To snapshot your initial db state, use startmigration --initial. To detect changes to your schema, use startmigration --auto. The startmigration command will, in addition to other things, dump out the current db schema into the migration file. Every generated migration file contains a schema declaration towards the end. It also contains a migrate forwards and a migrate backwards command, for applying the migration and unapplying it automatically. The schema changes are detected by comparing the model data declarations with the migration schema dumps.

Some examples, for a db named clientportal and an app named portal:

# create migration named portal/migrations/0001_initial.py
./manage.py startmigration portal initial --initial

The initial migration is already in your database, so you don’t have to apply it yourself. Then you edit the portal/models.py file, and add a field. Then you can have south detect this and create a migration that applies the change to the database.

./manage.py startmigration portal add_field --auto  

You can use the newly created migration to change the database:

# will find all migrations named 0002_add_field
# and apply them (in alpha order of app name)
./manage.py migrate 0002_add_field 

I’m not sure how to better control the migrating naming and order of application. For instance, it seems that migrations are numbered sequentially within applications, but you don’t specify the number. So if you have more than one app (app a and app b), and you create a migrations in this order:

b/migration/0001_initial.py
a/migration/0001_initial.py
b/migration/0002_add_field.py

then you run ./manage.py migrate, the migrations will be applied in this order:

a/migration/0001_initial.py
b/migration/0001_initial.py
b/migration/0002_add_field.py

And I’m afraid that if I made a new migration (add_another_field) for app a, it would be called 0002_add_another_field and would be applied with all the other migrations (on ./manage.py migrate on a new db):

a/migration/0001_initial.py
a/migration/0002_add_another_field.py
b/migration/0001_initial.py
b/migration/0002_add_field.py

A little annoying when the migrations should have been applied in this order:

b/migration/0001_initial.py
a/migration/0001_initial.py
b/migration/0002_add_field.py
a/migration/0002_add_another_field.py

Hopefully there is a way to handle it, even if I don’t yet know what it is. Just being able to specify the 000n numbers would do it.

To list out the available migrations, you can ./manage.py migrate --list. The migrations that have been installed have a * next to them.

UPDATE 2011/Jan/12: to revert to an old version: ./manage.py migrate appname 0006_shortname_not_null

brings you to the state just after 0006_shortname_not_null has been applied.

Posted by: bjb

I’m trying out byteflow under django 1.2, and I finally have it working. There were only a few changes to make.

First off, the databases are specified differently in django 1.2 — there is the option to connect to multiple databases now. Strangely however, django did not force me to change my database settings … I guess there is some backward compabitility stuff for now.

old settings_local.py

    DATABASE_ENGINE = 'postgresql_psycopg2'
    DATABASE_NAME = 'byteflow'
    DATABASE_USER = 'db_user'
    DATABASE_PASSWORD = 'sekrit'

new settings_local.py

    DATABASES = { 'default' :
        {
            'ENGINE' : 'postgresql_psycopg2',
            'NAME'   : 'byteflow',
            'USER'   : 'db_user',
            'PASSWORD' : 'sekrit',
        }
    }

There are also some deprecation warnings in the logs about admin.site.root, in urls.py (I may have added all those ‘settings.BLOG_URLCONF_ROOT’ in when using ‘bjb’ as my URL_PREFIX):

old:
    url(r'^%sadmin/(.*)' % settings.BLOG_URLCONF_ROOT,
        admin.site.root, name='admin'),

new:
    url(r'^%sadmin/(.*)' % settings.BLOG_URLCONF_ROOT,
        include(admin.site.urls)),

But this didn’t work for me so I went back to the old way. The problem was that when I asked to edit a blog post, it brought me to the main admin page. When I clicked on the Change link, it stayed on the main admin page. I’ll have to look into that another time.

An update was required to apps/tagging/managers, in usage_for_queryset, to update the database query for the django 1.2 database scheme (multiple databases). I found this hint.

Also, in order to run django 1.2 on my stable machine, I set up a virtualenv (with --no-site-packages) in which to run it. Had to install all the site-packages into the virtualenv:

  • BeautifulSoup-3.1.0.1
  • Django-1.2.1
  • PIL
  • mx
  • openid
  • psycopg2-2.0.7

That’s about it except for infrastructure:

  • easy-install
  • pip-0.8
  • setuptools-0.6c8

Well I suppose I should have started by upgrading byteflow, I’ll have to try that another time. Maybe some of my changes have been fixed in upstream already. However I did quickly note that the byteflow install page still says it requires django 1.0.

Posted by: bjb

Using my blog for it’s intended purpose: notes to self.

django on my server is an older version, my apps developed on my desktop need a newer version of django. I need to deploy virtualenv on my server so I can run a newer version of django for some apps. Like ipaddr and byteflow.

byteflow in particular, needs jquery.js (for attaching an image to a post, via postimage app and TAGGING_AUTOCOMPLETE_JS_BASE_URL setting) and that is not included in django 1.0.2.

But I’m out of time for today, so this is a task for another day.

Categories: , ,
Posted by: bjb

I created a new django app on my development machine running Debian SID (updated Aug 14-ish) and wanted to run it on my server machine running Debian stable (lenny). It didn’t work:

 machine    |           devel                   server
----------------------------------------------------------
Debian      |        sid (Aug 14)           stable (lenny)
django      |            1.2                      1.0
python      |            2.6                      2.5

There were complaints about missing modules, missing middleware, etc etc. Commenting out those bits resulted in more complaints about other things. So I figured I’d just have to run a “private” copy of django for that application.

To figure out what to copy, I looked at the python-django package contents using

dpkg -L python-django

and that pointed me to this directory full of django implementation files:

/usr/lib/pyshared/django

So I copied that over to the stable machine next to the little application, told the application to use that django by inserting that directory at the beginning of PYTHONPATH in the wsgi script, and ran the application.

It couldn’t find the module core.handlers.wsgi

That file was there …. but no __init__.py file in core/handlers.

It turns out there were lots of missing __init__.py files … and it turns out that although Debian installs the django python implementation files in /usr/lib/pyshared/django, it uses them from another directory /usr/lib/pymodules/python2.6/django, which is a mirror directory structure with a bunch of links to /usr/lib/pyshared/django for the files in /usr/lib/pyshared/django, plus some other files (like the missing (and usually empty) __init__.py files, plus a pile of .pyc files).

My guess is that Debian makes the mirror directory so that the .pyc files will not mess up the “source” install directory.

The upshot is, that if you’re going to run another django by copying django to a directory local to the application and altering the PYTHONPATH, copy the /usr/lib/modules/python2.6/django directory and not the /usr/lib/pyshared/django directory.

But, the better solution is to use python virtual environments (venv). My app is a tiny thing that only uses django and nothing else (django was really overkill for my app) but it’s a Bad Idea to solve the “wrong-django” problem this way. For example, any time a file disappears from the new version of django, my app would still find it in the old path (which didn’t get removed when I altered PYTHONPATH).