Posted by: bjb

Recently I recorded an episode for Hacker Public Radio. It answers some questions that another contributor, knox, had about virtualenvwrapper. My episode will (if all goes well) be published on June 27 as episode 2322.

 continue reading
Posted by: bjb

I looked up a question on stackoverflow, and even found it. But it had no answer, so I wrote one.

With luck, it’s not too far wrong.

… well I guess I answered the wrong question. Oops.

Can an existing folder be redeclared a virtual enviroment with VirtualEnvWrapper?

I thought this was a project that had been made with virtualenv, and they wanted to put it under virtualenvwrapper. But apparently it was made with virtualenvwrapper, and I don’t know what they want to do with it.

They can probably get the answer to their question from my answer, though.

My answer:

Yes.

Anatomy of virtualenvwrapper

The existing project has two parts:

  • the virtualenv, where python and the python libs are installed, and
  • the project directory where your code is (that uses the virtualenv).

Virtualenvwrapper adds a third part, the virtualenvwrapper hooks. These are mainly shell functions that are called at certain times in a project or virtualenvs life cycle. They live in a third directory — by default, they are installed to ~/.virtualenvs (at least that was true on my Debian wheezy system). The hooks include postactivate, which we will edit below, and a bunch of others such as premkproject, premkvirtualenv, etc. The following list of keywords gives you the flavour of the hooks: initialize, pre/post, mk/rm, project/virtualenv, activate/deactivate. virtualenvwrapper puts these scripts in $VIRTUALENVWRAPPER_HOOK_DIR, which defaults to $WORKON_HOME.

virtualenvwrapper assumes

  • all the virtualenvs are in one place ($WORKON_HOME) (defaults to ~/.virtualenvs) Let’s say you have two virtualenvs, one called MYVENV and the other called MYOTHERVENV
  • all the project directories are in another place ($PROJECT_HOME).

Let’s say you have a project “is” in directory /home/me/where/my/proj that uses virtualenv MYVENV.

how to use workon to work on your pre-existing code and virtualenv

Here I’m assuming all your virtualenvs are in one place (in my case, they are all in /usr/local/virtualenv).

one-time operations

** edit ~/.virtualenvs/postactivate (+) to have

    case $env_name in
        MYVENV)
            cd /home/me/where/my/proj/is
            ;;
        MYOTHERVENV)
            cd /home/me/where/my/other/project/is2
            ;;
    esac

** links

    for hk in get_env_details initialize postactivate postdeactivate \
            postmkproject postmkvirtualenv postrmproject \
            postrmvirtualenv preactivate predeactivate premkproject \
            premkvirtualenv prermproject prermvirtualenv; do \
        ln -s ~/.virtualenvs/$hk /usr/local/pythonenv/$hk; \
    done

any time you want to work on your project that uses MYVENV virtualenv

    WORKON_HOME=/usr/local/pythonenv workon MYVENV

Of course if all your virtualenvs are indeed in the same place, you can define WORKON_HOME in your .profile and you won’t have to specify it on the command line every time.

Categories: , , ,
Posted by: bjb

I’ll be giving a talk tonight at OPAG called “Enough git for julython”. That’s right, julython is coming up in a few days. I want to help boost the Ottawa, Canada participation by removing a possible barrier to participation: I will be showing people how to use git and github.

Julython is basically an informal contest that encourages people to write some python code and check it in publicly into github or bitbucket.

The talk is at Shopify tonight, Thursday June 27, at 19:30.

Categories: ,
Posted by: bjb

One of the nice things about the Ottawa Python Author’s Group irc channel (oftc.net, #opag) is that they occasionally mention a great but under-advertized reference, like this one for setuptools:

http://peak.telecommunity.com/DevCenter/setuptools#basic-use

Thanks Ian!

Categories: ,
Posted by: bjb

I found a site where there is some not-just-good, but right-out excellent twisted documentation.

http://krondo.com/blog/?page_id=1327

Posted by: bjb

I have a database full of strings where the accented characters have been replaced by their non-accented equivalents, and a spreadsheet full of strings with accents in them. I’m supposed to look up the info in the database given the info in the spreadsheet.

I found this great stackoverflow post that helped me out:

title = u"some string with accented characters in it like b\xe9cancour"
import unicodedata
unicodedata.normalize('NFKD', title).encode('ascii', 'ignore')
'some string with accented characters in it like becancour'

Normalize with ‘NFKD’ will decompose each character in the string into its composing characters. For example, if there was an e with acute accent, it separates it into an e and an acute accent. The K part of NFKD ensures the ‘e’ is the simplest possible e (presumably if there is an ‘e’ in ASCII, it will prefer that one). Then the encode ('ascii', 'ignore') will drop all the non-ASCII characters, which by now are just the accents which have been separated from the rest of the letter.

Awesome. And it works in python 2.5.

Categories: , , ,
Posted by: bjb

Handy quick conversion for timestamps in Nagios or unix logs: time.localtime (time.time);

eg:

bjb@bjb-pc:~/work/redmine/redmine-abitibi/app/views/mailer$ python
Python 2.5.2 (r252:60911, Jan 24 2010, 17:44:40)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.time
<built-in function time>
>>> time.time ()
1284568746.503253
>>> time.localtime (1284523200)
(2010, 9, 15, 0, 0, 0, 2, 258, 1)
>>> time.localtime (1284520137)
(2010, 9, 14, 23, 8, 57, 1, 257, 1)
>>> 
Posted by: bjb

I’m trying out byteflow under django 1.2, and I finally have it working. There were only a few changes to make.

First off, the databases are specified differently in django 1.2 — there is the option to connect to multiple databases now. Strangely however, django did not force me to change my database settings … I guess there is some backward compabitility stuff for now.

old settings_local.py

    DATABASE_ENGINE = 'postgresql_psycopg2'
    DATABASE_NAME = 'byteflow'
    DATABASE_USER = 'db_user'
    DATABASE_PASSWORD = 'sekrit'

new settings_local.py

    DATABASES = { 'default' :
        {
            'ENGINE' : 'postgresql_psycopg2',
            'NAME'   : 'byteflow',
            'USER'   : 'db_user',
            'PASSWORD' : 'sekrit',
        }
    }

There are also some deprecation warnings in the logs about admin.site.root, in urls.py (I may have added all those ‘settings.BLOG_URLCONF_ROOT’ in when using ‘bjb’ as my URL_PREFIX):

old:
    url(r'^%sadmin/(.*)' % settings.BLOG_URLCONF_ROOT,
        admin.site.root, name='admin'),

new:
    url(r'^%sadmin/(.*)' % settings.BLOG_URLCONF_ROOT,
        include(admin.site.urls)),

But this didn’t work for me so I went back to the old way. The problem was that when I asked to edit a blog post, it brought me to the main admin page. When I clicked on the Change link, it stayed on the main admin page. I’ll have to look into that another time.

An update was required to apps/tagging/managers, in usage_for_queryset, to update the database query for the django 1.2 database scheme (multiple databases). I found this hint.

Also, in order to run django 1.2 on my stable machine, I set up a virtualenv (with --no-site-packages) in which to run it. Had to install all the site-packages into the virtualenv:

  • BeautifulSoup-3.1.0.1
  • Django-1.2.1
  • PIL
  • mx
  • openid
  • psycopg2-2.0.7

That’s about it except for infrastructure:

  • easy-install
  • pip-0.8
  • setuptools-0.6c8

Well I suppose I should have started by upgrading byteflow, I’ll have to try that another time. Maybe some of my changes have been fixed in upstream already. However I did quickly note that the byteflow install page still says it requires django 1.0.

Posted by: bjb

Using my blog for it’s intended purpose: notes to self.

django on my server is an older version, my apps developed on my desktop need a newer version of django. I need to deploy virtualenv on my server so I can run a newer version of django for some apps. Like ipaddr and byteflow.

byteflow in particular, needs jquery.js (for attaching an image to a post, via postimage app and TAGGING_AUTOCOMPLETE_JS_BASE_URL setting) and that is not included in django 1.0.2.

But I’m out of time for today, so this is a task for another day.

Categories: ,
Posted by: bjb

I had to look at a 9-MB json file this weekend. Here’s how I converted it from one-line to indented multi-line:

$ sudo apt-get install python-simplejson
$ dpkg -L python-simplejson
$ less /usr/share/pyshared/simplejson/tests/test_pass1.py

$ python
>>> import simplejson as json
>>> f = open (big.json, 'r')
>>> oneline = f.read ()
>>> f.close
>>> ds = json.loads (oneline)                  # ds = "data structure"
>>> multiline = json.dumps (ds, indent="  ")   # two spaces / indent level
>>> f = open (formatted.json, 'w')
>>> f.write (multiline)
>>> f.close ()
>>> ^D
$ 

In reviewing the steps for this blog post, I note that there is also a “load” function, that might be even easier.

>>> import simplejson as json
>>> dir (json)
['Decimal', 'JSONDecodeError', 'JSONDecoder', 'JSONEncoder',\
 'OrderedDict', '__all__', '__author__', '__builtins__', '__doc__',\
'__file__', '__name__', '__package__', '__path__', '__version__',\
'_default_decoder', '_default_encoder', '_import_OrderedDict',\
'_import_c_make_encoder', '_speedups', '_toggle_speedups',\
'decoder', 'dump', 'dumps', 'encoder', 'load', 'loads', 'ordered_dict',\
'scanner']
>>> 

Example input:

[{'pk': '5', 'model': 'theModel', 'fields': {'two': 'two', 'one': 'one'}}]

Example output:

[
  {
    "pk": "5", 
    "model": "theModel", 
    "fields": {
      "two": "two", 
      "one": "one"
    }
  }
]