I’m lucky enough to be sitting in the netconf conference, with Pablo Ayuso, Jamal Hadi Salim, David Miller and others. They are discussing current and proposed future work on the networking subsystem of the Linux kernel.
This is the by-invitation-only mini-conference. Many of the attendees here will speak at the open-registration netdev01 conference from Feb 14 — 17 at the same venue (the Westin Hotel, downtown Ottawa). It’s looking like that conference will be really great. It’s not too late to sign up! I’ll definitely be there.
I looked up a question on stackoverflow, and even found it. But it had no answer, so I wrote one.
With luck, it’s not too far wrong.
… well I guess I answered the wrong question. Oops.
Can an existing folder be redeclared a virtual enviroment with VirtualEnvWrapper?
I thought this was a project that had been made with virtualenv, and they wanted to put it under virtualenvwrapper. But apparently it was made with virtualenvwrapper, and I don’t know what they want to do with it.
They can probably get the answer to their question from my answer, though.
Anatomy of virtualenvwrapper
The existing project has two parts:
- the virtualenv, where python and the python libs are installed, and
- the project directory where your code is (that uses the virtualenv).
Virtualenvwrapper adds a third part, the virtualenvwrapper hooks. These are mainly shell functions that are called at certain times in a project or virtualenvs life cycle. They live in a third directory — by default, they are installed to ~/.virtualenvs (at least that was true on my Debian wheezy system). The hooks include postactivate, which we will edit below, and a bunch of others such as premkproject, premkvirtualenv, etc. The following list of keywords gives you the flavour of the hooks: initialize, pre/post, mk/rm, project/virtualenv, activate/deactivate. virtualenvwrapper puts these scripts in $VIRTUALENVWRAPPER_HOOK_DIR, which defaults to $WORKON_HOME.
- all the virtualenvs are in one place ($WORKON_HOME) (defaults to ~/.virtualenvs) Let’s say you have two virtualenvs, one called MYVENV and the other called MYOTHERVENV
- all the project directories are in another place ($PROJECT_HOME).
Let’s say you have a project “is” in directory /home/me/where/my/proj that uses virtualenv MYVENV.
how to use workon to work on your pre-existing code and virtualenv
Here I’m assuming all your virtualenvs are in one place (in my case, they are all in /usr/local/virtualenv).
** edit ~/.virtualenvs/postactivate (+) to have
case $env_name in MYVENV) cd /home/me/where/my/proj/is ;; MYOTHERVENV) cd /home/me/where/my/other/project/is2 ;; esac
for hk in get_env_details initialize postactivate postdeactivate \ postmkproject postmkvirtualenv postrmproject \ postrmvirtualenv preactivate predeactivate premkproject \ premkvirtualenv prermproject prermvirtualenv; do \ ln -s ~/.virtualenvs/$hk /usr/local/pythonenv/$hk; \ done
any time you want to work on your project that uses MYVENV virtualenv
WORKON_HOME=/usr/local/pythonenv workon MYVENV
Of course if all your virtualenvs are indeed in the same place, you can define WORKON_HOME in your .profile and you won’t have to specify it on the command line every time.
10/9: testing https
The problem I’m trying to solve here is to see if my web app is setting the right headers on the response. But, the web app is under https so it’s hard to sniff. Here’s how I did it. Hopefully next time I have to do this, I will find this hint and get going a lot quicker.
In the thing below, pretendhostname is a name that the remote web server uses for the server name, and the local machine has as an alias to 127.0.0.1. Also, I have used ssh port forwarding to send the data to the remote web server.Remote machine
<VirtualHost *:443> ServerName pretendhostname ... </VirtualHost>
~/.ssh/config stanza for some remote network on which the remote web server (10.0.0.26) runs:
Host remote_gateway ... LocalForward 8443 10.0.0.26:443
in /etc/hosts on my local machine:
127.0.0.1 localhost pretendhostname
Type these commands:
$ ssh remote_gateway
then in a different terminal on my local machine:
$ curl -L -b cookie.data -c cookie.data \ -d 'username=theusername&password=thepassword' \ -k -D headers.txt \ https://pretendhostname:8443/login/ $ curl -L -b cookie.data -c cookie.data \ -d 'username=theusername&password=thepassword' \ -k -D headers.txt \ https://pretendhostname:8443/some/path/somefile.data
This will put the headers of the response into headers.txt, where I can inspect them.
More main-stream media coverage on “the cloud”, this time from The Current
“They ask young people what do you think the cloud looks like? They use almost religious terminology to describe it when in fact the internet is just a bunch of data farms on the Colombia river or routers in a room in suburban Ottawa. Everything’s beige.”
Douglas Coupland on Alcatel-Lucent, in a The Current episode produced by Kristin Nelson.
I was pleased to see “the cloud” described in plain terms in this CBC news article:
“Don’t call it ‘the cloud.’ Call it someone else’s computer,” wrote security blogger Graham Cluley in a post last December, saying that mentality will make people more mindful of the security risks.
Hopefully, we can go back to calling data centres “data centres” again, soon.
08/18: list the files in a revision
git ls-tree -r HEAD
I tried searching for how to list the files in a revision in git, and found only how to list the changed files in a revision.
I knew that gitk shows a list of files and directories in any given repo. So I looked in it (it’s a wish script) to find out how it gets the list of files for each revision. Found it around 3/4 of the way down, in
Probably most anything you’d want to do with git, you can find it in gitk.
A friend recently was going to update his Facebook app on his Android tablet. Even he was shocked to see what permissions Facebook was requiring for the upgrade. Facebook was requiring the following new permissions in order to complete the upgrade:
- YOUR MESSAGES
- read your text messages (sms or mms)
- SYSTEM TOOLS
- change network connectivity
- connect and disconnect from wifi
- HARDWARE CONTROLS
- change your audio settings
- YOUR PERSONAL INFORMATION
- add or modify calendar events and send email to guests without owner’s knowledge
- read calendar events plus confidential information
- read your own contact card
This is in addition to the permissions that the Facebook app already has:
- SYSTEM TOOLS
- prevent tablet from sleeping
- toggle sync on and off
- HARDWARE CONTROLS
- record audio
- take pictures and videos
- YOUR PERSONAL INFORMATION
- modify your contacts
- read call log
- read your contact
- write call log
- modify or delete the contents of your USB storage
- YOUR LOCATION
- approximate (network-based) location
- precise (GPS) location
- YOUR ACCOUNTS
- add or remove accounts
- create accounts
- set passwords
- NETWORK COMMUNICATIONS
- full network access
- PHONE CALLS
- read phone status and identity
- the fold (below are “hidden” permissions — it’s an Android thing I guess)
- SYSTEM TOOLS
- install shortcuts
- read sync settings
- send sticky broadcast
- DEVELOPMENT TOOLS
- test access to protected storage
- NETWORK COMMUNICATION
- download files without notification
- receive data from internet
- view wifi connections
- view network connections
- google play billing service
- HARDWARE CONTROLS
- control vibration
- YOUR ACCOUNTS
- find accounts on the device
And people want to keep money and other sensitive info on their Android devices, along with these promiscuous permissions? They wonder why their privacy is being eroded and their identities stolen, when they are giving away the info — and control — themselves?
At this point, even my friend is not keen to upgrade his Facebook app.
If enough people complain this time, we can watch for Facebook to relinquish a couple of the new permissions, then sneak them back in in future upgrades.
01/2: Android backups
It seems there are two ways to back up the android phone that I didn’t know before.
One is to use the “adb backup” command from the android developer kit. You can supply some switches to control what type of stuff gets backed up, even to the point of choosing particular apps. It probably backs up all the data for a given app in a lump, and also probably makes assumptions on where that data is. If the app writer followed Android conventions, you’re probably ok in terms of backing up what you’re interested in.
The other works if you have installed clockworkmod. You can boot to recovery mode, and select “backup and restore”, then “backup”. It will copy your data to the
/sdcard/clockworkmod/backup directory. Copy the backup off, and you have your backup. I’m not yet sure what is in it. Supposedly this method is automatable.
I will have to try the restore for each method at some point I guess — because a backup isn’t complete until you know you can restore from it.
Of course I tried the above two methods in the other order, and so I’m probably backing up my backup. Oops. It’s still running.
… time passes
Ok, seems to have finished. Let’s have a look.
The clockworkmod backup (that copies to
/sdcard/clockworkmod/backup directory) produced the following on the android:
shell@android:/sdcard/clockworkmod/backup $ find . ./2014-01-02.18.21.15 ./2014-01-02.18.21.15/boot.img ./2014-01-02.18.21.15/recovery.img ./2014-01-02.18.21.15/system.ext4.dup ./2014-01-02.18.21.15/data.ext4.dup ./2014-01-02.18.21.15/cache.ext4.dup ./2014-01-02.18.21.15/nandroid.md5
Simple enough to copy that off with an adb pull command:
$ ~/projects/android/android-sdk-linux-r22.3/platform-tools/adb pull /sdcard/clockworkmod/backup/2014-01-02.18.21.15/ . pull: building file list... pull: /sdcard/clockworkmod/backup/2014-01-02.18.21.15/nandroid.md5 -> ./nandroid.md5 pull: /sdcard/clockworkmod/backup/2014-01-02.18.21.15/cache.ext4.dup -> ./cache.ext4.dup pull: /sdcard/clockworkmod/backup/2014-01-02.18.21.15/data.ext4.dup -> ./data.ext4.dup pull: /sdcard/clockworkmod/backup/2014-01-02.18.21.15/system.ext4.dup -> ./system.ext4.dup pull: /sdcard/clockworkmod/backup/2014-01-02.18.21.15/recovery.img -> ./recovery.img pull: /sdcard/clockworkmod/backup/2014-01-02.18.21.15/boot.img -> ./boot.img 6 files pulled. 0 files skipped. 3328 KB/s (21445584 bytes in 6.291s) $
Note the ‘/’ at the end of the path that is being pulled, that’s how you get a directory and its contents.
The .md5 file has md5sums for the other files. The .dup files seem to be lists of paths on the android device. Not sure how they map to the .img files. I’m guessing the .img files are the full flash contents of those two partitions (boot and recovery).
To restore some but not all, you can boot to recovery mode, select backup and restore, then advanced restore. Now you can choose to restore:
boot system data cache sd-ext
I suppose being able to choose these things is better than nothing, but I was hoping for the ability to restore, say, the calendar but leave the other things alone. Guess I have to keep looking.
This produced a single mega-sized file. File says it is of type “data”. It starts with the following string: “ANDROID BACKUP”.
I guess you can back up only a single app (or a few apps) if you give the app (or apps) name(s).
“you can list the package names (e.g. com.google.android.apps.plus) specifically that you would like to backup.”
adb backup -all -apk -shared -system
adb backup -apk -shared -system com.droidwave.offlinecalendar
The adb command directs you to “unlock the phone and enter the password”. It just means to enter your pin or do whatever you do to access your phone normally — not referring to “rooting” your phone here. During the backup, a screen appears on the phone over top of everything else, asking for your password (which you set in settings, developer settings, desktop backup) and permission to do the backup — and showing the progress by giving the name of the app (file? app?) bing handled. Note that I’ve seen two warnings in different articles on the web saying the backup/restore will not work unless the password is set.
android.sharedstorage.backup might be shown for a long time, esp. if there is a big fat backup from the other method sitting in it. Oops.
It takes quite a while — on the order of half an hour or so if you asked for a full backup (-all -apk -shared -system) and there is a clockworkmod backup in
In the end there is a humongous file (nearly 3 GB in my case) in the named place (-f option on the adb backup command line) on your desktop.
I don’t think there is a way to restore just part of it.
Deleting the other backup and doing this backup again resulted in a slightly smaller humongous file:
$ ls -la total 8109684 drwxrwx--- 3 bjb bjb 4096 Jan 3 00:05 . drwxrwx--- 4 bjb bjb 4096 Jan 2 14:11 .. -rw-r----- 1 bjb bjb 2775642349 Jan 2 14:47 20140102 -rw-r----- 1 bjb bjb 2775658565 Jan 2 23:06 20140102-a -rw-r----- 1 bjb bjb 2744876613 Jan 3 00:50 20140102-b drwxrwx--- 2 bjb bjb 4096 Jan 2 19:22 clockworkmod $
After a few trials I have:
$ ls -la total 12469752 drwxrwx--- 3 bjb bjb 4096 Jan 3 02:40 . drwxrwx--- 4 bjb bjb 4096 Jan 2 14:11 .. -rw-r----- 1 bjb bjb 2775642349 Jan 2 14:47 20140102 -rw-r----- 1 bjb bjb 2775658565 Jan 2 23:06 20140102-a -rw-r----- 1 bjb bjb 2744876613 Jan 3 00:50 20140102-b -rw-r----- 1 bjb bjb 1486766101 Jan 3 01:45 20140102-c -rw-r----- 1 bjb bjb 1486766101 Jan 3 02:28 20140102-d -rw-r----- 1 bjb bjb 4213 Jan 3 02:29 20140102-e -rw-r----- 1 bjb bjb 4213 Jan 3 02:36 20140102-f -rw-r----- 1 bjb bjb 4213 Jan 3 02:38 20140102-g -rw-r----- 1 bjb bjb 1486766101 Jan 3 02:46 20140102-h drwxrwx--- 2 bjb bjb 4096 Jan 2 19:22 clockworkmod $
I got an Android phone a while ago. I’m trying very hard not to have to connect to any “cloud” services when I use it. It pretty much makes the phone useless as a PIM and I am actually still using my old Palm.
However the battery on my aging Z22 is holding less and less charge: its days are numbered. So I’m going to have to make the Android phone more functional.
Step one. Backup the apps I do use to my own computer.
Apps I use:
|Messaging||com.android.providers.telephony. Or not. Restoring this did not get the messages back.|
I got K-9 Mail from fDroid, the free-and-open-source-app store for Android.
It seems I will have to backup all this data app by app. I’m making a list of apps I use … and adding to it as I discover more of them that I use.
settings — import/export — Export to storage — saves as a file to
Can be copied off with adb. I don’t think my phone has a separate SD Card for general storage, although it is a separate partition in flash.
I’m already using digikam to copy the photos off.
The photos are saved to
/data/media/DCIM/Camera/ by the camera app.
Edited photos are in
settings — more — export textnotes to SD-Card
There is also a directory
/mnt/sdcard/noteeverything/ /mnt/sdcard/noteeverything/text /mnt/sdcard/noteeverything/paintings /mnt/sdcard/noteeverything/voices /mnt/sdcard/noteeverything/photos /mnt/sdcard/noteeverything/videos /mnt/sdcard/noteeverything/backup
It may be that Note Everything saves to SD Card even without the export step.
Perusing fDroid, I see there are some more choices now for note apps than there were when I first got Note Everything. Will have to revisit this. Although I like the way Note Everything stores data, I’d prefer an open source app.
Not sure yet what info is stored in the different directories.
No need to save — this is just a view on an IMAP folder that I can see elsewhere. Although I suppose it might be nice to save the account info and settings.
Messages are stored in
in some kind of binary DB.
Contents of that directory:
mmssms.db mmssms.db-journal telephony.db telephony.db-journal
I suppose the telephony.db is the list of phone calls I’ve received/initiated. Could be handy to save that too.
Some items in
Some items in
One item in
/mnt/sdcard (!) Perhaps that’s an anomaly. Not sure why it’s in two places. Maybe
/data/media is the same as
/mnt/sdcard. Apart from having (some of?) the same files, I have no other evidence of this.
/mnt/sdcard are owned by user 0.1015 (
/data/media are owned by user 1023.1023 (
Moving along …
I have nothing in here at the moment
That could be backed up I guess
clock info to back up
/data/data/com.android.deskclock/shared_prefs/AlarmClock.xml /data/data/com.android.deskclock/shared_prefs/com.android.deskclock_preferences.xml /data/data/com.android.deskclock/databases/alarms.db /data/data/com.android.deskclock/databases/alarms.db-journal
I wish these apps would have an “about” entry in the settings, I can’t remember what came with Android and what I got afterwards.
Not sure where the phone calls rec’d/initiated are kept.
I probably got this from fdroid also
Very nice app.
Perhaps settings are stored
And the app itself
The android stock settings app, if the settings can be said to be an “app”. And apparently it can.
Apparently a fork of ancal. It needs work — on my platform at least.
- … etc.
06/27: Enough git for julython
I’ll be giving a talk tonight at OPAG called “Enough git for julython”. That’s right, julython is coming up in a few days. I want to help boost the Ottawa, Canada participation by removing a possible barrier to participation: I will be showing people how to use git and github.
The talk is at Shopify tonight, Thursday June 27, at 19:30.