Categories: ,
Posted by: bjb

I was pleased to see “the cloud” described in plain terms in this CBC news article:

“Don’t call it ‘the cloud.’ Call it someone else’s computer,” wrote security blogger Graham Cluley in a post last December, saying that mentality will make people more mindful of the security risks.

Hopefully, we can go back to calling data centres “data centres” again, soon.

Posted by: bjb

git ls-tree -r HEAD

I tried searching for how to list the files in a revision in git, and found only how to list the changed files in a revision.

I knew that gitk shows a list of files and directories in any given repo. So I looked in it (it’s a wish script) to find out how it gets the list of files for each revision. Found it around 3/4 of the way down, in proc gettree.

Probably most anything you’d want to do with git, you can find it in gitk.

Categories: , ,
Posted by: bjb

A friend recently was going to update his Facebook app on his Android tablet. Even he was shocked to see what permissions Facebook was requiring for the upgrade. Facebook was requiring the following new permissions in order to complete the upgrade:

    • read your text messages (sms or mms)
    • change network connectivity
    • connect and disconnect from wifi
    • change your audio settings
    • add or modify calendar events and send email to guests without owner’s knowledge
    • read calendar events plus confidential information
    • read your own contact card

This is in addition to the permissions that the Facebook app already has:

    • prevent tablet from sleeping
    • toggle sync on and off
    • record audio
    • take pictures and videos
    • modify your contacts
    • read call log
    • read your contact
    • write call log
    • modify or delete the contents of your USB storage
    • approximate (network-based) location
    • precise (GPS) location
    • add or remove accounts
    • create accounts
    • set passwords
    • full network access
    • read phone status and identity
  • the fold (below are “hidden” permissions — it’s an Android thing I guess)
    • install shortcuts
    • read sync settings
    • send sticky broadcast
    • test access to protected storage
    • download files without notification
    • receive data from internet
    • view wifi connections
    • view network connections
    • google play billing service
    • control vibration
    • find accounts on the device

And people want to keep money and other sensitive info on their Android devices, along with these promiscuous permissions? They wonder why their privacy is being eroded and their identities stolen, when they are giving away the info — and control — themselves?

At this point, even my friend is not keen to upgrade his Facebook app.

If enough people complain this time, we can watch for Facebook to relinquish a couple of the new permissions, then sneak them back in in future upgrades.

Categories: , ,
Posted by: bjb

It seems there are two ways to back up the android phone that I didn’t know before.

One is to use the “adb backup” command from the android developer kit. You can supply some switches to control what type of stuff gets backed up, even to the point of choosing particular apps. It probably backs up all the data for a given app in a lump, and also probably makes assumptions on where that data is. If the app writer followed Android conventions, you’re probably ok in terms of backing up what you’re interested in.

The other works if you have installed clockworkmod. You can boot to recovery mode, and select “backup and restore”, then “backup”. It will copy your data to the /sdcard/clockworkmod/backup directory. Copy the backup off, and you have your backup. I’m not yet sure what is in it. Supposedly this method is automatable.

I will have to try the restore for each method at some point I guess — because a backup isn’t complete until you know you can restore from it.

Of course I tried the above two methods in the other order, and so I’m probably backing up my backup. Oops. It’s still running.

… time passes

Ok, seems to have finished. Let’s have a look.

clockworkmod backup

The clockworkmod backup (that copies to /sdcard/clockworkmod/backup directory) produced the following on the android:

shell@android:/sdcard/clockworkmod/backup $ find                               

Simple enough to copy that off with an adb pull command:

$ ~/projects/android/android-sdk-linux-r22.3/platform-tools/adb pull /sdcard/clockworkmod/backup/2014-01- .
pull: building file list...
pull: /sdcard/clockworkmod/backup/2014-01- -> ./nandroid.md5
pull: /sdcard/clockworkmod/backup/2014-01- -> ./cache.ext4.dup
pull: /sdcard/clockworkmod/backup/2014-01- -> ./data.ext4.dup
pull: /sdcard/clockworkmod/backup/2014-01- -> ./system.ext4.dup
pull: /sdcard/clockworkmod/backup/2014-01- -> ./recovery.img
pull: /sdcard/clockworkmod/backup/2014-01- -> ./boot.img
6 files pulled. 0 files skipped.
3328 KB/s (21445584 bytes in 6.291s)

Note the ‘/’ at the end of the path that is being pulled, that’s how you get a directory and its contents.

The .md5 file has md5sums for the other files. The .dup files seem to be lists of paths on the android device. Not sure how they map to the .img files. I’m guessing the .img files are the full flash contents of those two partitions (boot and recovery).

To restore some but not all, you can boot to recovery mode, select backup and restore, then advanced restore. Now you can choose to restore:


I suppose being able to choose these things is better than nothing, but I was hoping for the ability to restore, say, the calendar but leave the other things alone. Guess I have to keep looking.

adb backup

This produced a single mega-sized file. File says it is of type “data”. It starts with the following string: “ANDROID BACKUP”.

I guess you can back up only a single app (or a few apps) if you give the app (or apps) name(s).

you can list the package names (e.g. specifically that you would like to backup.”

adb backup -all -apk -shared -system


adb backup -apk -shared -system com.droidwave.offlinecalendar

The adb command directs you to “unlock the phone and enter the password”. It just means to enter your pin or do whatever you do to access your phone normally — not referring to “rooting” your phone here. During the backup, a screen appears on the phone over top of everything else, asking for your password (which you set in settings, developer settings, desktop backup) and permission to do the backup — and showing the progress by giving the name of the app (file? app?) bing handled. Note that I’ve seen two warnings in different articles on the web saying the backup/restore will not work unless the password is set. android.sharedstorage.backup might be shown for a long time, esp. if there is a big fat backup from the other method sitting in it. Oops.

It takes quite a while — on the order of half an hour or so if you asked for a full backup (-all -apk -shared -system) and there is a clockworkmod backup in /sdcard/. Oops.

In the end there is a humongous file (nearly 3 GB in my case) in the named place (-f option on the adb backup command line) on your desktop.

I don’t think there is a way to restore just part of it.

Deleting the other backup and doing this backup again resulted in a slightly smaller humongous file:

$ ls -la
total 8109684
drwxrwx--- 3 bjb bjb       4096 Jan  3 00:05 .
drwxrwx--- 4 bjb bjb       4096 Jan  2 14:11 ..
-rw-r----- 1 bjb bjb 2775642349 Jan  2 14:47 20140102
-rw-r----- 1 bjb bjb 2775658565 Jan  2 23:06 20140102-a
-rw-r----- 1 bjb bjb 2744876613 Jan  3 00:50 20140102-b
drwxrwx--- 2 bjb bjb       4096 Jan  2 19:22 clockworkmod

After a few trials I have:

$ ls -la
total 12469752
drwxrwx--- 3 bjb bjb       4096 Jan  3 02:40 .
drwxrwx--- 4 bjb bjb       4096 Jan  2 14:11 ..
-rw-r----- 1 bjb bjb 2775642349 Jan  2 14:47 20140102
-rw-r----- 1 bjb bjb 2775658565 Jan  2 23:06 20140102-a
-rw-r----- 1 bjb bjb 2744876613 Jan  3 00:50 20140102-b
-rw-r----- 1 bjb bjb 1486766101 Jan  3 01:45 20140102-c
-rw-r----- 1 bjb bjb 1486766101 Jan  3 02:28 20140102-d
-rw-r----- 1 bjb bjb       4213 Jan  3 02:29 20140102-e
-rw-r----- 1 bjb bjb       4213 Jan  3 02:36 20140102-f
-rw-r----- 1 bjb bjb       4213 Jan  3 02:38 20140102-g
-rw-r----- 1 bjb bjb 1486766101 Jan  3 02:46 20140102-h
drwxrwx--- 2 bjb bjb       4096 Jan  2 19:22 clockworkmod
aunknownadb backup -f 20140102-a -all -apk -shared -system
bunknownadb backup -f 20140102-b -all -apk -shared -system (after deleting the extra backup)
cunknownadb backup -f 20140102-c -apk -shared -system
d6m 17sadb backup -f 20140102-d -apk -shared
e0m25.650sadb backup -f 20140102-e -apk
f0m24.420sadb backup -f 20140102-f
g0m19.870sadb backup -f 20140102-g -system
h6m8.702sadb backup -f 20140102-g -shared -system
Categories: , , ,
Posted by: bjb

I got an Android phone a while ago. I’m trying very hard not to have to connect to any “cloud” services when I use it. It pretty much makes the phone useless as a PIM and I am actually still using my old Palm.

However the battery on my aging Z22 is holding less and less charge: its days are numbered. So I’m going to have to make the Android phone more functional.

Step one. Backup the apps I do use to my own computer.

Apps I use:

Note Everything
K-9 Mail com.fsck.k9
Messaging Or not. Restoring this did not get the messages back.
qPDF Viewer com.qoppa.activities.viewer
Apollo com.andrew.apollo
Timer org.dpadgett.timer
Terminal jackpal.androidterm
Offline Calendar com.droidwave.offlinecalendar

I got K-9 Mail from fDroid, the free-and-open-source-app store for Android.

It seems I will have to backup all this data app by app. I’m making a list of apps I use … and adding to it as I discover more of them that I use.


settings — import/export — Export to storage — saves as a file to /mnt/sdcard/nnnnn.vcf

Can be copied off with adb. I don’t think my phone has a separate SD Card for general storage, although it is a separate partition in flash.


I’m already using digikam to copy the photos off.

The photos are saved to /data/media/DCIM/Camera/ by the camera app.

Edited photos are in /data/media/Edited/

Note Everything

settings — more — export textnotes to SD-Card

saved to /mnt/sdcard/noteeverything/text

There is also a directory /mnt/sdcard/noteeverything/backup

The directories:


It may be that Note Everything saves to SD Card even without the export step.

Perusing fDroid, I see there are some more choices now for note apps than there were when I first got Note Everything. Will have to revisit this. Although I like the way Note Everything stores data, I’d prefer an open source app.

See also

  • /data/data/

Not sure yet what info is stored in the different directories.

K-9 Mail

No need to save — this is just a view on an IMAP folder that I can see elsewhere. Although I suppose it might be nice to save the account info and settings.

  • /data/data/com.fsck.k9


Messages are stored in


in some kind of binary DB.

Contents of that directory:


I suppose the telephony.db is the list of phone calls I’ve received/initiated. Could be handy to save that too.

qPDF Viewer

Some items in


Some items in


One item in /data/media and /mnt/sdcard (!) Perhaps that’s an anomaly. Not sure why it’s in two places. Maybe /data/media is the same as /mnt/sdcard. Apart from having (some of?) the same files, I have no other evidence of this.

Items in /mnt/sdcard are owned by user 0.1015 (root.sdcard_rw)

Items in /data/media are owned by user 1023.1023 (media_rw.media_rw)

Moving along …


I have nothing in here at the moment

However …


That could be backed up I guess


clock info to back up


I wish these apps would have an “about” entry in the settings, I can’t remember what came with Android and what I got afterwards.


Phone preferences


See also


Not sure where the phone calls rec’d/initiated are kept.


I probably got this from fdroid also


Very nice app.


Perhaps settings are stored


And the app itself



The android stock settings app, if the settings can be said to be an “app”. And apparently it can.

Offline Calendar

Apparently a fork of ancal. It needs work — on my platform at least.


for investigation

  • /data/data/
  • /data/data/
  • /data/data/
  • /data/data/
  • /data/data/
  • /data/data/
  • /data/data/
  • /data/data/
  • /data/data/
  • /data/data/
  • /data/data/
  • /data/data/
  • /data/data/
  • /data/data/
  • /data/data/
  • /data/data/
  • /data/data/
  • /data/data/org.fdroid.fdroid
  • /system/app/Apollo.apk
  • /system/app/Calculator.apk
  • /system/app/Camera.apk
  • /system/app/Calendar.apk
  • /system/app/CalendarProvider.apk
  • /system/app/SuperUser.apk
  • … etc.
Categories: , , ,
Posted by: bjb

I’ll be giving a talk tonight at OPAG called “Enough git for julython”. That’s right, julython is coming up in a few days. I want to help boost the Ottawa, Canada participation by removing a possible barrier to participation: I will be showing people how to use git and github.

Julython is basically an informal contest that encourages people to write some python code and check it in publicly into github or bitbucket.

The talk is at Shopify tonight, Thursday June 27, at 19:30.

06/23: prism-break

Categories: ,
Posted by: bjb

A site that promotes alternatives to the software and cloud services that the US government (and others) uses as its own databases for mining.

Categories: , ,
Posted by: bjb

The EFF is raising funds to pay for a challenge to Personal Audio’s patent, that they are using to squeeze podcasters. Now might be a good time to contribute, or even join and make regular contributions.

It looks like they’ve already raised their goal, but it doesn’t hurt to support or join the EFF in this and other causes.

Also, in order to make their case that the patent is baseless, they have issued a call for prior art. If you can contribute information to their case, that will also help to win this case.

Categories: ,
Posted by: bjb

One of the nice things about the Ottawa Python Author’s Group irc channel (, #opag) is that they occasionally mention a great but under-advertized reference, like this one for setuptools:

Thanks Ian!

Categories: ,
Posted by: bjb

I needed to install awstats into an existing web installation recently, and finding the info needed for that was a bit annoying. The documentation I could find gets into the nitty gritty without giving you the big picture.

So here is the big picture for awstats. Because it is meant to be a “big picture”, I’m putting the configuration discusson last. I want to cover the overall view of how the system works before getting into configuration specifics.

Overview for awstats

awstats is a script for analyzing web server logs (it has been extended to analyze other types of logs like mail logs). It analyzes the logs, and stores the statistics, and you can see the results as graphs and charts on a web page. It is a venerable old tool (meaning it doesn’t quite fit into modern ways of handling log files, init scripts, script parameters or whatever), and also is designed to be lean so it can analyze quite large logfiles without bogging down the whole system (so the parser for the log lines is a bit simple and can get confused — this just means that line is thrown away but the rest of the file does get processed). is a perl script. On my Debian system it got installed into /usr/lib/cgi-bin/ It can run as a cgi-bin script, but doesn’t have to.

After configuring, you use it in two stages:

  1. analyze the web server logs
  2. generate the results page.


In stage 1, you run -update on the log file. This will produce a bunch of .txt files. There will be a .txt file for each time period (usually a month, but could be a year). There will generally be a .txt file series for each set of logfiles for a domain or virtualhost. If one log file spans two calendar months (say, covers Jan 28 — Feb 3), then it will produce two .txt files — one for January and one for February. When you process the next logfile (that might span Feb 3 — Feb 10), then no new .txt files will be created but the existing one for February will be updated.

Generally, the documentation assumes you will not be trying to “catch up” with your old log files. If you want to run your old log files through awstats, you will need to analyze them in chronological order, as is meant to run on the same logfiles over and over, and only process the new items since last session. It does this by storing a date and comparing each log record to the date to find out if it is old or new. I wrote a script that processes all the old log files in order (

Also, as far as I know, awstats doesn’t understand compressed files, so you will have to uncompress the logfiles before analyzing them. My script handles that too, but for that it needs write permission in the logfile directory.

The “chronological ordering” requirement implies that all the things that log to that log file better agree on the time. If one app is logging in local time (say -0500) and another is logging in UTC time, then generally only the records that are 5 hours later will be picked up by The other records will be regarded as “corrupted” and ignored.

You can run this stage as a cgi script — but it can also be run by a cron job. Running it as a cron job means you don’t have to give your web server user permission to write to its DocumentRoot. Running it as a cgi script means you can see the very latest statistics (right up to the moment before you run the update) — but if you don’t do it often enough, you may miss analyzing some of the web server logs (eg, if they get rotated before you run on them). If that happens you have the relatively painful task of trying to fix the mess, or just abandoning stats for those months. You could run it as a cron job and still allow web users to run it as well, to avoid losing info when logs are rotated.

Stage 2

Once you have the web server logs digested into statistics in .txt files, then you can view the results. There are two ways to view the results:

  1. dynamically, via a cgi script
  2. statically, as pre-generated static html pages

To see the results dynamically, you need to configure your web server to call the cgi script.

To see the results statically, you need to make a place for the generated html, and then call -output for each report you might want to have available. There are quite a few reports, and you need to do it for each time period as well. awstats supplies a script (in my Debian system it went here:
/usr/share/awstats/tools/ that will generate all the reports for a given time period (i.e. month) so you just have to loop over the months. And virtualhosts, if you’re doing it for more than one web server/domain name.

Configuration Considerations

There are two things to configure with awstats: one is awstats itself (a config file for each “web site”) and one is the web server that you will use to view the results (if that is how you are going to view the results). Below, I discuss only configuration of awstats itself.

The script is configured with files in /etc/awstats/awstats.domainname.conf (again, this is for my Debian system). You would copy the awstats example conf file to a file with your domain name in the middle, eg:

cp /etc/awstats/awstats.conf /etc/awstats/

And then edit the file to have the configuration you want.

awstats works best if you have a separate series of web server logfiles for each host for which you want graphs. If you have some virtualhosts, you might want to configure them each to have their own log files.

On my Debian system using apache2 for a web server, all the log files go into the same directory /var/log/apache2. The script can handle this — and it would be easy to make a set of cron commands that will each update a different virtual host. At the moment, I have all the stats files and static html files going into one directory — one for stats, one for all the static html files. Maybe I should have a directory per virtualhost for the html files, though — they are getting quite numerous. A directory per virtualhost means you can more easily apply different access policies to the different domains.

The things I changed in the awstats.conf file for my purposes were:


There are lots of other options, but customizing those was enough to get some charts to start wtih.

LogFile is used if you don’t specify -LogFile on the command line. The catchup script uses the -LogFile argument on the command line, but the cron jobs that keep the stats updated can probably use the most recent logfile name domain-access.log.

LogFormat — it’s important to match the LogFormat to the actual format that your logfiles are written in, or every line will be classified as corrupted. I used format 1 for my apache2 logfiles. awstats has 4 predefined log formats, or you can specify a custom log format in exquisite detail field by field.

SiteDomain is the name of your site as your web server knows it

HostAliases is a list of other names for “self” for the web server (for the domain being analyzed).

DirData is the directory where the statistical output will go (all the .txt files).

The web sites I administer have hired a service to monitor themselves, and I added those user agents to the robots file (/usr/share/awstats/lib/ in order to count them. Adding them to SkipHosts just meant they weren’t counted and didn’t show up in the stats at all.

Last words

Hopefully that will give you an idea of what you’re aiming for as you follow the other, more detailed explanations of how to set up awstats. Remember, when you lose the data in a logfile and have to leave it behind — it’s only stats. You’ll manage without them. The stats are approximate anyway — very little attempt is made in the program to give an exact account of the activity. Records are thrown away and not counted almost every time awstats is run — so don’t sweat it if you lose a log file or two on the way. Once the cron jobs are set up and time passes, you’ll get fairly good coverage of the activity of the web server. Keep tuning your web app (eg, ensure that times logged use the same timezone across all apps), look at the config options for awstats and tune your .conf files for more interesting reports, and eventually you’ll have a great resource for security monitoring, marketing analysis and for web site usability and effectiveness reviews.

In fact, you probably can just set up awstats and let it accumulate statistics over time — don’t bother with the catchup script. I did it because I did have old logs, and I wanted to see what a year’s worth of web log statistics looked like in awstats and other packages. It did help me with choosing a web log analysis package, and in choosing among the various extra options for awstats not discussed above — but it was also time-consuming.