Categories: , , , , ,
Posted by: bjb

Yesterday I upgraded LineageOS from 14.1 to 15.1, and TWRP from 3.1.1.0 to 3.2.1.0. It was a lot harder than it should have been, because I was missing a few details on how it all works.

  1. The phone has a few “modes” -

    • booted to the regular OS (system mode)

    • booted to recovery — this is like an alternate OS (think dual boot) on the phone rather than a boot loader or single user mode

    • booted to fastboot — this is like stopping the boot in the bootloader, before the bootloader proceeds into an OS.

  2. The phone has more than one vendor/product identifier for USB. It uses one for system and recovery modes and one for fastboot mode. As far as I can tell, you never see both at once. The udev rules (
    /etc/udev/rules.d/51-android.rules,
    in my case) had to be updated for both. (Extra note to self:


    udevadm control -R

    to make udevd reload rules.)

  3. There are a lot of flash partitions. I really don’t know what all of them are for, but one interesting one is the vendor partition that needs a “vendor.img” to go in it. You can get a “vendor.img” from a stock image (say from Google) by unpacking the vendor update. It will be called “vendor.img” all ready for flashing into the vendor partition. When I first booted into LineageOS 15.1 there was a message


    A vendor image mismatch has been
    detected. Typically this means your
    vendor image is out of date. Please
    ensure your vendor image matches OPM6.171019.030.B1.

    Later boots only showed:


    SYSTEM UI HAS STOPPED

    And that big fat message was plunked right in the middle of the screen, obscuring the <power on> and <power off> icons whenever I tried to reboot … a tad inconvenient.

    Eventually I remembered the earlier message (fortunately had noted it down), flashed the appropriate vendor.img (I was able to do that in a separate step, flashing just the vendor partition), and LineageOS booted properly. Phew.

    In hindsight, I could probably have added the vendor image into the LineageOS update (see next point on how that might work).

  4. One more problem I ran into: with all the mistakes I was making, my phone lost its “device” property. That meant that attempts to install LineageOS were failing with an error (yet another one …):


    Starting ADB sidelaod feature...
    Installing zip file '/sideload/package.zip'
    Warning: No file_contexts
    E3004: This package is for device: bullhead; this device is .
    Updater process ended with ERROR: 7

    The device type check is made in an install script that is packaged into the upgrade package. The upgrade package is delivered as a .zip file, which is really a “java archive file” (according to the “file” command). It contains a few files in a directory hierarchy, including


    META-INF/com/google/android/updater-script

    Edit this file to remove the first line, which contains the failing check.


    getprop("ro.product.device") == "N5100" || abort("This package is for \" N5100\" devices; this is a \"" + getprop("ro.product.device") + "\".");

    This is an example I got off the web, not the line I actually deleted — mine had “bullhead” where it says “N5100” above.

    Obviously, you want to be very sure you have the right package before you flash it. This is also an example of why it is better to install the prepared and released packages from LineageOS or Google or the phone vendor, than to flash things straight into the flash partitions with dd — they have these safeguards against shooting yourself in the foot.

    Of course, once the package was unpacked and that file edited, it had to put it back together.

    Take the package apart

    Open the ROM zip file with winrar. (I used unzip.)


    mkdir work
    cd work
    cp ../lineage-15.1-20180618-nightly-bullhead-signed.zip .
    unzip lineage-15.1-20180618-nightly-bullhead-signed.zip

    This produced the following directory structure:


    ./system.patch.dat
    ./boot.img
    ./file_contexts.bin
    ./system/build.prop
    ./system.new.dat.br
    ./system.transfer.list
    ./install/bin/backuptool.sh
    ./install/bin/otasigcheck.sh
    ./install/bin/backuptool.functions
    ./lineage-15.1-20180618-nightly-bullhead-signed.zip
    ./META-INF/com/google/android/updater-script
    ./META-INF/com/google/android/update-binary
    ./META-INF/com/android/metadata
    ./META-INF/com/android/otacert
    ./META-INF/org/lineageos/releasekey

     

    Edit the updater-script

    I edited META-INF/com/google/android/updater-script with a plain-text editor and removed the first line.

    Put the package back together

    Then to put it back together again, from directory containing META_INF:


    rm lineage-15.1-20180618-nightly-bullhead-signed.zip # I could do this because it was a copy, still had original in directory above)
    zip -rZ store lineage-15.1-20180618-nightly-bullhead-signed.zip * # I deleted lineage because I wanted to use * here

    My first attempt omitted the -r, and failed to apply (since all the subdirs in the .zip file were empty). Sigh.

    One last note: the file utility reported the type of the package before being taken apart as a java archive file. After reassembly, file said it was a zip archive. It seemed to work anyway.

    Hmm. I seem to remember unpacking that second zip-file … and it was full of .img files as well as the updater script. I’ll just leave this here as a note for next time. Also, maybe if I had zipped back up in two stages, file might have recognized the final package as a jar file. To be continued : -)

Extra research

In all my research to fix the problems I had made, I found a couple of other potential ways to get stuff into flashes. So I never got to use these, but they looked useful. I record them here for future reference.

  1. You can convert a .img into a .zip for purposes of flashing, by renaming the .img file after the flash partition into which it should be flashed (eg, recovery.img) and zipping it by itself with NO compression — (use the “store” compression method). This is handy if you have an img file to install but only the sideload method will currently work for you (due to various mistakes you may or may not have made up to this point, eliminating all the other methods that might work, and/or misunderstandings about fastboot that make you think it won’t work for you).


    zip -Z store newzipfilename.zip recovery.img
  2. Figure out the flash partition device, and dd the .img file into the flash partition. eg:


    dd if=/sdcard/recovery.img of=/dev/block/platform/soc.0/f9824900.sdhci/by-name/recovery

    Obviously you have to look at your own /dev/block/platform directory to get the exact path. Mine contains (in system mode):


    bullhead:/ # ls /dev/block/platform/soc.0/f9824900.sdhci/by-name
    DDR cmnlib fsg keymasterbak modem persistent sbl1 tz
    aboot cmnlibbak grow keystore modemst1 pmic sbl1bak tzbak
    abootbak config hyp laf modemst2 pmicbak sdi userdata
    apdp devinfo hypbak limits msadp recovery sec vendor
    boot dpo imgdata metadata oem rpm ssd
    cache fsc keymaster misc persist rpmbak system

    (Well sorry about the bad formatting … one day I will revisit and format this properly.)

  3. Use TWRP’s flash_image utility from TWRP’s shell. E.g.


    flash_image misc /mnt/sdcard/_update/misc.img

For these last two, you need to put the .img file onto the phone (perhaps using adb) and then run the appropriate command on the phone, perhaps in the shell supplied by TWRP. Be careful with these — it is probably best to flash recovery while booted to system, and to flash system partitions while booted to recovery. Not least because you know you have something you can still boot to, if the flash operation goes wrong. This is why I did not continue to try upgrading the TWRP partition after the phone would not boot to system any more. Once system was working again I was able to flash TWRP (with all my newfound knowledge above) and it was super quick and successful.

Final notes

So I hope I will find this handy, next time I need to upgrade the OS, the recovery, and/or just install new software on or transfer files to/from the phone.

One last note, the xda site seems to be a fountain of knowledge on working with android devices of all kinds (hardware and os-distro), even if you do have to read a lot of individual problem reports and solution attempts to get those nuggets. This page was especially useful: xda beginners how-to guides.

Well that was my weekend … at least I got a beefy blog article out of it : -)

Categories: , ,
Posted by: bjb

A friend recently was going to update his Facebook app on his Android tablet. Even he was shocked to see what permissions Facebook was requiring for the upgrade. Facebook was requiring the following new permissions in order to complete the upgrade:

  • YOUR MESSAGES
    • read your text messages (sms or mms)
  • SYSTEM TOOLS
    • change network connectivity
    • connect and disconnect from wifi
  • HARDWARE CONTROLS
    • change your audio settings
  • YOUR PERSONAL INFORMATION
    • add or modify calendar events and send email to guests without owner’s knowledge
    • read calendar events plus confidential information
    • read your own contact card

This is in addition to the permissions that the Facebook app already has:

  • SYSTEM TOOLS
    • prevent tablet from sleeping
    • toggle sync on and off
  • HARDWARE CONTROLS
    • record audio
    • take pictures and videos
  • YOUR PERSONAL INFORMATION
    • modify your contacts
    • read call log
    • read your contact
    • write call log
  • STORAGE
    • modify or delete the contents of your USB storage
  • YOUR LOCATION
    • approximate (network-based) location
    • precise (GPS) location
  • YOUR ACCOUNTS
    • add or remove accounts
    • create accounts
    • set passwords
  • NETWORK COMMUNICATIONS
    • full network access
  • PHONE CALLS
    • read phone status and identity
  • the fold (below are “hidden” permissions — it’s an Android thing I guess)
  • DEFAULT
    • com.sec.android.provider.badge.permission.READ
    • com.sec.android.provider.badge.permission.WRITE
  • SYSTEM TOOLS
    • install shortcuts
    • read sync settings
    • send sticky broadcast
  • DEVELOPMENT TOOLS
    • test access to protected storage
  • NETWORK COMMUNICATION
    • download files without notification
    • receive data from internet
    • view wifi connections
    • view network connections
    • google play billing service
  • HARDWARE CONTROLS
    • control vibration
  • YOUR ACCOUNTS
    • find accounts on the device

And people want to keep money and other sensitive info on their Android devices, along with these promiscuous permissions? They wonder why their privacy is being eroded and their identities stolen, when they are giving away the info — and control — themselves?

At this point, even my friend is not keen to upgrade his Facebook app.

If enough people complain this time, we can watch for Facebook to relinquish a couple of the new permissions, then sneak them back in in future upgrades.

Categories: , ,
Posted by: bjb

It seems there are two ways to back up the android phone that I didn’t know before.

One is to use the “adb backup” command from the android developer kit. You can supply some switches to control what type of stuff gets backed up, even to the point of choosing particular apps. It probably backs up all the data for a given app in a lump, and also probably makes assumptions on where that data is. If the app writer followed Android conventions, you’re probably ok in terms of backing up what you’re interested in.

The other works if you have installed clockworkmod. You can boot to recovery mode, and select “backup and restore”, then “backup”. It will copy your data to the /sdcard/clockworkmod/backup directory. Copy the backup off, and you have your backup. I’m not yet sure what is in it. Supposedly this method is automatable.

I will have to try the restore for each method at some point I guess — because a backup isn’t complete until you know you can restore from it.

Of course I tried the above two methods in the other order, and so I’m probably backing up my backup. Oops. It’s still running.

… time passes

Ok, seems to have finished. Let’s have a look.

clockworkmod backup

The clockworkmod backup (that copies to /sdcard/clockworkmod/backup directory) produced the following on the android:

shell@android:/sdcard/clockworkmod/backup $ find                               
.
./2014-01-02.18.21.15
./2014-01-02.18.21.15/boot.img
./2014-01-02.18.21.15/recovery.img
./2014-01-02.18.21.15/system.ext4.dup
./2014-01-02.18.21.15/data.ext4.dup
./2014-01-02.18.21.15/cache.ext4.dup
./2014-01-02.18.21.15/nandroid.md5

Simple enough to copy that off with an adb pull command:

$ ~/projects/android/android-sdk-linux-r22.3/platform-tools/adb pull /sdcard/clockworkmod/backup/2014-01-02.18.21.15/ .
pull: building file list...
pull: /sdcard/clockworkmod/backup/2014-01-02.18.21.15/nandroid.md5 -> ./nandroid.md5
pull: /sdcard/clockworkmod/backup/2014-01-02.18.21.15/cache.ext4.dup -> ./cache.ext4.dup
pull: /sdcard/clockworkmod/backup/2014-01-02.18.21.15/data.ext4.dup -> ./data.ext4.dup
pull: /sdcard/clockworkmod/backup/2014-01-02.18.21.15/system.ext4.dup -> ./system.ext4.dup
pull: /sdcard/clockworkmod/backup/2014-01-02.18.21.15/recovery.img -> ./recovery.img
pull: /sdcard/clockworkmod/backup/2014-01-02.18.21.15/boot.img -> ./boot.img
6 files pulled. 0 files skipped.
3328 KB/s (21445584 bytes in 6.291s)
$

Note the ‘/’ at the end of the path that is being pulled, that’s how you get a directory and its contents.

The .md5 file has md5sums for the other files. The .dup files seem to be lists of paths on the android device. Not sure how they map to the .img files. I’m guessing the .img files are the full flash contents of those two partitions (boot and recovery).

To restore some but not all, you can boot to recovery mode, select backup and restore, then advanced restore. Now you can choose to restore:

boot
system
data
cache
sd-ext

I suppose being able to choose these things is better than nothing, but I was hoping for the ability to restore, say, the calendar but leave the other things alone. Guess I have to keep looking.

adb backup

This produced a single mega-sized file. File says it is of type “data”. It starts with the following string: “ANDROID BACKUP”.

I guess you can back up only a single app (or a few apps) if you give the app (or apps) name(s).

you can list the package names (e.g. com.google.android.apps.plus) specifically that you would like to backup.”

adb backup -all -apk -shared -system

or

adb backup -apk -shared -system com.droidwave.offlinecalendar

The adb command directs you to “unlock the phone and enter the password”. It just means to enter your pin or do whatever you do to access your phone normally — not referring to “rooting” your phone here. During the backup, a screen appears on the phone over top of everything else, asking for your password (which you set in settings, developer settings, desktop backup) and permission to do the backup — and showing the progress by giving the name of the app (file? app?) bing handled. Note that I’ve seen two warnings in different articles on the web saying the backup/restore will not work unless the password is set. android.sharedstorage.backup might be shown for a long time, esp. if there is a big fat backup from the other method sitting in it. Oops.

It takes quite a while — on the order of half an hour or so if you asked for a full backup (-all -apk -shared -system) and there is a clockworkmod backup in /sdcard/. Oops.

In the end there is a humongous file (nearly 3 GB in my case) in the named place (-f option on the adb backup command line) on your desktop.

I don’t think there is a way to restore just part of it.

Deleting the other backup and doing this backup again resulted in a slightly smaller humongous file:

$ ls -la
total 8109684
drwxrwx--- 3 bjb bjb       4096 Jan  3 00:05 .
drwxrwx--- 4 bjb bjb       4096 Jan  2 14:11 ..
-rw-r----- 1 bjb bjb 2775642349 Jan  2 14:47 20140102
-rw-r----- 1 bjb bjb 2775658565 Jan  2 23:06 20140102-a
-rw-r----- 1 bjb bjb 2744876613 Jan  3 00:50 20140102-b
drwxrwx--- 2 bjb bjb       4096 Jan  2 19:22 clockworkmod
$ 

After a few trials I have:

$ ls -la
total 12469752
drwxrwx--- 3 bjb bjb       4096 Jan  3 02:40 .
drwxrwx--- 4 bjb bjb       4096 Jan  2 14:11 ..
-rw-r----- 1 bjb bjb 2775642349 Jan  2 14:47 20140102
-rw-r----- 1 bjb bjb 2775658565 Jan  2 23:06 20140102-a
-rw-r----- 1 bjb bjb 2744876613 Jan  3 00:50 20140102-b
-rw-r----- 1 bjb bjb 1486766101 Jan  3 01:45 20140102-c
-rw-r----- 1 bjb bjb 1486766101 Jan  3 02:28 20140102-d
-rw-r----- 1 bjb bjb       4213 Jan  3 02:29 20140102-e
-rw-r----- 1 bjb bjb       4213 Jan  3 02:36 20140102-f
-rw-r----- 1 bjb bjb       4213 Jan  3 02:38 20140102-g
-rw-r----- 1 bjb bjb 1486766101 Jan  3 02:46 20140102-h
drwxrwx--- 2 bjb bjb       4096 Jan  2 19:22 clockworkmod
$ 
aunknownadb backup -f 20140102-a -all -apk -shared -system
bunknownadb backup -f 20140102-b -all -apk -shared -system (after deleting the extra backup)
cunknownadb backup -f 20140102-c -apk -shared -system com.android.contacts com.android.providers.settings
d6m 17sadb backup -f 20140102-d -apk -shared com.android.contacts com.android.providers.settings
e0m25.650sadb backup -f 20140102-e -apk com.android.contacts com.android.providers.settings
f0m24.420sadb backup -f 20140102-f com.android.contacts com.android.providers.settings
g0m19.870sadb backup -f 20140102-g -system com.android.contacts com.android.providers.settings
h6m8.702sadb backup -f 20140102-g -shared -system com.android.contacts com.android.providers.settings
Categories: , , ,
Posted by: bjb

I got an Android phone a while ago. I’m trying very hard not to have to connect to any “cloud” services when I use it. It pretty much makes the phone useless as a PIM and I am actually still using my old Palm.

However the battery on my aging Z22 is holding less and less charge: its days are numbered. So I’m going to have to make the Android phone more functional.

Step one. Backup the apps I do use to my own computer.

Apps I use:

Contacts com.android.contacts
Gallery com.google.android.gallery3d
Note Everything de.softxperience.android.noteeverything
K-9 Mail com.fsck.k9
Messaging com.android.providers.telephony. Or not. Restoring this did not get the messages back.
qPDF Viewer com.qoppa.activities.viewer
Apollo com.andrew.apollo
Clock com.android.deskclock
Phone com.android.providers.telephony
Timer org.dpadgett.timer
Terminal jackpal.androidterm
Settings com.android.providers.settings
Offline Calendar com.droidwave.offlinecalendar

I got K-9 Mail from fDroid, the free-and-open-source-app store for Android.

It seems I will have to backup all this data app by app. I’m making a list of apps I use … and adding to it as I discover more of them that I use.

Contacts

settings — import/export — Export to storage — saves as a file to /mnt/sdcard/nnnnn.vcf

Can be copied off with adb. I don’t think my phone has a separate SD Card for general storage, although it is a separate partition in flash.

Gallery

I’m already using digikam to copy the photos off.

The photos are saved to /data/media/DCIM/Camera/ by the camera app.

Edited photos are in /data/media/Edited/

Note Everything

settings — more — export textnotes to SD-Card

saved to /mnt/sdcard/noteeverything/text

There is also a directory /mnt/sdcard/noteeverything/backup

The directories:

/mnt/sdcard/noteeverything/
/mnt/sdcard/noteeverything/text
/mnt/sdcard/noteeverything/paintings
/mnt/sdcard/noteeverything/voices
/mnt/sdcard/noteeverything/photos
/mnt/sdcard/noteeverything/videos
/mnt/sdcard/noteeverything/backup

It may be that Note Everything saves to SD Card even without the export step.

Perusing fDroid, I see there are some more choices now for note apps than there were when I first got Note Everything. Will have to revisit this. Although I like the way Note Everything stores data, I’d prefer an open source app.

See also

  • /data/data/de.softexperience.android.noteeverything

Not sure yet what info is stored in the different directories.

K-9 Mail

No need to save — this is just a view on an IMAP folder that I can see elsewhere. Although I suppose it might be nice to save the account info and settings.

  • /data/data/com.fsck.k9

Messaging

Messages are stored in

/data/data/com.android.providers.telephony/databases/mmssms.db

in some kind of binary DB.

Contents of that directory:

mmssms.db
mmssms.db-journal
telephony.db
telephony.db-journal

I suppose the telephony.db is the list of phone calls I’ve received/initiated. Could be handy to save that too.

qPDF Viewer

Some items in

/mnt/sdcard/Download/

Some items in

/data/media/Download

One item in /data/media and /mnt/sdcard (!) Perhaps that’s an anomaly. Not sure why it’s in two places. Maybe /data/media is the same as /mnt/sdcard. Apart from having (some of?) the same files, I have no other evidence of this.

Items in /mnt/sdcard are owned by user 0.1015 (root.sdcard_rw)

Items in /data/media are owned by user 1023.1023 (media_rw.media_rw)

Moving along …

Apollo

I have nothing in here at the moment

However …

/data/data/com.andrew.apollo
/system/app/Apollo.apk

That could be backed up I guess

Clock

clock info to back up

/data/data/com.android.deskclock
/data/data/com.android.deskclock/shared_prefs/AlarmClock.xml
/data/data/com.android.deskclock/shared_prefs/com.android.deskclock_preferences.xml
/data/data/com.android.deskclock/databases/alarms.db
/data/data/com.android.deskclock/databases/alarms.db-journal

I wish these apps would have an “about” entry in the settings, I can’t remember what came with Android and what I got afterwards.

Phone

Phone preferences

/data/data/com.android.phone/shared_prefs/
/data/data/com.android.phone/databases/

See also

/data/data/com.android.providers.telephony/

Not sure where the phone calls rec’d/initiated are kept.

Timer

I probably got this from fdroid also

/data/data/org.dpadgett.timer

Very nice app.

Terminal

Perhaps settings are stored

/data/data/jackpal.androidterm

And the app itself

/system/app/Term.apk

Settings

The android stock settings app, if the settings can be said to be an “app”. And apparently it can.

Offline Calendar

Apparently a fork of ancal. It needs work — on my platform at least.

Other

for investigation

  • /data/data/com.android/backupconfirm
  • /data/data/com.android.camera
  • /data/data/com.android.certinstaller
  • /data/data/com.android.magicsmoke
  • /data/data/com.android.mms
  • /data/data/com.android.packaginstaller
  • /data/data/com.android.providers.applications
  • /data/data/com.android.providers.calendar
  • /data/data/com.android.providers.contacts
  • /data/data/com.android.providers.downloads
  • /data/data/com.android.providers.downloads.ui
  • /data/data/com.android.providers.drm
  • /data/data/com.android.providers.media
  • /data/data/com.android.providers.settings
  • /data/data/com.android.providers.telephony
  • /data/data/com.android.providers.userdictionary
  • /data/data/com.android.soundrecorder
  • /data/data/org.fdroid.fdroid
  • /system/app/Apollo.apk
  • /system/app/Calculator.apk
  • /system/app/Camera.apk
  • /system/app/Calendar.apk
  • /system/app/CalendarProvider.apk
  • /system/app/SuperUser.apk
  • … etc.
Categories: , ,
Posted by: bjb

I was learning about repo, and noted that repo can use a “list of projects” to narrow down the scope of its actions. It will operate on the list of projects you give it, and not the others. But, as a newish member of the team, I didn’t know what the list of projects was. I knew there was a manifest file. It is in xml format.

I was never very successful with xml. However, I thought that time has passed, and maybe there is a newer easier to use tool out there. I looked again and found xmlstarlet. Maybe my previous attempts at xml have softened me up so I can understand xmlstarlet, or maybe it really is easier than previous xml tools.

I wanted an xmlgrep:

xmlstarlet sel -t -m "//project" -v "@path" -n default.xml
 continue reading
Categories: , ,
Posted by: bjb
My co-worker encouraged me to post this ….

Our nightly builds started failing a while ago, and we wanted to know why. Nothing was immediately obvious so, as a new member to the team, I wanted to see what was different between “day that it worked” and “day that it failed”. Or rather, as the builds fixed themselves after a few days, what was the difference between “day that it failed” and “day that it worked”.

Well the android build is made up of a bunch of subprojects, each with its own git repository. I wasn’t going to be cd into each directory and running “git log” to see if something changed there lately. So I learned about the “forall” subcommand in repo:
repo forall -c "pwd; git log | grep '^[dD]ate' | head -1" > /tmp/repo-last-update-16 
and again in the other build directory.

A little massaging of the outputs allowed me to compare them with diff and find a differing project.