linuxbutler

Yesterday I upgraded LineageOS from 14.1 to 15.1, and TWRP from 3.1.1.0 to 3.2.1.0. It was a lot harder than it should have been, because I was missing a few details on how it all works.

1. The phone has a few “modes” -

   • booted to the regular OS (system mode)

   • booted to recovery — this is like an alternate OS (think dual boot) on the phone rather than a boot loader or single user mode

   • booted to fastboot — this is like stopping the boot in the bootloader, before the bootloader proceeds into an OS.

2. The phone has more than one vendor/product identifier for USB. It uses one for system and recovery modes and one for fastboot mode. As far as I can tell, you never see both at once. The udev rules (
/etc/udev/rules.d/51-android.rules,
in my case) had to be updated for both. (Extra note to self:

   
udevadm control -R


   to make udevd reload rules.)

3. There are a lot of flash partitions. I really don’t know what all of them are for, but one interesting one is the vendor partition that needs a “vendor.img” to go in it. You can get a “vendor.img” from a stock image (say from Google) by unpacking the vendor update. It will be called “vendor.img” all ready for flashing into the vendor partition. When I first booted into LineageOS 15.1 there was a message

   
A vendor image mismatch has been
detected. Typically this means your
vendor image is out of date. Please
ensure your vendor image matches OPM6.171019.030.B1.


   Later boots only showed:

   
SYSTEM UI HAS STOPPED


   And that big fat message was plunked right in the middle of the screen, obscuring the <power on> and <power off> icons whenever I tried to reboot … a tad inconvenient.

   Eventually I remembered the earlier message (fortunately had noted it down), flashed the appropriate vendor.img (I was able to do that in a separate step, flashing just the vendor partition), and LineageOS booted properly. Phew.

   In hindsight, I could probably have added the vendor image into the LineageOS update (see next point on how that might work).

4. One more problem I ran into: with all the mistakes I was making, my phone lost its “device” property. That meant that attempts to install LineageOS were failing with an error (yet another one …):

   
Starting ADB sidelaod feature...
Installing zip file '/sideload/package.zip'
Warning: No file_contexts
E3004: This package is for device: bullhead; this device is .
Updater process ended with ERROR: 7


   The device type check is made in an install script that is packaged into the upgrade package. The upgrade package is delivered as a .zip file, which is really a “java archive file” (according to the “file” command). It contains a few files in a directory hierarchy, including

   
META-INF/com/google/android/updater-script


   Edit this file to remove the first line, which contains the failing check.

   
getprop("ro.product.device") == "N5100" || abort("This package is for \" N5100\" devices; this is a \"" + getprop("ro.product.device") + "\".");


   This is an example I got off the web, not the line I actually deleted — mine had “bullhead” where it says “N5100” above.

   Obviously, you want to be very sure you have the right package before you flash it. This is also an example of why it is better to install the prepared and released packages from LineageOS or Google or the phone vendor, than to flash things straight into the flash partitions with dd — they have these safeguards against shooting yourself in the foot.

   Of course, once the package was unpacked and that file edited, it had to put it back together.

   Take the package apart

   Open the ROM zip file with winrar. (I used unzip.)

   
mkdir work
cd work
cp ../lineage-15.1-20180618-nightly-bullhead-signed.zip .
unzip lineage-15.1-20180618-nightly-bullhead-signed.zip


   This produced the following directory structure:

   
./system.patch.dat
./boot.img
./file_contexts.bin
./system/build.prop
./system.new.dat.br
./system.transfer.list
./install/bin/backuptool.sh
./install/bin/otasigcheck.sh
./install/bin/backuptool.functions
./lineage-15.1-20180618-nightly-bullhead-signed.zip
./META-INF/com/google/android/updater-script
./META-INF/com/google/android/update-binary
./META-INF/com/android/metadata
./META-INF/com/android/otacert
./META-INF/org/lineageos/releasekey


    

   Edit the updater-script

   I edited META-INF/com/google/android/updater-script with a plain-text editor and removed the first line.

   Put the package back together

   Then to put it back together again, from directory containing META_INF:

   
rm lineage-15.1-20180618-nightly-bullhead-signed.zip # I could do this because it was a copy, still had original in directory above)
zip -rZ store lineage-15.1-20180618-nightly-bullhead-signed.zip * # I deleted lineage because I wanted to use * here


   My first attempt omitted the -r, and failed to apply (since all the subdirs in the .zip file were empty). Sigh.

   One last note: the file utility reported the type of the package before being taken apart as a java archive file. After reassembly, file said it was a zip archive. It seemed to work anyway.

   Hmm. I seem to remember unpacking that second zip-file … and it was full of .img files as well as the updater script. I’ll just leave this here as a note for next time. Also, maybe if I had zipped back up in two stages, file might have recognized the final package as a jar file. To be continued : -)

Extra research

In all my research to fix the problems I had made, I found a couple of other potential ways to get stuff into flashes. So I never got to use these, but they looked useful. I record them here for future reference.

1. You can convert a .img into a .zip for purposes of flashing, by renaming the .img file after the flash partition into which it should be flashed (eg, recovery.img) and zipping it by itself with NO compression — (use the “store” compression method). This is handy if you have an img file to install but only the sideload method will currently work for you (due to various mistakes you may or may not have made up to this point, eliminating all the other methods that might work, and/or misunderstandings about fastboot that make you think it won’t work for you).

   
zip -Z store newzipfilename.zip recovery.img


2. Figure out the flash partition device, and dd the .img file into the flash partition. eg:

   
dd if=/sdcard/recovery.img of=/dev/block/platform/soc.0/f9824900.sdhci/by-name/recovery


   Obviously you have to look at your own /dev/block/platform directory to get the exact path. Mine contains (in system mode):

   
bullhead:/ # ls /dev/block/platform/soc.0/f9824900.sdhci/by-name
DDR cmnlib fsg keymasterbak modem persistent sbl1 tz
aboot cmnlibbak grow keystore modemst1 pmic sbl1bak tzbak
abootbak config hyp laf modemst2 pmicbak sdi userdata
apdp devinfo hypbak limits msadp recovery sec vendor
boot dpo imgdata metadata oem rpm ssd
cache fsc keymaster misc persist rpmbak system


   (Well sorry about the bad formatting … one day I will revisit and format this properly.)

3. Use TWRP’s flash_image utility from TWRP’s shell. E.g.

   
flash_image misc /mnt/sdcard/_update/misc.img


For these last two, you need to put the .img file onto the phone (perhaps using adb) and then run the appropriate command on the phone, perhaps in the shell supplied by TWRP. Be careful with these — it is probably best to flash recovery while booted to system, and to flash system partitions while booted to recovery. Not least because you know you have something you can still boot to, if the flash operation goes wrong. This is why I did not continue to try upgrading the TWRP partition after the phone would not boot to system any more. Once system was working again I was able to flash TWRP (with all my newfound knowledge above) and it was super quick and successful.

Final notes

So I hope I will find this handy, next time I need to upgrade the OS, the recovery, and/or just install new software on or transfer files to/from the phone.

One last note, the xda site seems to be a fountain of knowledge on working with android devices of all kinds (hardware and os-distro), even if you do have to read a lot of individual problem reports and solution attempts to get those nuggets. This page was especially useful: xda beginners how-to guides.

Well that was my weekend … at least I got a beefy blog article out of it : -)

It’s annoying to have to type file:///usr/share/doc/debian-policy/fhs/fhs-2.3.html when you want to see the File Hierarchy Standard … so I made a “desktop icon” pointing to several debian references. After way too much fishing around, I ended up doing the following on my xfce4-based desktop:

Why do I want to run a second X server on my desktop? So I can log in with another account without logging out of the first.

UPDATE 20180611: It is called “acmeshell” now.

My ssl certificate expired the other day and I had to replace it.

Recently I recorded an episode for Hacker Public Radio. It answers some questions that another contributor, knox, had about virtualenvwrapper. My episode will (if all goes well) be published on June 27 as episode 2322.

I had a very smooth upgrade of a Debian wheezy system to Devuan jessie. This is a small system that doesn’t have a desktop or use any other dbus things — it mostly runs as an unattended server and I only connect to it occasionally.

Happy birthday Devuan!

Devuan is a systemd-less fork of Debian.

I’m lucky enough to be sitting in the netconf conference, with Pablo Ayuso, Jamal Hadi Salim, David Miller and others. They are discussing current and proposed future work on the networking subsystem of the Linux kernel.

This is the by-invitation-only mini-conference. Many of the attendees here will speak at the open-registration netdev01 conference from Feb 14 — 17 at the same venue (the Westin Hotel, downtown Ottawa). It’s looking like that conference will be really great. It’s not too late to sign up! I’ll definitely be there.

I looked up a question on stackoverflow, and even found it. But it had no answer, so I wrote one.

With luck, it’s not too far wrong.

… well I guess I answered the wrong question. Oops.

Can an existing folder be redeclared a virtual enviroment with VirtualEnvWrapper?

I thought this was a project that had been made with virtualenv, and they wanted to put it under virtualenvwrapper. But apparently it was made with virtualenvwrapper, and I don’t know what they want to do with it.

They can probably get the answer to their question from my answer, though.

My answer:

Yes.

Anatomy of virtualenvwrapper

The existing project has two parts:

• the virtualenv, where python and the python libs are installed, and
• the project directory where your code is (that uses the virtualenv).

Virtualenvwrapper adds a third part, the virtualenvwrapper hooks. These are mainly shell functions that are called at certain times in a project or virtualenvs life cycle. They live in a third directory — by default, they are installed to ~/.virtualenvs (at least that was true on my Debian wheezy system). The hooks include postactivate, which we will edit below, and a bunch of others such as premkproject, premkvirtualenv, etc. The following list of keywords gives you the flavour of the hooks: initialize, pre/post, mk/rm, project/virtualenv, activate/deactivate. virtualenvwrapper puts these scripts in $VIRTUALENVWRAPPER_HOOK_DIR, which defaults to $WORKON_HOME.

virtualenvwrapper assumes

• all the virtualenvs are in one place ($WORKON_HOME) (defaults to ~/.virtualenvs) Let’s say you have two virtualenvs, one called MYVENV and the other called MYOTHERVENV
• all the project directories are in another place ($PROJECT_HOME).

Let’s say you have a project “is” in directory /home/me/where/my/proj that uses virtualenv MYVENV.

how to use workon to work on your pre-existing code and virtualenv

Here I’m assuming all your virtualenvs are in one place (in my case, they are all in /usr/local/virtualenv).

one-time operations

** edit ~/.virtualenvs/postactivate (+) to have

    case $env_name in
        MYVENV)
            cd /home/me/where/my/proj/is
            ;;
        MYOTHERVENV)
            cd /home/me/where/my/other/project/is2
            ;;
    esac

** links

    for hk in get_env_details initialize postactivate postdeactivate \
            postmkproject postmkvirtualenv postrmproject \
            postrmvirtualenv preactivate predeactivate premkproject \
            premkvirtualenv prermproject prermvirtualenv; do \
        ln -s ~/.virtualenvs/$hk /usr/local/pythonenv/$hk; \
    done

any time you want to work on your project that uses MYVENV virtualenv

    WORKON_HOME=/usr/local/pythonenv workon MYVENV

Of course if all your virtualenvs are indeed in the same place, you can define WORKON_HOME in your .profile and you won’t have to specify it on the command line every time.