tuesday 16 december 2014
With all that soldering successfully performed on my LG Optomus Black (P970), I was finally able to load my own code and actually see the results, on UART.
Building and running
X-LoaderI first tried to build the X-Loader source code that was released by LG and, after addressing some relocation issues, got it to finally print something on serial! The first step was to get X-Loader to run on the device, to a point where it could load and run U-Boot. I had at disposal the version released by LG, which is not exactly the most recent one out there, and found various other versions on the Internet. Overall, I was told that X-Loader development is a bit of a mess: it was initiated by TI for OMAP devices, but there is no clear notion of upstream for it, just different branches that work on different OMAP devices. The closest thing I could find was the X-Loader project on Gitorious, which was an initiative to gather all the various X-Loader trees into one, as I was told. That felt like the right path to follow, so I decided to use it as a base for development for the Optimus Black (P970) codename sniper's X-Loader port. Most things worked nicely by importing code from the LG release, even RAM initialization was rather painless and worked straight away. Soon enough, I was at a point where my concern was loading and running the next stage bootloader: U-Boot.
Loading the next stage
The Optimus Black (P970)'s internal storage is eMMC. There is a partition for X-Loader and one for U-Boot too. As I wanted to keep the devices in an usable state (a working shell on a kernel that actually works can turn out to be an important resource when working close to hardware), I decided to avoid flashing the internal memory. And flashing requires the current system to work, so that would only have worked once. Not really a possibility. Of course, I would still need to be able to boot from internal memory for production, once everything is known to work. Internal storage is connected to the OMAP3's MMC2 controller while the external sdcard is connected to MMC1. The current X-Loader code would only deal with MMC1, so I decided to give booting from internal memory a shot. After all, there was already a bootloader in there (LK) and my X-Loader code should have been able to boot it as well. Given that the bootrom has to be able to boot directly from MMC2 (it is the first boot option), everything is wired according to the TRM: that is, using the appropriate TWL5030 regulators. Thanks to that, the code in X-Loader was able to use the eMMC without any change and it would soon load and run the preinstalled LK.
Booting from the external sdcard
Real trouble began when I started looking into getting X-Loader to read from MMC1 (the external sdcard). Modifying X-Loader so that it would also allow booting from MMC1 wasn't enough to do the trick. As it turned out, MMC1 is powered by an external PMIC, the LP8720: one of its LDOs is dedicated to powering MMC1 (3.0V_MMC). Looking at the schematics, it appeared that the LP8720 is contacted through I2C3. Thankfully, it had good enough documentation available (in addition to the reference code from LG), so it was really easy to figure out how it works. X-Loader's I2C code wasn't designed to allow changing the bus either, so I had to implement that too. With the bus switched to I2C3, I should have been able to communicate with the device. Except that it didn't work. I spent a crazy amount of time looking at the TRM, making sure that every necessary clock was enabled, every needed pin muxed properly, but it didn't do. The controller would only return the same error. Nevertheless, it worked perfectly on the preinstalled Android version and other buses were working normally too. It just didn't seem to make sense and after a week or so, this got me very frustrated.
Frustration and investigation
Spare boardAt some point, I even wrote an I2C probe function from scratch based on what the TRM recommends, with no better result than the original code. Looking at it more closely, it seemed that the controller wasn't getting any NACK. I was able to reproduce the same behavior by disabling input on another I2C bus, or by muxing the pins to GPIO. Still, it didn't make sense that it happened all the time on I2C3, so I asked on the TI E2E community forums but didn't really get a clue there. Since I had another working device at disposal, I decided to tear its board off, keep only the USB connector, load my code and look with a probe at what was actually going on on the bus. Lacking proper hardware, I used my buspirate as a logic analyzer but it didn't really work out, neither did my Arduino Uno. It was also particularly hard as I had to probe on two very tiny resistors to get I2C3's SCL and SDA lines. I decided to head back to LabX, where I knew that people would have a clue about how to do this properly. It turned out that nouveau developer Martin Peres was around at that time and kindly offered to bring his own digital oscilloscope. That was really a life saver! After a short while, we were able to probe the SCL and SDA lines and see things actually going on in there. A few times, we were even able to catch a full trace with SDA and SCL simultaneously. Martin knew the I2C protocol well, so we were able to decode the transaction, only to find out that everything was going on just fine. NACK was there and I just couldn't understand why the controller wasn't getting it.
GPIO to the rescue!
Probing the spare boardIn the meantime, while I was discussing the issue with fellow developer GNUtoo, he suggested that I mux the pins to GPIO to see what reads. The device with UART attached did show 1 for GPIO read on I2C1 SDA and SCL lines but showed 1 on SDA and 0 on SCL on I2C3. It was as if the SCL line wasn't pulled high as it should have been. I could not actually measure the tension there, as opening up the case would tear off my UART setup. On the spare board, it was clear that SCL was pulled high by reading the voltage, so the 0 GPIO read did seem like the I2C controller on the OMAP wasn't reading the values properly. It still worked normally on Android though, so this was clearly not a hardware problem. That's when I decided to take an extra step: checking that the spare board actually read 0 like the board with UART did. Since the spare board didn't have UART, I needed some other way to read back the values. The most straightforward solution was to use the same I2C3 pins, read the boolean value, then configure the GPIO to output and write back the opposite of the read value (so that I would clearly see the difference).
Sub-board connectedAnd it turned out that reading those values with my voltmeter indicated that the spare board was reading a logical 1 on I2C3 SCL. Thus, my two devices were behaving in two different ways. Moreover, it was likely that I2C3 had been working correctly on the spare board since the very beginning. Now what is the difference between those boards? The most obvious thing is that one is loaded with all the sub PCBs while the other one had only the USB connector attached, which would allow me to probe the I2C3 resistors. Regarding the I2C3 bus, there were indeed a handful of slave devices attached to it on a sub-board that I removed, so I plugged the sub-board back and read the GPIO values again. It then indicated 1 for SDA and 0 for SCL! So the differences between both setups were apparently caused by that sub-board, and something in there was bringing SCL to 0. The slave devices were mostly sensors and looking at the schematics revealed that they were using dedicated regulators. After a quick check, it appeared that 1.8V_MOTION_VIO and 3.0V_MOTION were possibly not enabled. Those are LDOs from the TWL5030, so I just thought I would give a shot at powering them up. Right after that change, the GPIO read on the UART-enabled devices turned to 1 for SCL! Another build to switch the muxing back to the I2C controller and I could finally probe the LP8720 regulator correctly!
This post is part of a series of articles about freeing the LG Optimus Black (P970):
saturday 13 december 2014
While I got this LG Optimus Black (P970) phone thinking it would just be another mainstream device to port Replican to, with the usual flaws of those devices, it already caught my attention more than any other phone I stumbled upon in quite some time. Having a GP device meant that the bootrom would allow me to run any code I want as first bootloader: this opens up the way of having a free bootloader running on that device. To this day, only the GTA04 has Replicant support and a free bootloader, so having a second device alike would be a huge step forward for freedom, especially since it is still very common and easy to find second hand.
Loading a bootloader
Boot order resistorsAll of this felt really great, but I still needed a way to load code for developing a free bootloader. The stock bootloader is installed in MMC2, which is the first boot medium. Boot order is defined by pull-up and pull-down resistors attached to some pins of the SoC and thanks to the extensive documentation I had in hands regarding both the phone and the SoC, I was easily able to figure out how it works (not to mention that it's exactly the same thing as the GTA04). After identifying which resistors were actually in place on the board (the schematics shows some resistors that may or may not be present), it struck me while looking closely at it that I was literally one resistor away from having USB boot first and MMC2 boot second.
One resistor awayThis is an ideal situation as it allows to boot code from USB and would still leave the device usable to boot the default system after the USB timeout expires. Even though I didn't have a very the best soldering iron around at the time, I was so excited that I decided to go with a general-purpose iron and thanks to a bit a practice, I was able to remove the tiny tiny resistor without doing any harm to the rest of the device. I just plugged the phone in with no battery and started jumping around as this showed up on dmesg:
usb 3-4: new high-speed USB device number 9 using xhci_hcd [usb 3-4: unable to get BOS descriptor usb 3-4: New USB device found, idVendor=0451, idProduct=d00e usb 3-4: New USB device strings: Mfr=33, Product=37, SerialNumber=0 usb 3-4: Product: OMAP3630 usb 3-4: Manufacturer: Texas Instruments
Tiny tiny resistorIndicating that the bootrom had booted from USB first. Another approach to all this would have been to write the boot order to a particular register from the running system in order to tell the bootrom to boot from USB first, but this would have introduced a longer development overhead and taken all the fun out of it! A set of convenient tools that is well known to people working on OMAP bootloaders quickly enabled me to converse with the bootrom, as described in the TRM. Eventually, I was able to upload the stock X-Loader build via USB and get it to boot the system that way.
Seeing the light
DUMMY INTERFACENevertheless, I was still blind. Being able to run code is one thing, but debugging a whole bootloader is another much more complex one. Mostly, I needed a way to get serial output from the code I would submit. The usual way to do this is to use UART I/O. On the OMAP3, UART3 is usually reserved for that purpose: I wasn't so surprised when I found that UART3 connectors were exported on pads at the back of the device (under a sticker). Sadly, those pads were not very easy to access, nor were they big enough to allow for easy soldering. Feeling adventurous, I decided to give it a try with my messy soldering iron and while I got a tiny wire to hold in place for a moment, it eventually ended up in tearing the pad off the board, making any further soldering there impossible. That was a big turn off but instead of giving up, I looked for other ways to get UART3. There was only one option left, soldering on the pads of the UART switch module. Again, it was some very precise work there and I failed again, leaving a mess out of the connectors. The phone still booted, but I lost all my chances of debugging bootloader code on it.
A glimpse of hope
That's when I recalled meeting a member of one of the local free software groups at an install party, a few months back who had told me about the same device that he bricked. With all the information I had in hand, it was hard to believe such a device could actually be bricked, so I got in touch with him, explained the situation and as it turns out the non-functional device was still sitting around in a drawer, I quickly bought his “bricked” device at a fair price. With a new device in hands, the adventure could resume where it was so abruptly suspended. To do things right, I decided to forget about my own unadapted soldering iron and do the soldering at the local hackerspace, the LabX. While I'm sadly not a member nor a frequent visitor of the LabX (mostly because schoolwork takes most of my week evenings), I knew some of the people there and was very happy to see that, as usual in our community, many were happy to help and give me hints on how to do this right. Despite all that good intention, the dumbest thing happened. The very afternoon before I planned to get the soldering done, I plugged the phone to a wall charger and left it there for a while. When I returned to it, it was incredibly hot and wouldn't turn on again, which got me worried. Nevertheless, I packed it and moved to LabX, where it didn't show any sign of improvement. Still, I decided to go through with the soldering, as it would be good practice anyway. I got both the resistor removal and the UART pad soldering right, but the device still wouldn't work. After some investigation, it turned out that the battery V+ and GND signals were shorted, perhaps as a result of a melted component I couldn't locate. I was never able to get that device to turn on again.
Happily ever after
UART3 Tx and GND padsBack to the same situation I was a while ago, with no suitable device for development. Since I just couldn't give up after all the progress that was made, I bought yet another device second hand and went back to LabX for another session of intense soldering. The resistor removal went well, but we got the UART pad teared off the board again. I then headed for the backup solution, the UART switch, which succeeded thanks to the precise tools I had at disposal. The same evening, I was able to get the stock bootloader's debug output! After a bit of thinking and tinkering, I ended up with two nice pads made out of aluminium foil for the OMAP3 UART3's TX and GND. I didn't want to risk going as far as getting the RX signal out there, as it felt too much of an unnecessary risk (debug output is fairly sufficient for debugging).
This post is part of a series of articles about freeing the LG Optimus Black (P970):
saturday 06 december 2014
Every once in a while, an unexpected combination of circumstances ends up enabling us to do something pretty awesome. This is the story of one of those times. About a year ago, a member of the Replicant community started evaluating a few targets from CyanogenMod and noticed some interesting ones. After some early research, he picked a device: the LG Optimus Black (P970), bought one and started porting Replicant to it. After a few encouraging results, he was left facing issues he couldn't overcome and decided to give up with the port. As the device could still be an interesting target for Replicant, we decided to buy the phone from him so that I could pick up the work where he stalled.
Upon receiving the item, I decided to dig a little more into what that device was made of and found that very precise technical documentation about it leaked online (EN_LG-P970_SVC_ENG_110415.pdf). This mostly contains all the electrical schematics of the phone, a precise list of the components used and a few other interesting things. Such in-depth documentation is rather rare for such mainstream devices: we usually get service manuals with some of that information, but seldom something to reach this level of completeness. That really got me interested in the device.
In the meantime, I found a few interesting threads at XDA mentioning the bootloader of a couple of OMAP4 devices from LG. Apparently, LG leaked the private keys for signing the first bootloader on these devices in a release of the bootloader's (copylefted) source code. As often on OMAP devices, the bootloader is X-Loader, a minimalistic copy of U-Boot source code that can run with a small memory footprint. While all of this is great fun, it's nothing we can seriously use in Replicant. At best, it enables developers to play around with the device's bootloader. And we know all too well how much of an issue signed bootloaders represent to software freedom on those devices, so this is kinda nice. Ironically, those private keys aren't the only things that we leaked aside free software releases for these LG OMAP4 devices: that's exactly where the recent PowerVR Series 5 source code leak originates, rising its own share of issues for free software.
Seeing that people were playing around with bootloaders on other LG OMAP devices got me interested in my recently-acquired LG Optimus Black. A quick look at the first bootloader's binary data revealed that it is X-Loader as well, but without the usual headers for signature checking. In addition, the second stage bootloader turned out to be U-Boot on early versions of the devices, which was replaced by LK on the ICS update. Nevertheless, X-Loader and U-Boot are copylefted free software, so I went ahead and tried to grab the released source code from LG. After downloading a few different archives, I finally spotted the one with the bootloaders' source code. All of it looked nice and nearly complete (only a small piece of software specific to LG, called FOTA, was missing from the release and broke the build, but I found equivalent code in one of the OMAP4 device's source code and it turned out to be nothing of interest).
At this point, I just couldn't stand in place as one question was left unanswered: can I actually build and run that code on the device? That is, are signature checks enforced on the LG Optimus Black (P970)? Even if the lack of dedicated header on the binary was a good sign, I am usually not too optimistic about these things, so I wanted to double check. As it turns out, OMAP platforms are also incredibly well documented by TI, so I just grabbed a copy of the Technical Reference Manual (TRM as we call it) and looked for a way to identify clearly whether the device enforces signature checks. Of course, that part is not publicly documented in-depth, but it seemed very clear from the documentation that OMAP SoCs are produced in (at least) two different fashions: a General Purpose (GP) version that doesn't check the first bootloader's signature and a High Security (HS) version that does check those signatures. Usually, GP versions of OMAP SoCs are shipped on reference or development boards, such as the BeagleBoard. They also enable projects like the GTA04 to create a device that runs a free bootloader. However, it is rare to see GP OMAP chips being used on mainstream devices. While very happy from that finding, I still needed a reliable way to figure out whether the Optimus Black is using a GP or HS version of the OMAP3630. And it turns out that a couple of registers hold that information. One of them, CONTROL_STATUS is 4 byte-long and holds the device type with a 1 byte offset, at 0x480022f0. It also holds the boot order information (SYS_BOOT) on the lower byte. On android, it can easily be accessed with devmem:
devmem 0x480022f0 16The higher byte is 0x03, which clearly indicates a GP device: HS devices usually return 0x0A.
This post is part of a series of articles about freeing the LG Optimus Black (P970):
friday 27 june 2014
During the past few months, I've been working on bringing Replicant 4.2 support to the Goldelico GTA04, a montherboard replacement for the Openmoko FreeRunner (GTA02) that is manufactured by Golden Delicious with the intent of being free software friendly. The board design is also released under a free licence. There is an active community of developers and users dedicated to the GTA04 and other similar projects, brougt together under the hood of OpenPhoenux. Other similar projects include the upcoming Neo900, a motherboard replacement for the Nokia N900 using similar hardware as the GTA04 and also aiming to be free software friendly, with a particular emphasis on security and privacy features.
I recently published a blog post covering the status of the Replicant 4.2 port to the Goldelico GTA04 on the Replicant blog, with a nice video highlighting which features already work well. However, in the present post, I would like to cover some of the technical aspects of the port, especially regarding the use of a recent mainline Linux kernel with Android 4.2.
Currently, Android devices ship with Linux 3.4, after a long time of using version 3.0, which started with Android 4.0 Ice Cream Sandwich. To this day, the reference common Android kernel repos have branches with Linux 3.10 and experimental work is done on Linux 3.14. That's pretty close to the Linux 3.12 version we're using on the GTA04!
The first challenge to overcome was dealing with suspend and resume. Android uses a particular mechanism to implement opportunistic suspend, using wakelocks: both the kernel and userspace can register these locks to prevent the device from going to suspend. In Android kernels, there is another mechanism that allows some non-critical chips to reach suspend state before the rest of the system: this is earlysuspend. As the whole Android system is built around the concept of wakelocks to handle power management, something similar needed to be implemented in the mainline Linux kernel. After efforts from the Android kernel maintainers, wakelocks were implemented in a clean fashion some time ago. In order to implement opportunistic suspend, a separate interface was implemented on mainline Linux, known as autosleep, which uses different nodes than /sys/power/state (that the Android kernels use directly for opportunistic suspend). Starting in Android 4.1, a library was added to Android in order to detect and handle these different modes.
So thankfully, everything was already in place to use autosleep properly on Android 4.2. Except that it didn't work. This turned out to be because of a dedicated capability that was missing from the Android userspace: CAP_BLOCK_SUSPEND. It turned out to be easier to just revert the patch restricting access to wakelocks to users with that capability in the kernel.
Android USB Gadget and Android Debug Bridge
Android also went their own way in implementing USB device drivers for the various things that are used by Android: mass storage, rndis for USB networking, MTP for file access and a couple others, including ADB. The Android Debug Bridge (ADB) is a great way of debugging what's going on with an Android device, providing easy access to the logs, to a shell and file transfer (these are the features of ADB I use the most). All of that is not integrated at all in the mainline Linux kernel, so there was some substantial work to do here. The first thing to do was importing the related commits from the android-3.10 branch of the common Android Linux kernel. All of that built nicely with only minor code corrections, to follow some API changes in 3.12 and some features like rndis or MTP worked right away, but the most interesting part was left not working: that's the Android Debug Bridge. A few kernel versions back, there used to be a dedicated composite function driver for ADB, but a commit by one of Android's kernel maintainers totally gets rid of it, calling it obsolete with no further information. While attempts to restore it failed, I tried to find out in details why it is obsolete and if that meant the final death of ADB, that I found rather convenient. Thankfully, someone found out what happened and wrote about it: the dedicated ADB driver was being replaced by another one using FunctionFS, a more flexible and generic way of implementing such drivers, directly from userspace. It turns out that FunctionFS support for the userspace ADB server was merged nearly entirely in Android 4.2. I had to backport a missing fix to have it fully working properly and also had to import adaptation a patch in recovery to have sideload working with FunctionFS too. A few bits were also needed in the initialization procedure to have things set up right. Once all of that was done, it could finally run flawlessly!
Headset/headphones detection with SoC Jack
So Jack's a good guy. Everyone knows about that. Whether he's saving the homeland from yet another threat or just letting us know something of interest just got plugged in one hole or another, it always feels great to hear from Jack. However, the Android kernel guys didn't seem to appreciate his participation in the show as much as we all do, or at least until recently.
The traditional way of reporting a headset or headphones plug/unplug in Android kernels was using a switch called h2w, reporting these events to userspace. The mainline kernel prefers another approach, using our beloved Jack SoC architecture. It also provides a convenient way of reporting button pushes, which is quite nice. So instead of rewriting it all using the h2w switch, it struck me that there is a frameworks config option to politely ask Android to give Jack some consideration. And when it does that, everything works great, including button press reports!
saturday 15 march 2014
A few days ago, I disclosed (on behalf of the Replicant projet) our research regarding a back-door found in a proprietary program running on Samsung Galaxy devices' applications processor. This back-door lets the modem perform I/O operations on the device's storage.
For the full statement, please refer to the article posted at Free Software Foundation's website. The relevant technical analysis is available at the Replicant wiki and a complementary statement was issued at the Replicant blog.
In the few hours following the publication, an outstanding number of technology-oriented websites relayed the news, including Phoronix, Slashdot, LWN and XDA-Developers. I'm very glad the press found interest in that research and I'm confident it'll help more and more individuals realize the importance of being in control of their computing: that is, to understand what's at stake with free software.
A few recent developments particularly caught my attention: Ars technica bothered to ask an actual security researcher, Dan Rosenberg his thoughts on our findings. Good thing they decided to go deeper than only duplicating the information. On the other hand, Samsung issued a statement about this issue:
Samsung takes the security of its products extremely seriously. We have investigated the claims that have been made and can confirm that there is no security risk. The Free Software Foundation’s recent allegations are based on a false understanding of the software feature that enables communication between the modem and the Application Processor chipset.
Mostly, the point that is argued by Dan Rosenberg is that there is no evidence of any ability for a remote party to use the back-door, nor any known exploit to make use of it remotely. As a matter of fact, we didn't look at how this could be used over the air: this was not the point of our research. The problem we intended to highlight is not so much about how in practical terms an intruder could use this anti-feature remotely to access and modify the data stored on the device, but rather to show that a particular proprietary software implements a feature that could be used to let the modem gain data I/O access over the device. This is where we find the back-door to be: at the interface between the modem and the applications processor. We do consider the modem to be an “unknown” area that offers no guarantee at all regarding security, since it is running proprietary software. Hence, we believe it is relevant to assume the worse and consider it compromised and subject to remote control. Several indications tend to make us think this is actually what is going on: Craig Murray described how a mobile phone had been remotely converted to a spying device in Murder in Samarkand. Considering the recent revelations regarding the practices of several governments' intelligence agencies, we find it hard to believe there is no way modems cannot be remotely compromised.
The goal of our action was to make people aware of that particular issue. One might consider it to have no value, provided they don't think modems can be remotely compromised and others might see it as a crucial security flaw in the event the modem is compromised, as we do. The fact that it was implemented for another purpose or was not intended to be used in malicious ways doesn't change anything at all: an attacker with remote access to the modem will be able to issue the incriminated requests. There is no possible “false understanding”, in the way Samsung seems to imply here.
For the record, we didn't at any point intend to distort the truth to bring attention to our project or our research, nor did we intend to ruin Samsung's reputation. We simply felt it was our moral responsibility to spread the word about it. I believe anyone can decide for themselves whether they have faith in Samsung's good word that this introduces no further security risk, but let it be clear that it doesn't get any more certain than what good faith can provide. We are still looking forward to working with Samsung to make things right, in case they decide to abandon their current position of denial.