This series of blog posts is set to cover the journey of adding upstream U-Boot and Linux support for the F60 Action Camera, covering another exciting new use case with freedom in digital technology!
Allwinner and Replicant Work
As I previously mentioned on the Replicant blog, I have started a final engineering internship at Bootlin (formerly Free Electrons) in Toulouse, France. The internship is focused on bringing upstream support for the VPU used in Allwinner platforms, as a continuation of the long-standing reverse engineering effort carried out by the linux-sunxi community.
XFCE running on an A10 tablet
This effort is also a continuation of my ongoing personal interest in these platforms, especially in mobile form factors such as widely-available tablets. Although I am no longer involved with contributing on the technical side of Replicant, I still care for the project and would like to see it move forward. Finding new developers for Replicant is hard and it's fair to say that a large part of the project is kept alive by its community of users and contributors. The project inherits from its upstream (currently LineageOS, that counts more developers by far), but a significant number of adaptations have to be introduced to workaround the use of proprietary blobs for hardware support, for each and every new version and device we want to support.
In the free software community, most of the effort related to liberating hardware support in a sustainable and future-proof direction happens (sometimes inadvertently so) in upstream projects such as Linux, coreboot and U-Boot. These projects usually drag more than enough contributions to keep their maintainers busy. Since features are added through a code review process (that requires submitting clean and maintainable code in the first place), the set of features supported by mainline is slow to pick-up with downstream modified versions provided by manufacturers, that are used instead in Replicant. For the mobile use cases that we aim to cover, the constraints are high in terms of required features and power management. This is why it did not seem like a good idea to ship mainline Linux in Replicant at first. However, using downstream software components has a cost in terms of maintenance (no kernel updates), security (no kernel updates) or long-term sustainability and compatibility with newer technologies (still no kernel updates). The amount of required time spent dealing with code where only the very minimum amount of structure and conventionality is in place is also very significant.
After a while working on Replicant, I became more and more reluctant to dedicate time to new contributions, mainly for the fact that I had to deal with downstream code that was just very unpleasant. It also felt like all the time and effort put-in could only ever have a very short term impact. That the level of contributed software support was only useful as long as the Android and Linux versions stayed relevant (and well, we can't say that they stay relevant for a very long time for these use cases).
This is why I was dragged more and more towards mainline software.
A Plea for Upstream Software
The effort to liberate hardware support is much more significant with mainline, but these contributions do tag along and stay, as long as there are people to care for them. The decisions regarding the code in future versions are also taken by the community instead of a single entity, although the community may also take decisions detrimental to freedom in digital technology. With mainline support landed in Linux, the userspace interfaces for systems are standardized and much less work has to be put-in on the userspace side (or more precisely, it only has to be done once and incrementally updated with new features). This means less involvement required on the Replicant side, with the ultimate goal of making Replicant a sheer rebranding of Lineage OS without proprietary blobs and using only generic Hardware Abstraction Layers (HALs) shared across Android systems. The Android-x86 project already has such HALs for various aspects of userspace software support (some of which where developed by companies like Intel that stick close to mainline), but there is still work left to do to adapt these HALs for mobile uses cases. Since these could benefit any Android system, it would be interesting to bring a coordinated effort for supporting mainline kernel interfaces through generic Android HALs (something an entity like Linaro could care for). This would allow redirecting the development effort for hardware support away from the downstream userspace side of Android systems and focused on mainline instead. Of course, mainline support also benefits other GNU/Linux systems and opens up the possibility of properly supporting mobile use cases in these systems, while also enabling new use cases.Purism is currently working in that direction, with projects to integrate KDE and GNOME-based interfaces to a mobile telephony-enabled device. Running GNU/Linux on mobile devices is not exactly new though, as a first wave of interest rose around 2008 from the Openmoko community, revolving around the Neo FreeRunner phone running with GNU/Linux. One of the systems supporting that phone was SHR (for Stable Hybrid Release) that was based on the Freesmartphone.org (FSO) framework for mobile devices and went on to support other devices such as the N900 or the Nexus S. But just like with the original systems for the FreeRunner, using regular GNU/Linux applications on SHR was hard and required using a stylus. The interface was obviously not designed nor adapted for the mobile use case, just like plenty of other system components. On the kernel side, the amount of changes required for the FreeRunner made the upstreaming process slow (and it was still not fully completed to this day).
The GTA01 development device, the Neo FreeRunner (GTA02) and the GTA04
Nowadays mobile devices revolve around the ARM instruction set (and the very commonly found ARM Cortex core implementations from the company) and ARM support in the mainline Linux kernel has come a long way since 2008. Most notably, the introduction of the OpenFirmwaredevice-tree (initially designed for supporting PowerPC devices) allowed a clean distinction between the hardware description and the drivers, which were previously glued together in (plenty of) platform data initialization laid out directly in the source code. Back then, adding support for a new device was tedious and each kernel had to be compiled for a precise device. Device-tree changed the game here, as it became much easier to introduce new device-specific support and possible at all to run the same kernel binary on different devices with the same ARM instruction set. This also made platform bringup a more streamlined task (by easily reusing similar blocks from previous generations and applying appropriate quirks at run-time). The number of supported ARM platforms, features and use cases kept on increasing, with the commitment of big companies like TI, Samsung, Google, ARM, Freescale/NXP or Rockchip, entities like Linaro or the linux-sunxi community but also smaller players (often providing upstreaming services to companies) like Bootlin (formerly Free Electrons), Collabora or Pengutronix. While most of the supported platforms are not ready for conventional mobile use cases yet, some come close and already offer support for a reduced number of use cases.
Signature Verifications Gone Wrong
Despite free software support in Linux, some platforms are fatally flawed when it comes to the boot software they are running. This is the case with enforced boot software signature verifications with missing keys, where the circuitry of the platform validates the digital signature of the first software running at boot with a public key stored in read-only memory. If the signature was not created using the (missing) private key associated with the public key, the platform will simply refuse to boot. A significant number of ARM platforms, such as the ones made by Qualcomm (Snapdragon and others), Amlogic or Samsung (Exynos) are plagued by enforced boot software signature verifications with missing keys. While signature verifications are a really good thing to have, it only makes sense if the user of the device has access to the associated keys. When the keys are kept secret by the manufacturer of the device or the platform, the whole security model of the device is delegated to this third party. The user cannot decide of their own security model and consider potential threats based on their own situation, that may not set this third party as a fully trusted peer. Instead, the implemented security model only covers this third party's threat model, that is designed to consider the user as a threat. While it might sound very odd, this is the result of enforcing technical limitations such as DRM, that are designed to forbid the user from accessing the unlocked raw multimedia contents.
LG Security Error
This modus operandi looks very similar to what a parent would do to prevent their newborn child from wandering around during night time: applying a technical restriction such as bars on the infant's bed. This is perfectly fine when the individual subjected to the restriction is not able to decide what is good for themselves. But when it comes to individuals with that capacity, imposing such intentional restrictions designed to serve this third party's interests first and foremost feels very inappropriate. But that's not the end of it: this limitation removes the practical possibility for the user to replace the software signed by these missing keys, making it proprietary software de-facto. It sometimes occurs that free copylefted software is used in devices that enforce such signature verifications, as it first occurred with the Tivo (and coined the name Tivoization for the process). Even with source code and the appropriate compilers available, only intermediate forms of software can be obtained. The product form of software (that is the form adapted for practical use) is intrinsically tied to the processing unit that executes the associated software instructions. This is why the practical ability to install the generated intermediate form into the code storage from where the code is executed (effectively making it the product form of the software) is an absolute requirement. When signature verifications are enforced, the part of the installation process related to signing the binaries (so that they can be executed) is missing. The software thus never reaches its product form, leaving the user with freedom over the source and intermediate forms of the software but not over its product form. As a result, the software cannot be considered to be free as a whole, although its license says so.
Current Status of Free Upstream Support
Thankfully, a number of ARM platforms are not flawed in this manner and allow running free boot software. Examples of such platforms include Allwinner, Rockchip, i.MX, OMAP GP and Tegra (without keys burned). U-Boot is widely used on devices with these platforms and manufacturers often use modified downstream versions based off ancient upstream releases. The associated source code for both U-Boot and Linux is not always available, although these projects are covered by a copyleft license. And when it does, it often takes the form of a board support package tarball with no version control information. Some parts of the code required for hardware support can also come missing or be voluntarily moved out of copylefted projects (something that Android seems to vastly encourage). This is often the case with DRAM initialization and features related to multimedia. Nevertheless, these source releases are useful and often allow adding mainline support for these devices.
The Google OP1 (Rockchip RK3399) as found on the Samsung Chromebook Plus
At this point in time, the platforms mentioned and many of the devices that use them are supported, to a more or less advanced degree, by upstream free software. Many significant tasks are still ahead of us, but there is work in progress going at a good rate. Support for Vivante GPUs (found in i.MX platforms) was merged and is being improved with time. There is also work in progress for both Mali Utgard, with the revival of the Lima project and Mali Midgard/Bifrost, with the Panfrost project that concerns a number of platforms such as Allwinner and Rockchip. Multimedia features in Linux are also work in progress, with the introduction of the media request API that will allow supporting stateless VPUs (that don't have a dedicated processor, nor require any firmware) in the V4L2 framework, coupled with the MEM2MEM framework that already allows supporting stateful VPUs such as the one found on Exynos and the i.MX6 platform (that require a proprietary loaded firmware). Support for stateless VPUs has been working in progress on the Rockchip side, with a reference driver available on the Chromium OS kernel tree, using an early proposal of the media request API, as well as an early driver from developer Ayaka, in addition to the reference Rockchip kernel driver. On the Tegra side of things, developer digetix from the Grate project has been working on VPU support for early generations of Tegra, that should apply to newer generations with limited effort. On Allwinner, VPU support is work in progress and a number of versions of the Sunxi-Cedrus VPU driver have been submitted already.
Innards of an i.MX6 CuBox-i
In addition to VPUs support, a number of image processing units are made available (especially so on Rockchip) with features ranging from colorspace conversion, and scaling to 2D operations like bit blit and porter-duff alpha blending. Some of this processing is useful for general-purpose image processing, like colorspace conversion and scaling and seems relevant to the V4L2 subsystem, while 2D operations seem more relevant for the DRM subsystem. There seems not to be much support for 2D operations acceleration through generic DRM interfaces in regular userspace implementations from freedesktop.org. Support for such acceleration in compositors would allow significantly speeding up these operations without resorting to the GPU. Since GPUs in graphics cards are often required for video features in x86 systems, it has become a common assumption that GPU support is available for all types of graphical operations. Needless to say that GPU support on ARM is still work in progress and that it is currently generally not available without proprietary software (especially on the many platforms that embed Mali or PowerVR GPUs). Moreover, using the GPU is not power-efficient compared to dedicated hardware components and sometimes does not even perform as fast.
However, the DRM subsystem already leverages plane overlays, that each have their own framebuffer and position on the CRTC and are blended by the hardware before hitting the encoder, as a single frame. But planes support in Xorg is currently not as easy thing to implement, especially when it comes to showing YUV 420 video frames in a dedicated plane. The historical approach for supporting this use case consists in setting a color key on the display hardware, that sets a particular color to alpha, and blend the result with the video frame, that integrates with the rest of the display contents. One downside is that the coloured box bounding the video plane has to follow the movements of the window displaying it, so some coordination is required. This coordination as well as color-keying was historically provided by the Xv extension to the X server. Alas, this extension relies on buffer copies, introducing a significant bottleneck in performance and sharing buffers directly is a much preferred approach nowadays (that wade possible with DMABUF handles). Thankfully, DRI3 and Wayland bring solutions to the issues found on Xorg.
Other features related to multimedia are slowly picking up support in Linux, with support for media pipelines and camera sensor input controllers. Other specific features also require significant effort, such as proper power management (a crucial feature on mobile devices). With all these developments happening, the idea of running mobile devices on upstream free software with a significant set of features allowing usual daily use cases slowly becomes a reality. These days, embedded mobile devices only seem to get more and more diversified: from laptops to phones and tablets, they are nowadays found in all sorts of form factors such as convertible laptops, single-board computers, mini PCs, dongles, home automation equipment or even connected surveillance, car and action cameras. All these new use cases are running with wild downstream software that causes a great deal of issues in terms of freedom and privacy/security. The proposed solution to allowing a somewhat sane use of this technology clearly has to start with upstream software support. Some of these products and use cases are not necessarily a good thing to have on a society-wide scale, but provided that they do exist, we might as well try and sanitize the technology that supports these new use cases.
Covering a New Interesting Use Case
When going through some of the latest released devices on Chinese marketplaces like Aliexpress or GearBest, I noticed a number of 4K action cameras featuring Wi-Fi and HDMI. With an ever-renewed interest for finding out about the guts of unusual devices (well, come to think of it, of usual devices too), I ran through the specs and found that the SoC is an Allwinner V3. One that probably has the VPU that I am writing support for. One that also has an 8M pixel sensor and a wide-angle lense attached, screaming to be piped to the VPU's encoder. But unfortunately, one that is not yet supported in upstream U-Boot and Linux. Thankfully, the community was quick to pick-up support for the V3s SoC from Allwinner. It was contributed to mainline U-Boot and Linux by linux-sunxi community member Icenowy, who has also designed a board using the V3s. The V3s is an integrated LQFP package grouping 64 MiB of DDR2 DRAM and the same processor and controllers as Allwinner's V3 SoC.
Allwinner V3 SoC
In the past, Allwinner has already released chips with minor variations under different names, such as the A10s matching the A13 or the R16 matching the A33. These chips share nearly the exact same set of registers as the platforms they are based on (if not the very same), so when it came to adding support for them in U-Boot and Linux, only a minimal amount of work was required. With V3s support in good shape in free upstream hardware support projects, adding bringing up the V3 based on the existing V3 support with a device in hands looks like a rather reasonable task.
And so I clicked and typed and clicked and typed until I received a little package with my name on it and a F60 Action Camera inside. Next up in the process: discovering and documenting the components used in the camera, starting with a hardware teardown!
This year's edition of RMLL/LSM, the free software conference that travels in and out of France (with an international aim) just ended. Time to take a step back and look at what happened during the 4/5 days I was there.
Thankfully, I get to travel to such conferences using money from the Replicant fund, so I will be refunded both my train tickets and my stay this time again. It makes it much easier (and to be honest, possible at all) for me to attend such conferences. This way, I don't have to worry about finding a summer job and can instead focus on what I do best, reverse engineering proprietary stuff and writing (free) replacement code.
Monday
This time, I arrived on Monday afternoon and could attend a first talk after a quick chat with the lovely people from the information booth. The talk, that was part of the security track, was presented by Lunar (Tor and Debian developer) and reported the current state of the art of reproducible builds for Debian (and more). It was really nice to see such overwhelming progress accomplished, after I attended the initial talk during which he announced the reproducible builds initiative a year and a half back, at FOSDEM. Lunar's talk answered most of the questions I had regarding how to make software reproducible. I am especially interested in making the U-Boot bootloader reproducible. I had that idea at the back of my head for some time now and decided to jump in after seeing a contribution in that direction on the U-Boot mailing list. Eventually, we managed to get some of that work done (right) later in the week. The rest of the afternoon was filled with chatting around in the village. In the evening, I met people from the event at a local bar, were free music was being played. It was a nice atmosphere and we had some interesting technical discussions (and let's be honest, many trolls as well)! I was thrilled to see that people were not only aware of Replicant, but also had a lot of interest in it.
Tuesday
On Tuesday, it was time to get to the workshop I was supposed to co-host. The whole day was filled with various activities around different kinds of embedded devices (some were about scientific measurements, some about Arduino, etc). In addition, most of these were built with education in mind. When the first one ended, it was time for me to leave in order to reach the room where I was to present my first talk. The video recording seemed to be done right and hopefully, the video of the whole thing will be available eventually. Not that many people showed up, but the ones that were there seemed really interested. I got to meet and talk with a few people after my presentation, some of whom decided to come to RMLL only to have a chat with me. What a surprise! The afternoon went on and I attended a few talks, including a round table around the concept of civilian re-appropriation. It was presented by Veronique Bonnet, who's a philosopher and a member of April, one of the French associations that take a stand for freedom on digital devices (and actually get it right). Richard Stallman (RMS) was also there, even though he apparently didn't quite understand the wording of the subject in French. Still, some interesting things were said and RMS displayed his usual sense of humour here and there, sometimes making the audience burst into laughter. Once it was over, we got to chat a bit, in a very friendly environment, which was very nice. A free music concert was organized near the event, so a few youngsters (including myself) decided to go before calling it a day.
Wednesday
Wednesday was the occasion for me to be around the workshop more often, but very few people showed up because it was missing from the printed schedule, something I only came around to realize once it was too late, a week before the event or so. Despite some paper indications and the addition of the workshop to the online program, the place remained rather quiet, which wasn't so much of a problem given my aggravating state of sleep deprivation. Before lunch, I gave my other talk about Replicant, a longer and much more technical one. To my surprise, many more people showed up (perhaps the result of meeting a few people during the first few days). The talk itself went well and everything fit on schedule. For the record, the content of both talks (which summed up to 1 hour and 40 minutes, mostly excluding questions) was what I had planned on delivering during my (50-minute long) talk at FOSDEM this year: no wonder I had to stop half-way back then! Afterwards, I was lucky to get help for making U-Boot reproducible from Lunar, whose efficiency, vivacity and kindness really made the task painless. There are still bits and pieces to bring together to craft a proper patch, but I'll get around doing it sooner or later. After alternating between the workshop and talking to great people at the village, I ended up meeting back lots of interesting people at a Harry Potter-themed bar, le Chaudron Baveur (not that the owner deserves any particular good word about it, given that he wasn't exactly pleasant).
Thursday
The next day went on pretty much similarly, except that I had no talk left to give, and thus no particular pressure or place to be at (except for the workshop, that remained desperately empty). Just like any other day at RMLL, I met tons of incredible people and had lots of interesting talks. In the afternoon, the main “political” event of the week took place, with a round table regarding interoperability and DRMs. The speakers were a high-ranking official from HADOPI and Marie Duponchelle, who conducted a thesis on the very subject. Overall, it was very strange, mostly because the nature of the debate soon revealed to be astonishingly stupid and a pure waste of time. The main question was how to allow the entertainment industry to use DRMs while maintaining interoperability. The answer is plain and simple: it can't be done. Despite that very clear statement, that was introduced eventually by Marie Duponchelle (in spite of the situation Videolan was in), the debate went on and the HADOPI representative produced vague statements with apparently no ties to the technical reality one after the other. At some point, the audience got pissed off and started expressing our community's point of view in very clear ways, such as encouraging everyone to share culture in the most efficient ways: torrent, VPNs and Tor. All that followed by rounds of applause, naturally. More serious questions were raised, such as the existence of public domain in practice when only copies of an piece of art exist with DRMs. The HADOPI representative answered that any piece of art is itself distinct from the media it is distributed on, which may be a fair point, but doesn't solve anything. She also suggested that the BNF could receive non-protected copies of it, but this is neither its mission nor a reliable solution for people who will find a DRM-tainted copy decades later, unable to read it despite the fact it is in public domain. The talk ended with François Revol (Haiku developer) handing over a big coin of 1 Hadopi to the representative, a way to show our community's support for this organism at a time of budget cuts. Bottomline: this was purely a waste a time (despite providing some form of entertainment). No wonder some decided to master the fine art of origami during the talk instead of listening to that whole mess. Hopefully, the main political talk will prove to be more interesting next year. In any case, it probably cannot sink much lower. Later that day was the repas du libre, the traditional classy-ish dinner where we all meet together and look back at the week (everybody knows Friday is mostly for getting over the hangover induced by the previous night's drinking and also for packing). I didn't plan on attending at first, since the food wasn't really worth it last year, but changed my mind given some pretty solid arguments. Or maybe just pretty at all. In any case, I got to formally meet Benjamin Bayart (some fine blood forensics can probably assess for that) who not only showed interest in Replicant (and other things I'm doing these days) but offered me his help in every way possible. That evening is probably the time I had the most fun at RMLL, thanks to Benjamin, Fabien, Frédéric, jfefe and plenty others. Kudos to them for their support in times of great needs, that was a relief. Thankfully, my LG Optimus Black (P970) booted just fine, so in the end, it's fair to say that the various issues encountered were accounted for and that the whole thing provided a working result, that will certainly become a base for future developments, now that the initial trouble is behind us.
Friday
Friday was a bit less fun than the other days, in part for reasons of a physical nature. I still managed to reach the event in time to be reminded that Trinity does use nmap and it's fair to say that it's the coolest thing. Sadly, some people had to leave early and I couldn't conclude some of the ongoing arguments that had developed throughout the week. Hopefully, there will be other occasions to meet (and certainly closer than Beauvais), but that's ultimately not really up to me, despite my best intentions.
A few days ago, I disclosed (on behalf of the Replicant projet) our research regarding a back-door found in a proprietary program running on Samsung Galaxy devices' applications processor. This back-door lets the modem perform I/O operations on the device's storage.
In the few hours following the publication, an outstanding number of technology-oriented websites relayed the news, including Phoronix, Slashdot, LWN and XDA-Developers. I'm very glad the press found interest in that research and I'm confident it'll help more and more individuals realize the importance of being in control of their computing: that is, to understand what's at stake with free software.
A few recent developments particularly caught my attention: Ars technica bothered to ask an actual security researcher, Dan Rosenberg his thoughts on our findings. Good thing they decided to go deeper than only duplicating the information. On the other hand, Samsung issued a statement about this issue:
Samsung takes the security of its products extremely seriously. We have investigated the claims that have been made and can confirm that there is no security risk. The Free Software Foundation’s recent allegations are based on a false understanding of the software feature that enables communication between the modem and the Application Processor chipset.
Mostly, the point that is argued by Dan Rosenberg is that there is no evidence of any ability for a remote party to use the back-door, nor any known exploit to make use of it remotely. As a matter of fact, we didn't look at how this could be used over the air: this was not the point of our research. The problem we intended to highlight is not so much about how in practical terms an intruder could use this anti-feature remotely to access and modify the data stored on the device, but rather to show that a particular proprietary software implements a feature that could be used to let the modem gain data I/O access over the device. This is where we find the back-door to be: at the interface between the modem and the applications processor. We do consider the modem to be an “unknown” area that offers no guarantee at all regarding security, since it is running proprietary software. Hence, we believe it is relevant to assume the worse and consider it compromised and subject to remote control. Several indications tend to make us think this is actually what is going on: Craig Murray described how a mobile phone had been remotely converted to a spying device in Murder in Samarkand. Considering the recent revelations regarding the practices of several governments' intelligence agencies, we find it hard to believe there is no way modems cannot be remotely compromised.
The goal of our action was to make people aware of that particular issue. One might consider it to have no value, provided they don't think modems can be remotely compromised and others might see it as a crucial security flaw in the event the modem is compromised, as we do. The fact that it was implemented for another purpose or was not intended to be used in malicious ways doesn't change anything at all: an attacker with remote access to the modem will be able to issue the incriminated requests. There is no possible “false understanding”, in the way Samsung seems to imply here.
For the record, we didn't at any point intend to distort the truth to bring attention to our project or our research, nor did we intend to ruin Samsung's reputation. We simply felt it was our moral responsibility to spread the word about it. I believe anyone can decide for themselves whether they have faith in Samsung's good word that this introduces no further security risk, but let it be clear that it doesn't get any more certain than what good faith can provide.
We are still looking forward to working with Samsung to make things right, in case they decide to abandon their current position of denial.
I am currently working on writing a free software replacement for the Galaxy S3 camera module, based on the Exynos Camera module I wrote a couple months ago for the Galaxy S2. Both are using V4L2, but the implementation differs in details. Especially, the Galaxy S3's back camera, the Samsung S5C73M3, uses an interleaved format for picture capture.
As an interleaved format, there is no standard and readily-usable implementation to decode the data. After searching for a long time, all I could find was a commit by one of Samsung's developers that introduced that format to mainline, through a LinuxTV patch. First of all, I can't seem to understand why such a patch was accepted mainline given that there is no decoder implementation for that format out there. Moreover, the only camera chip that uses it, the S5C73M3, has a driver that was also accepted in mainline. It seems to me like it was blindly included and nobody cared so much about how it works in practice. Moreover, it seems that this camera chip is mostly found in the Galaxy S3, and I doubt anyone tested mainline on the Galaxy S3 to see whether the S5C73M3 driver works and gives appropriate results.
However, let's not complain too much, that patch gave me crucial info to understand how to properly extract YUV and JPEG from the interleaved data. For reference, here are the explanations given with the patch:
Two-planar format used by Samsung S5C73MX cameras. The first plane contains interleaved JPEG and UYVY image data, followed by meta data in form of an array of offsets to the UYVY data blocks. The actual pointer array follows immediately the interleaved JPEG/UYVY data, the number of entries in this array equals the height of the UYVY image. Each entry is a 4-byte unsigned integer in big endian order and it's an offset to a single pixel line of the UYVY image. The first plane can start either with JPEG or UYVY data chunk. The size of a single UYVY block equals the UYVY image's width multiplied by 2. The size of a JPEG chunk depends on the image and can vary with each line.
The second plane, at an offset of 4084 bytes, contains a 4-byte offset to the pointer array in the first plane. This offset is followed by a 4-byte value indicating size of the pointer array. All numbers in the second plane are also in big endian order. Remaining data in the second plane is undefined. The information in the second plane allows to easily find location of the pointer array, which can be different for each frame. The size of the pointer array is constant for given UYVY image height.
In order to extract UYVY and JPEG frames an application can initially set a data pointer to the start of first plane and then add an offset from the first entry of the pointers table. Such a pointer indicates start of an UYVY image pixel line. Whole UYVY line can be copied to a separate buffer. These steps should be repeated for each line, i.e. the number of entries in the pointer array. Anything what's in between the UYVY lines is JPEG data and should be concatenated to form the JPEG stream.
At first, I was only getting the first 0xA00000 bytes, which is in fact only the first plane. Hence, I couldn't find the offset to that pointers array (even though I could locate it manually). I had to enable embeded data with the V4L2_CID_EMBEDDEDDATA_ENABLE control. With that, the buffer gets 0x1000 more bytes: that's the second plane. Then by applying an offset of 4084 bytes to the start of that second plane, I could locate the offset to the pointers array.
Since I complained it was lacking, I wrote a reference implementation that separates the YUV (it's actually UYVY) and JPEG data from the interleaved format: s5c73m3_interleaved_decode.c.
2013-08-06 Update: As I sent an email to the Samsung developers involved in the mainline patch, I was given details on the format (that I already figured out though) as well as a C implementation to separate JPEG and UYVY. The developer also told me he is going to release sample code to decode the format, publicly. So I think things are going to be fine, and my criticism will soon no longer be valid. Yay!
While reversing the Galaxy Tab 2 sensors, I have been looking for a way to calculate the orientation vector from acceleration and magnetic field vectors: I've looked at any sensors implementation I could find and each time, this was being held into some proprietary component, to the point that the Galaxy Tab 2 has an user-space blob dedicated to this task (orientationd). Since I am not an expert at physics, I soon gave up on writing a free orientationd implementation, which was really a shame given the time I spent making the geomagnetic sensor work properly. I just realized that there was one last implementation I didn't look at, that is the free software user-space program for AKM8975. So many thanks to Asahi Kasei: I was able to reuse that code directly and it worked perfectly at first try. That's pretty amazing!
