tag:blogger.com,1999:blog-14169696561676592892024-03-18T09:34:01.875+03:00I hate softwareI hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.comBlogger40125tag:blogger.com,1999:blog-1416969656167659289.post-42931643542415076682022-11-03T07:50:00.004+03:002022-11-04T05:01:10.508+03:00Why CVE-2022-3602 was not detected by fuzz testing<p>So recently a very hyped memory corruption security vulnerability was discovered in the OpenSSL punycode parser.</p><p>Some folks including Hanno (https://twitter.com/hanno/status/1587775675397726209) asked why this is still happenning, why no one wrote a fuzzer for the punycode parser and if we as the security community have learned nothing from Heartbleed.</p><p>I think we should give the developers the benefit of doubt and assume they were acting in good faith and try to see what could be improved.</p><p>In fact, there already exists a fuzz testing harness for the X.509 in the OpenSSL source code.</p><p>All of the fuzzers from the OpenSSL source tree are also supposedly automatically deployed to ClusterFuzz via OSS-Fuzz: https://github.com/google/oss-fuzz/blob/master/projects/openssl/build.sh</p><h3 style="text-align: left;">Examining call chains</h3><p>Let’s start by examining the call chain for the vulnerable function.</p><p></p><ul style="text-align: left;"><li>ossl_punycode_decode: called by ossl_a2ulabel</li><li>ossl_a2ulabel: ossl_a2ucompare is not really referenced anywhere in C code, only mentioned in documentation.</li></ul><p></p><p>Let's examine who calls "ossl_a2ulabel" then.</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p>openssl/crypto$ grep -rI ossl_a2ulabel .</p><p>./x509/v3_ncons.c: if (ossl_a2ulabel(baseptr, ulabel + 1, &size) <= 0) {</p></blockquote><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1S5DXCOzxUNgBr4HEmkkzR2PJd-lA4NC4wHZ9FabJ5eZ4W34STScCbEOsCotD3p966G5ssLKujy4UGjQUlghOwOamBcVgX9ZMV4KT4C2HY89-jbaNxgL5drxt1LwmDoLrvm3U26sODL-D0ztSWfWj6aCmLzFg1jhlswx_osxvWYRaokVenMsnLP7b/s1352/punycode.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1352" data-original-width="1236" height="454" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1S5DXCOzxUNgBr4HEmkkzR2PJd-lA4NC4wHZ9FabJ5eZ4W34STScCbEOsCotD3p966G5ssLKujy4UGjQUlghOwOamBcVgX9ZMV4KT4C2HY89-jbaNxgL5drxt1LwmDoLrvm3U26sODL-D0ztSWfWj6aCmLzFg1jhlswx_osxvWYRaokVenMsnLP7b/w416-h454/punycode.png" width="416" /></a></div><br /><p> </p><p>Let's remember the name of this file and examine the coverage produced by the corpus shipped with the OpenSSL source for the X.509 fuzzing harness.</p><p></p><ul style="text-align: left;"><li>nc_email_eai <- nc_match_single <- nc_match <- NAME_CONSTRAINTS_check, NAME_CONSTRAINTS_check_CN</li><li>NAME_CONSTRAINTS_check, NAME_CONSTRAINTS_check_CN <- check_name_constraints in crypto/x509/x509_vfy.c</li><li>check_name_constraints <- verify_chain <- X509_verify_cert</li><li>X509_verify_cert: this one has A LOT of callers in the OpenSSL code, but was not reached by the fuzzing harness.</li><li>X509_verify_cert: (other ways to reach it are circular - looks like we have to call it directly): check_crl_path <- check_crl <- check_cert <- check_revocation <- verify_chain</li></ul><p></p><h3 style="text-align: left;">Examining coverage</h3><p>Here is what I did:</p><p></p><ul style="text-align: left;"><li>Compiled the fuzzing harness with coverage (added -fprofile-instr-generate -fcoverage-mapping) before -DPEDANTIC when building fuzzers.</li><li>Minimized the x.509 fuzzing corpus to speed up the next step:</li><ul><li>./fuzz/x509 -merge=1 fuzz/corpora/x509_min fuzz/corpora/x509</li></ul><li>Ran the executable on all input vectors. This is very slow because while parsing is fast, executable takes time to start up. One solution here could be to use the toolchain from OSS-Fuzz which replaces libFuzzer with a library which triages inputs somewhat like AFL persistent mode.</li><ul><li>for i in corpora/x509_min/*; do ./x509 $i; mv default.profraw $(basename $i).profraw; done</li><li>llvm-profdata-10 merge -sparse *.profraw -o default.profdata</li><li>llvm-cov-10 show --output-dir=cov_html --format=html -instr-profile=default.profdata x509</li></ul><li><b>Update (regarding the persistent mode/perf comment above)</b>: once you build the harness with coverage flags, there is no need to execute each input file separately, one can just use the "runs=N" option of libFuzzer:</li><ul><li>./x509 -runs=3000 ./corpora/x509_min </li></ul></ul><h3 style="text-align: left;">So, why did this all happen?</h3><p>My first (un)educated guess was: fuzzing will waste time in ASN.1 deserialization with little time spent on parsing decoded fields. Turns out, it's slightly worse.</p><p><b>Short answer: the code is not reachable by the current corpus and harness. As there exists an X.509 fuzzer, perhaps developers and other folks assumed it could theoretically reach all parsers, but this is not the case.</b></p><p>The file through which it’s reachable (v3_ncons.c) has little coverage.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZmPkdPrQbYrlvnkTUX8UUTAAtOO3dENjCITAYs2SVOR_l6X4llF5Ffmq5ZCO35qqujbzGnnOssAcAqEaxrJJqzntaRNduQ4bjgwFW6Lc0J_Fifps0Ew7j4MtRGFEyco21iIufDJId_09nS-CaqwInAlO41liCfWNlM2LQZGsDOV3uffbsdnJEACY1/s2197/x509_coverage_overall.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1594" data-original-width="2197" height="461" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZmPkdPrQbYrlvnkTUX8UUTAAtOO3dENjCITAYs2SVOR_l6X4llF5Ffmq5ZCO35qqujbzGnnOssAcAqEaxrJJqzntaRNduQ4bjgwFW6Lc0J_Fifps0Ew7j4MtRGFEyco21iIufDJId_09nS-CaqwInAlO41liCfWNlM2LQZGsDOV3uffbsdnJEACY1/w637-h461/x509_coverage_overall.png" width="637" /></a></div><p><br /></p><p>The specific call chain which we traced to "check_name_constraints" ends up in "crypto/x509/x509_vfy.c" which has ZERO coverage.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtwa7KZtjbOXt_4QIGAqKGkBFdfzDaVx3x5k6s6dl1biSESowF4xDI4c7Ee64_Z_LZ4kXbDQ0YUxYxKuVzRNHnExaXPLm4pQqyVDkdrLr3jv7w7EgTDQ7ElQ8kLYyP3dzT-odLjr9z2k9ThcqfXeFtt5qL3oGGSZMfxHgdDcSLnSL7e_c1Cine3NeO/s1817/x509_coverage_vfy.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="851" data-original-width="1817" height="310" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtwa7KZtjbOXt_4QIGAqKGkBFdfzDaVx3x5k6s6dl1biSESowF4xDI4c7Ee64_Z_LZ4kXbDQ0YUxYxKuVzRNHnExaXPLm4pQqyVDkdrLr3jv7w7EgTDQ7ElQ8kLYyP3dzT-odLjr9z2k9ThcqfXeFtt5qL3oGGSZMfxHgdDcSLnSL7e_c1Cine3NeO/w662-h310/x509_coverage_vfy.png" width="662" /></a></div><br /><h3 style="text-align: left;">(Update): "<span style="background-color: #f0f0f0; font-family: monospace; white-space: pre;">verify_chain" is reachable in other fuzzers, but it's still not enough.</span></h3><p>Jonathan Metzman from Google's Open Source Security Team pointed me to an OSS-Fuzz dashboard where "verify_chain" is reachable.</p><p></p><ul style="text-align: left;"><li>https://twitter.com/metzmanj/status/1588174176229199873</li><li>https://storage.googleapis.com/oss-fuzz-coverage/openssl/reports/20221031/linux/src/openssl/crypto/x509/x509_vfy.c.html#L221</li></ul><p></p><p>Unfortunately, this is still NOT enough:</p><p></p><ul style="text-align: left;"><li>"verify_chain" is covered 4.6K times whereas the inner loop of the X.509 fuzzer is invoked 12.4 times so it's reachable but NOT by the X.509 fuzzer.</li><li>Whichever harness reaches "verify_chain" (most likely the "server" ssl test but not the X.509 one) needs to be modified to either set up a valid certificate chain for verification or mark the certificate as self-signed so that "build_chain" does not return error</li><li>https://twitter.com/astarasikov/status/1588175841615261702</li></ul><h4 style="text-align: left;">(Update 2): making "verify_chain" and "build_chain" pass (still not there).</h4><div>I modified the X.509 test by adding some code from the "test_self_signed" function I took from "tests". With that, we can pass the "build_chain" and exercise most of "verify_chain". Unfortunately, name verification still requires a well-formed proxy certificate.</div><div><br /></div><div>I think the way to go could be to take the code from "test/sslapitest.c", function "test_client_cert_verify_cb", use the provided certificates as input vectors and fuzz them.</div><div>Ultimately, one needs to add a custom certificate to the untrusted chain and sign the certificate to be verified with it. As one can see, it's a lot of work which requires getting familiar with using OpenSSL.<br /><ul style="text-align: left;"><li>https://gist.github.com/astarasikov/4a60bb17499d4351bb27189e5e8ba8f4</li></ul><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYEcjTMwP1vI02UhmbQ8TLwSFZouIJiiYpOIubw5mwukwdG5D328yCC7p1zb4B75dtvVKRWKLM4umF6yNClDD0QZEjLu0ryQmdytsVqrXejjenwJXG_Ce5uUv9zTOpdO5RJWTGwXn8AJbIuLwkVHFOg9WDJkeMUtp2yoHquojdHB-MOItaqMuWZRWw/s1531/cert_working_but_no_proxy.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1419" data-original-width="1531" height="554" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYEcjTMwP1vI02UhmbQ8TLwSFZouIJiiYpOIubw5mwukwdG5D328yCC7p1zb4B75dtvVKRWKLM4umF6yNClDD0QZEjLu0ryQmdytsVqrXejjenwJXG_Ce5uUv9zTOpdO5RJWTGwXn8AJbIuLwkVHFOg9WDJkeMUtp2yoHquojdHB-MOItaqMuWZRWw/w598-h554/cert_working_but_no_proxy.png" width="598" /></a></div><br /><div><br /></div></div><p></p><h3 style="text-align: left;">What could we try improving?</h3><p></p><ul style="text-align: left;"><li>Write separate parsers for each function (like Hanno did) - for that, it'd be necessary to examine coverage to see 1. where coverage is low and 2. where the code processes decoded ASN.1 elements</li><li>Write a harness to cover X509_verify_cert. Looks like this is currently only called from "test" but not "fuzz" tests. While it may be slow to fuzz the verification, it will definitely cover a larger attack surface.</li><ul><li><b>Update: while this function is reachable via "client" and "server" tests, it returns early. To really cover it and the "punycode" parsing, it's necessary to set up certificate chains in these tests, as well as generate valid "proxy" certificates and add them to the corpus.</b></li></ul><li>Periodically examine coverage reports. This is a bit tedious to do manually, but if there was a public access to the OSS-Fuzz coverage report from Google, this would be much easier. Additionally, the OSS-Fuzz Introspector tool could be helpful in identifying roadblocks/unreachable code.</li></ul><p></p><p>Generally, the Introspector does not always work perfectly and is easy to break - it's a static analysis tool so it gets confused by function pointers and cannot infer anything that happens at runtime, like when your code is heavily using C++ classes or a hashtable for lookups. In case of X.509 code, however, it may work fine - function pointers are mostly used by the ASN.1 code for its internals (which we actualy do NOT want to fuzz or review in most cases) whereas the core top-level logic is reachable by direct function calls from the entrypoint - a good candidate for static analysis.</p><p></p><ul style="text-align: left;"><li>https://github.com/ossf/fuzz-introspector</li><li>https://openssf.org/blog/2022/06/09/introducing-fuzz-introspector-an-openssf-tool-to-improve-fuzzing-coverage/</li></ul><p></p><h4 style="text-align: left;">If we had the harness which could theoretically reach X509_verify_cert</h4><p></p><ul style="text-align: left;"><li>Add well-formed (encoded) ASN.1 elements to the dictionary file (-dict= option for libFuzzer). This one is currently not used by the OpenSSL fuzz/helper.py, but at least oids.txt is used by OSS-Fuzz as the dictionary.</li><li>Add well-formed X.509 certificates which make use of the "name constraints" field. And strictly speaking, all other fields too - instead of just storing the libFuzzer-generated corpus in the tree, it would be better to manually provide various inputs exercising difficult functionality. However, as libFuzzer is spending far too much time on ASN.1 and is overloaded with "features", this will likely only uncover new issues during long (days) runs on ClusterFuzz. Whereas parsers for individual leaf functions, as demonstrated by Hanno, can find (some) bugs in mere seconds.</li></ul><p></p><h2 style="text-align: left;">Thanks to:</h2><p>Hanno for his twitter thread for motivating me to look into this.</p><p>My colleagues for introducing me to the Introspector tool.</p><p>P.S. Linking to my evergreen tweet https://twitter.com/astarasikov/status/1122364899362054144</p>I hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com0tag:blogger.com,1999:blog-1416969656167659289.post-78767094643375481192021-08-13T01:31:00.004+03:002021-08-13T01:42:12.462+03:00building CLVK OpenCL support for Android phones + OpenCV notes<h3>
Compiling CLVK for Android.</h3>
<div>Many Android devices, especially Google Pixel, ship without the OpenCL library.</div><div>At some point I needed OpenCL for my OpenCV prototyping, and I was also interested in using either CU2CL or a similar project to run CUDA code.</div><div>Needless to say, as soon as I saw a project which promised to implement OpenCL on top of Vulkan, I decided to see if I can run it on Android.</div><div><br /></div><div>It worked fine, although my approach was kind of nasty: integrating the project along with its LLVM library into the app code. That was good enough for prototyping, although the debug binaries took a few hundred megabytes.</div><div><ul style="text-align: left;"><li>The original project: <a href="https://github.com/kpet/clvk">https://github.com/kpet/clvk</a></li><li>I put my notes here: <a href="https://github.com/astarasikov/clvk/blob/android_test/README_android.md">https://github.com/astarasikov/clvk/blob/android_test/README_android.md</a></li></ul></div>
I've added instructions to cross-compile this project for Android.<br />
Additionally, I wrote a simple Android app to demonstrate how to integrate the pre-built<br />
OpenCL library and how to deploy the OpenCL compiler ("clspv" binary) to the device.<br />
<br />
Two example OpenCL apps are compiled: "clinfo" which prints some basic information and "BitonicSort" from Intel OpenCL demos.<br />
FWIW both of them work so it's a good start.<br />
<a href="https://github.com/astarasikov/clvk/tree/android_test">https://github.com/astarasikov/clvk/tree/android_test</a><br />
<br />
Also, it might be curious to compare the run times for this app on the phone with CLVK and on the desktop with native OpenCL drivers and with CLVK + RADV.<br />
It seems that on desktop CLVK with RADV is 10 times slower than the native driver.<br />
<br />
However, since RADV or any other Vulkan driver uses most of the same LLVM-based codegen as the native OpenCL drivers, this difference is very likely caused by some hardcoded allocation size or another similar parameter rather than some major architectural issue.<br />
I have not looked into it yet for lack of time though.<br />
<br />
<ul>
<li><span style="font-family: inherit;">ARM Mali G72 MP18: <span style="background-color: white; color: #24292e; white-space: pre;">96.818924 ms</span></span></li><li><span style="font-family: inherit;"><span style="background-color: white; color: #24292e; white-space: pre;">Qualcomm Adreno 630: 33.18 ms</span></span></li>
<li><span style="font-family: inherit;"><span style="background-color: white; color: #24292e; white-space: pre;">AMD RX480 with CLVK and RADV: </span><span style="background-color: white; color: #24292e; white-space: pre;">7.491112 ms.</span></span></li>
<li><span style="font-family: inherit;">AMD RX480 with ROCm OpenCL driver: <span style="background-color: white; color: #24292e; white-space: pre;">0.651121 ms.</span></span></li>
</ul>
<br />
<a href="https://github.com/astarasikov/clvk/blob/android_test/results.txt">https://github.com/astarasikov/clvk/blob/android_test/results.txt</a><div><br /></div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3gwEnigJYRmaWlVThfI6DLh1QVyEaSJnBs9KtCzd7Sa4rQWY5BxNHC4mF5TzYBixOK4r4wFqs8g52jPoCZosJ2PaavG8a41S0LQzZqMw7qKWO5BBYC8tjr31H7pl07z55gdPcyh0m0QM/s2048/clvk_second.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1272" data-original-width="2048" height="398" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3gwEnigJYRmaWlVThfI6DLh1QVyEaSJnBs9KtCzd7Sa4rQWY5BxNHC4mF5TzYBixOK4r4wFqs8g52jPoCZosJ2PaavG8a41S0LQzZqMw7qKWO5BBYC8tjr31H7pl07z55gdPcyh0m0QM/w640-h398/clvk_second.png" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">OpenCL with CLVK on top of Android GL driver on a device with no OpenCL</td></tr></tbody></table><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhH0VAoEOnFCNAqGwBpH5ZxMTBTXme6Y31le9WRXh6B_ostlfqvvTFrrQ7nOEb6FrXhXPoqVT82ytywsSIgvpcXOztWUjbV2lkmCl-uex92QDgqblqT_bu_MetjXAYjKgar-UX42vkHxmk/s2048/clvk_first.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1395" data-original-width="2048" height="436" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhH0VAoEOnFCNAqGwBpH5ZxMTBTXme6Y31le9WRXh6B_ostlfqvvTFrrQ7nOEb6FrXhXPoqVT82ytywsSIgvpcXOztWUjbV2lkmCl-uex92QDgqblqT_bu_MetjXAYjKgar-UX42vkHxmk/w640-h436/clvk_first.png" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">OpenCL with CLVK on top of Android GL driver on a device with no OpenCL<br /><br /></td></tr></tbody></table><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjN2qhCmihTRX6fNCku8tsYgchUQUTwh8BQmbl_3AZbpnwv7d9B-6VIiKfnhtklTJjsqtucm0CFHlqtLtijveU53lPp2yebgfYiYo5nxAyoqUagLKdldKbBcLykiE51Qt81OfmmZ-D6V0Y/s1132/clvk_diff.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="311" data-original-width="1132" height="88" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjN2qhCmihTRX6fNCku8tsYgchUQUTwh8BQmbl_3AZbpnwv7d9B-6VIiKfnhtklTJjsqtucm0CFHlqtLtijveU53lPp2yebgfYiYo5nxAyoqUagLKdldKbBcLykiE51Qt81OfmmZ-D6V0Y/s320/clvk_diff.png" width="320" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Not a real fix, but that's enough to make most OpenCL samples run.</td></tr></tbody></table><br /><div><br />
<h3>
Building OpenCV with OpenCL support for Android.</h3>
For my personal project in 2016 I needed to check if it is possible to use the GPGPU accelerated version of OpenCV that is implemented using OpenCL on Android phones.<br />
<h3>
OpenCL SDK for Android.</h3>
Where to get the SDK? Welp. Build one yourself!<br />
To make OpenCV recognize our SDK and build successfully, we need the following things:<br />
<ul>
<li>OpenCV.mk - can be empty, but the OpenCV build system needs it to be present</li>
<li>Khronos OpenCL headers - can be gathered using OpenCL headers and CL-CPP SDK.</li>
<li>The loadable dynamic libraries - can be pulled from the device. Generally you can use one from any other device with the same architecture because the ABI and API is the same as it is defined by the OpenCL standard.</li>
</ul>
Here is how out "SDK" tree should look like. For the time being I only used the "armeabi-v7a" architecture, but one can also add the 64-bit binaries to the "arm64-v8a" directory. I've put up the "SDK" to GitHub, but only the header part (which are publicly available from Khronos). You will need to find the "libOpenCL.so" yourself. (If you're using Google Pixel with MSM8996, you can take the proprietary binaries from Xiaomi Mi5 or Zuk Z2 Pro).<br />
<br />
<div class="p1">
<span class="s1"><b>.</b></span></div>
<div class="p1">
<span class="s1"><b>├── OpenCV.mk</b></span></div>
<div class="p1">
<span class="s1"><b>├── include</b></span></div>
<div class="p1">
<span class="s1"><b>│ └── CL</b></span></div>
<div class="p1">
<span class="s1"><b>│ ├── cl.h</b></span></div>
<div class="p1">
<span class="s1"><b>│ ├── cl.hpp</b></span></div>
<div class="p1">
<span class="s1"><b>│ ├── cl_d3d10.h</b></span></div>
<div class="p1">
<span class="s1"><b>│ ├── cl_d3d11.h</b></span></div>
<div class="p1">
<span class="s1"><b>│ ├── cl_dx9_media_sharing.h</b></span></div>
<div class="p1">
<span class="s1"><b>│ ├── cl_dx9_media_sharing_intel.h</b></span></div>
<div class="p1">
<span class="s1"><b>│ ├── cl_egl.h</b></span></div>
<div class="p1">
<span class="s1"><b>│ ├── cl_ext.h</b></span></div>
<div class="p1">
<span class="s1"><b>│ ├── cl_ext_intel.h</b></span></div>
<div class="p1">
<span class="s1"><b>│ ├── cl_gl.h</b></span></div>
<div class="p1">
<span class="s1"><b>│ ├── cl_gl_ext.h</b></span></div>
<div class="p1">
<span class="s1"><b>│ ├── cl_platform.h</b></span></div>
<div class="p1">
<span class="s1"><b>│ ├── cl_va_api_media_sharing_intel.h</b></span></div>
<div class="p1">
<span class="s1"><b>│ └── opencl.h</b></span></div>
<div class="p1">
<span class="s1"><b>└── lib</b></span></div>
<div class="p1">
<span class="s1"><b> └── armeabi-v7a</b></span></div>
<style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 17.0px 'Ubuntu Mono'; color: #91db9f; background-color: #000000}
span.s1 {font-variant-ligatures: no-common-ligatures}
</style>
<br />
<div class="p1">
<span class="s1"><b> └── libOpenCL.so</b></span></div>
<br />
<h3>
Building OpenCV with OpenCL support.</h3>
<a href="https://gist.github.com/astarasikov/9088745a49401fced5f1a3503b07e593">https://gist.github.com/astarasikov/9088745a49401fced5f1a3503b07e593</a><br />
<br />
The most important change is to actually enable the OpenCL on Android in CMakeLists.txt:<br />
<span face="SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace" style="background-color: #ffeef0; color: #b31d28; font-size: 12px; white-space: pre;">OCV_OPTION(WITH_OPENCL "Include OpenCL Runtime support" NOT ANDROID IF (NOT IOS AND NOT WINRT) )</span><br />
<table class="highlight tab-size js-file-line-container" data-tab-size="8" style="background-color: white; border-collapse: collapse; border-spacing: 0px; box-sizing: border-box; color: #24292e; font-family: -apple-system, system-ui, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 14px; tab-size: 8;"><tbody style="box-sizing: border-box;">
<tr style="box-sizing: border-box;"></tr>
<tr style="box-sizing: border-box;"><td class="blob-code blob-code-inner js-file-line" id="file-opencv_android-diff-LC16" style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; overflow-wrap: normal; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre;"><span class="pl-mi1" style="background-color: #f0fff4; box-sizing: border-box; color: #22863a;"><span class="pl-mi1" style="box-sizing: border-box;">+</span>OCV_OPTION(WITH_OPENCL "Include OpenCL Runtime support" (NOT IOS AND NOT WINRT) )</span></td></tr>
<tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="17" id="file-opencv_android-diff-L17" style="box-sizing: border-box; color: rgba(27, 31, 35, 0.3); cursor: pointer; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; user-select: none; vertical-align: top; white-space: nowrap; width: 50px;"></td></tr>
</tbody></table>
I also had to disable some warning flags and unsupported compiler options in "OpenCVCompilerOptions.cmake"<br />
<br />
Compiler Options.<br />
Extra debugging.<br />
CMake options.<br />
<br />
Please also see the following blog which describes building OpenCV with OpenCL, although not the case when you don't have a ready-made SDK: <a href="http://www.ysagade.nl/2014/11/02/opencv-android-setup/">http://www.ysagade.nl/2014/11/02/opencv-android-setup/</a><br />
<h3>
OpenCL without root.</h3>
<div>
Many vendors who ship the phones with the official Android (that passes the CTS, compatibility test suite) do not ship the OpenCL drivers. Some vendors (most Chinese ones like such as Xiaomi and Zuk, and also Sony), however, do ship the drivers. If you have root on your phone or are building a custom firmware, you can just take the binaries from the other device's ROM and that's it.<br />
<h4>
Dynamic linking banned by Google?</h4>
Currently it seems that one can still use mmap() and mprotect() to write their own dynamic library loader, but this might get patched in future because Google is looking into both security and control.<br />
<br />
What could we do then?<br />
In principle, we could develop an application that would take a bunch of ".so" libraries and produce a single object file (.o) containing the data and code from all the libraries, with all dynamic symbols resolved. In other words, write a static compile-time linker.<br />
<br />
The ultimate obstacle to this approach would be if device vendors re-worked the driver architecture in such a way that the OpenCL frontend would not be loaded to the application address space but a separate server. If that happens, the only way forward would be using a device that ships the OpenCL driver or building a custom firmware.<br />
<br /></div>
Package all the relevant ".so" objects directly to your application APK.<br />
Paths. For this prototype, I manually edited the paths to the libraries using a hex editor. This part can be automated, but it was good enough for the proof of concept stage.<br />
Essentially packaging a good half of the other phone's firmware into your app :)<br />
<br />
<h4>
Tweaking the application makefiles.</h4>I edited Android.mk to specify the STL version and disable exceptions.<br /><h4>The fail.</h4>
<div>OpenCL library loaded fine, but now we had two independent EGL/OpenGL contexts - one from GL and one from CL with no standard way of sharing textures between them.</div><div>(2020 edit) In retrospect, I could have hooked "open" routine to steal KGSL file descriptor from GL for CL, but still there might have been some globals shared between libraries.</div>
<h4>
Zero-copy memory sharing.</h4>
It should be observed that all modern SoCs have the shared RAM for the CPU, GPU and other units (such as the camera frontend). Moreover, on more recent models, there is guaranteed cache coherency between the units (typically maintained by the AXI bus). Android used the kernel-side API called ION to manage memory buffers shared between the devices.<br />
<br />
It should be possible to map the GPU texture into the CPU memory using either the OpenGL extension or the underlying GraphicBuffer API which is a higher-level interface for ION. I may be looking into this option in future. For now, I've decided to use the phone that ships with the OpenCL driver.<br />
<br />
One other direction to explore might be using the Java API. I was able to create a GraphicBuffer from Java using reflection <a href="https://gist.github.com/astarasikov/2ebd216fcafa389174b58c6d9913e397">https://gist.github.com/astarasikov/2ebd216fcafa389174b58c6d9913e397</a> .<br />
Apparently John Carmack has managed to do it the other way round, using SurfaceTexture API ("<span face=""helvetica neue" , "helvetica" , "arial" , sans-serif" style="background-color: #f5f8fa; color: #14171a; font-size: 14px; white-space: pre-wrap;">SurfaceTexture -> Surface, pass the surface in an intent to the other process, turn that into an ANativeWindow -> EGLSurface") </span><span face=""helvetica neue" , "helvetica" , "arial" , sans-serif" style="color: #14171a;"><span style="font-size: 14px; white-space: pre-wrap;"><a href="https://twitter.com/ID_AA_Carmack/status/776099691586998272">https://twitter.com/ID_AA_Carmack/status/776099691586998272</a> </span></span>.</div><div><br /></div><div>EDIT 2021:</div><div>You really should just use ImageReader class, that's it.</div><div><a href="http://nezarobot.blogspot.com/2016/03/android-surfacetexture-camera2-opencv.html">http://nezarobot.blogspot.com/2016/03/android-surfacetexture-camera2-opencv.html</a></div><div><h3>
Further thoughts.</h3>
Sadly, it looks like there's not much we (the users/independent developers) can do to change the situation. Some projects like OpenCV indeed contain a lot of code that would be difficult to port to other languages, especially if you don't want to introduce additional bugs in the process.<br />
<br />
Perhaps as a developer the best plan is to limit your app to the phones that support OpenCL out of the box or build a custom firmware for the phone to add the drivers. While this limits your app to a very narrow set of devices, it will allow to re-use existing OpenCL code and build a working prototype quickly which is crucial for many projects at the early stage.<br />
<br />
P.S.<br />
This line is from the blog draft in 2017 before CLVK came out :)<br />
<br />
I think an interesting direction to explore would be to create the custom OpenCL/CUDA driver and runtime that would generate code in GLSL (for OpenCL ES with Compute Shaders) or Metal.</div>I hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com0tag:blogger.com,1999:blog-1416969656167659289.post-11160507681696597982020-08-05T17:22:00.000+03:002020-08-05T17:22:29.975+03:00SVE-2019-15230: A bug collision<div>Researchers from Team T5 recently published their write-up on exploiting a bug in S-Boot and obtaining code execution in the Samsung Secure Bootloader (S-Boot).</div><div>This week, they're going to present it at the BlackHat 2020 conference.</div><div>Their write-up contains a lot of technical details and I recommend you to read it.</div><div><a href="https://teamt5.org/en/posts/blackhat-s-talk-breaking-samsung-s-root-of-trust-exploiting-samsung-secure-boot/">https://teamt5.org/en/posts/blackhat-s-talk-breaking-samsung-s-root-of-trust-exploiting-samsung-secure-boot/</a></div><div><br /></div><div>In their report, they say "Another security research team found this vulnerability at the same time and report it to Samsung. ID: SVE-2019-15230".</div><div>This one-man security research team was me.</div><div>As I described in the previous two posts, in 2019 I got myself a Samsung Galaxy S10 phone with an Exynos SoC and decided to hunt for security bugs.</div><div>After finding the first issue (which is also my first SVE and my first report rated "Critical"), "SVE-2019-14371", I decided to carefully review the code around the location where I found the first bug.</div><div><br /></div><div>I found an integer overflow which could potentially lead to memory corruption, overriding the entirety of S-Boot code and data.</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXGS8qV8DRwcZFhyKCyd1hFNny54_usN1WXqaO2etyW0xMA09AovkK6M44xBegoipw9inMqRhJK5y6SKl5TioRptKsGkfurSZ9VPzM09QwOBLWWDCLSZ7GrBHevjChK6fFB9So6_FtICU/s2048/ghidra_code.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1280" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXGS8qV8DRwcZFhyKCyd1hFNny54_usN1WXqaO2etyW0xMA09AovkK6M44xBegoipw9inMqRhJK5y6SKl5TioRptKsGkfurSZ9VPzM09QwOBLWWDCLSZ7GrBHevjChK6fFB9So6_FtICU/s640/ghidra_code.png" width="640" /></a></div><div><br /></div><div><br /></div><div>Funnily enough, I got the SVE even though I have not submitted the POC which achieves code execution (well, to be fair, I reported it almost two months earlier).</div><div>I have submitted the one which demonstrates that the device handles non-underflowed values correctly whereas a "huge" buffer size causes it to freeze.</div><div>I came to the conclusion it's hard to exploit because I could not find a device with the good memory layout.</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNyj4VP-qQZCBqbbwEu1rOaC3jPWwTc7Pu9t5DHwO2cDa3TYoyLoPb-CBVfFuAmlUYCoka4SBQU_f01pOoh6yos9HfwdwKFH_UMZZcABUEWuVhZM7TwPlhMo1hlmKhWDZFFFhTgLB7nyc/s2048/sve_credit.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1063" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNyj4VP-qQZCBqbbwEu1rOaC3jPWwTc7Pu9t5DHwO2cDa3TYoyLoPb-CBVfFuAmlUYCoka4SBQU_f01pOoh6yos9HfwdwKFH_UMZZcABUEWuVhZM7TwPlhMo1hlmKhWDZFFFhTgLB7nyc/s640/sve_credit.png" width="640" /></a></div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyqkD1UGSs5Btrjy7IDlbEJu0Edy5sxkFmrXWbVmmaQKVeJgVckYnGxIVlg0OlHqvv30XdxJ_MWHA4LeXYR7CiyZ8jgTztIbvNaReSpUa0ijPsr-i97yC6-kiSPLzL79TjoDlPi3SQzOU/s1396/sve.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="407" data-original-width="1396" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyqkD1UGSs5Btrjy7IDlbEJu0Edy5sxkFmrXWbVmmaQKVeJgVckYnGxIVlg0OlHqvv30XdxJ_MWHA4LeXYR7CiyZ8jgTztIbvNaReSpUa0ijPsr-i97yC6-kiSPLzL79TjoDlPi3SQzOU/s640/sve.png" width="640" /></a></div><div><br /></div><div><br /></div><div>I have downloaded a ton of images: for S10, S9, S7, A50, J series phones.</div><div>Team T5's trick was to find a condition where the error handler code will make the memory layout good.</div><div>I have unfortunately overlooked that in S8 the download buffer is right before the S-Boot code AND it was not using the newer ("compressed" or "smp") download modes.</div><div>Since I never realized how to make that S-Boot falls back to the "legacy" buffer at 0xc0000000, I was focusing on the first underflow here, and came to the conclusion that there's not much I could do about it.</div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhnqNR8HL-dLYbMma7c46wQDN__OYHcT035C8_4xbwxykGua9H5XDtZlYctmbyZOmJexR4rxbGNVUyWqI1Jw8dNFj-J-9lzrytKDmNvBD99lfPzZn5iEobElKpmKG_citKZANe4OCj_0bA/s1351/giveup.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="249" data-original-width="1351" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhnqNR8HL-dLYbMma7c46wQDN__OYHcT035C8_4xbwxykGua9H5XDtZlYctmbyZOmJexR4rxbGNVUyWqI1Jw8dNFj-J-9lzrytKDmNvBD99lfPzZn5iEobElKpmKG_citKZANe4OCj_0bA/s640/giveup.png" width="640" /></a></div><div><br /></div><div>I wrote in my report to Samsung that USB transfer is done with DMA and I have not seen S-Boot initialize SMMU so it's surely exploitable.</div><div>If we could make the buffer point before S-Boot, of course (which I have not found out how to do).</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDHKDlFlc7YKKoryYyYD4wbGy2seZd8THkOFctoFKh3zt1RQMZn64M0DAcQSb0732d9Qx5w9leQ5BmyqMYskUeIgsq2vwLsF7KxrTsQRhBnj0tjE6qjfJB9ZlaJPU8L2HMKkc_OGrvzMY/s1343/dma.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="307" data-original-width="1343" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDHKDlFlc7YKKoryYyYD4wbGy2seZd8THkOFctoFKh3zt1RQMZn64M0DAcQSb0732d9Qx5w9leQ5BmyqMYskUeIgsq2vwLsF7KxrTsQRhBnj0tjE6qjfJB9ZlaJPU8L2HMKkc_OGrvzMY/s640/dma.png" width="640" /></a></div><div>I like how Team T5 discusses that normally it would be hard to exploit the bug as most code and data could be cached, but as they found a bunch of pointers in an uncached area, overwriting them works, even if it's done by the USB controller (which is not necessarily cache-coherent) and the CPU is unaware.</div><div><br /></div><div>Although it was already the third Critical issue I found in S-Boot by that time and I was completely burnt out on trying to develop yet another POC.</div><h3 style="text-align: left;">Is the security of Samsung phones bad? NO.</h3><div>I think, it's quite on par with the competitors. While the Android Security Updates page (https://security.samsungmobile.com/securityUpdate.smsb) regularly lists "High" and "Critical" issues, very few of them happen in the S-Boot bootloader (which is one of the earliest pieces of code which execute on the device). What it means is that the attacker need to first unlock the phone. So keeping your phone locked and BT/WLAN off should give a reasonable level of protection in the settings when you can't keep an eye on your phone.</div><div><br /></div><div>There are of course things that could be improved, like adding some mitigations to the bootloader (stack cookies, heap guard pages). However, in this case this would not help at all, because the memory copy (and thus overwriting code/data) is not done by the S-Boot code, but by the USB DMA controller.</div><div><br /></div><div>Bootloaders almost never implement ASLR, and even kernels which do only implement it for the virtual memory, the physical address remains constant or predictable. In fact, as it's overwriting uncached code (exception handlers), it could even work if the CPU supported ARM MTE. So this is in some sense the mother of all S-boot bugs.</div><div><br /></div><div>In some aspect, the root cause here is very similar to the IROM bug found by Frederic Basse: an integer overflow and copying data from USB, although in case of IROM it seems that IROM's code is copying USB data by small chunks.</div><div><a href="https://fredericb.info/2020/06/exynos-usbdl-unsigned-code-loader-for-exynos-bootrom.html#exynos-usbdl-unsigned-code-loader-for-exynos-bootrom">https://fredericb.info/2020/06/exynos-usbdl-unsigned-code-loader-for-exynos-bootrom.html#exynos-usbdl-unsigned-code-loader-for-exynos-bootrom</a></div><div><br /></div><div>Of course it would not be fair to judge the design decisions now that we know about these bugs. It would be nice to add some checks to ensure DMA regions don't intersect with code/data. Enabling SysMMU would not hurt. This is becoming somewhat worrisome now that USB4 has been announced with Thunderbolt-like DMA capabilities. It's unfortunate that most bootloaders do not focus on this.</div><div><br /></div><div>Now, the problem is that adding mitigations is quite hard as well as reasoning about their effectiveness. Without memory safety you can never be sure that the code is not exploitable. And as we've just seen before, it's unlikely that even a hardware safety oriented at memory safety are likely to be bypassed. Maybe a combination of MTE/KASAN and instrumenting all DMA memory management would work, but again it relies on the individual developers thinking of all corner cases. At some point bootloaders/firmwares could become as complex as the Linux kernel itself.</div><div>To this end, an interesting approach is moving firmware update, including USB download, to user-space and indeed running it from Linux, as Google started doing recently.</div><div><a href="https://source.android.com/devices/bootloader/fastbootd">https://source.android.com/devices/bootloader/fastbootd</a></div><div><br /></div><h3 style="text-align: left;">Why are the bugs present then?</h3><div>It often happens that bugs appear in two areas: the just-written code with new functionality (which not so many people had a chance to review) or the old code (people got tired of trying to find bugs there and gave up).</div><div><br /></div><div>I think we can draw two conclusions from this.</div><div><ul style="text-align: left;"><li>As a vendor, one should not assume that security or code review is a one-off effort and they need to re-review their stuff once in a while, especially bringing in the perspective of new team members.</li><li>As the user, if you expect maximum security, perhaps don't switch to the new tech immediately, give it 2-3 months for the most obvious and annoying issues (not only security bugs) to get ironed out.</li></ul></div><h3 style="text-align: left;">What can I do to protect myself?</h3><div>Enable the "Find My Mobile" feature of your Samsung phone.</div><div><br /></div><div>Before I and TeamT5 reported a few issues to Samsung, S-Boot implemented one mechanism of preventing the phone from being tampered with: when MDM (device administration by a workspace or school) is enabled, certain partitions were not allowed to be flashed.</div><div>This, unfortunately, did not protect the user in two scenarios: when they were not using MDM (which is the majoriy of users) or when the vulnerability is in the USB stack before the flashing code (in which case even enabling MDM would buy you nothing).</div><div>In mid-2019, Samsung changed the "Find My Mobile" in such way that it would disallow any USB operations (such as ODIN mode) when the device is not unlocked and FMM is enabled.</div><div>This should provide a reasonable level of protection for most users.</div><div>It is now much harder for an attacker to hack your phone in a few seconds when you're not looking at it (such as at a conference or in the hotel room). Of course, if is still possible to re-program the storage chip by desoldering it, which could expose other potential vulnerabilities, but it's hard to do it quickly and without tamper evidence.</div><h3 style="text-align: left;">Will you blog about other findings?</h3><div>Not sure. It's a tricky question.</div><div>In general, Samsung is asking to not disclose the issues, at least until the patching and reward process is done, which is understandable. I am grateful that Samsung allowed me to participate in their security reporting program, even though I work for another mobile SoC company, so I'm not particularly interested in making this relationship go sour.</div><div><br /></div><div>I do, however, see the immense value in describing the bug contents, because personally for me write-ups on Phrack or later Google Project Zero have been useful for understanding how attackers think, which came handy when both writing code and later working as a security engineer.</div><div><br /></div><div>I guess vendors can have their own reasons to not like when bugs are disclosed. Before working as a security engineer myself, I thought negative PR/news was the major reason. Turns out, it's impossible to predict this factor, and often the minor bugs are over-hyped but serious ones go unnoticed. So maybe this factor is not <b>that</b> important after all.</div><div><br /></div><div>Another thing I've noticed is that there is a significant interest in some hacking and "mobile repair" forums in bugs even for older firmware revisions, which means there are regions where people rarely update (expensive internet) or... a source of phones which for <b>some</b> reason remain unused and therefore not updated for a while.</div><div><br /></div><div>Given that mobile phones are supported (security updates and carrier contracts), maybe half of this term (1.5-2 years from the time the bug is patched) is a reasonable delay for holding off disclosure.</div><div><br /></div>I hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com8tag:blogger.com,1999:blog-1416969656167659289.post-88082777889029786872020-05-05T03:14:00.002+03:002020-05-05T03:53:29.111+03:00On Samsung and Exynos hacking, again<h2>
Introduction.</h2>
Last year I published a post (<a href="http://allsoftwaresucks.blogspot.com/2019/05/reverse-engineering-samsung-exynos-9820.html">http://allsoftwaresucks.blogspot.com/2019/05/reverse-engineering-samsung-exynos-9820.html</a>) about reverse-engineering TEEGRIS and S-Boot<br />
on Samsung Exynos Galaxy S10. This is kind of a follow-up to that post<br />
which has received a lot of attention and led to interesting conversations<br />
with fellow security researchers.<br />
<br />
Funnily enough, this very blog with its distinctive URL got into academic papers and<br />
conference talks.<br />
I guess that counts as a success because that's more citations than all my<br />
previous academic work combined. Slowly but steadily I'm progressing on track to receive my PhD from the Shitposting University.<br />
<h3>
Citations.</h3>
(All links have been retrieved on 2020-04-17).<br />
<a href="https://gsec.hitb.org/materials/sg2019/D2%20-%20Launching%20Feedback-Driven%20Fuzzing%20on%20TrustZone%20TEE%20-%20Andrey%20Akimov.pdf">https://gsec.hitb.org/materials/sg2019/D2%20-%20Launching%20Feedback-Driven%20Fuzzing%20on%20TrustZone%20TEE%20-%20Andrey%20Akimov.pdf</a><br />
Andrey Akimov: Launching feedback-driven fuzzing on TrustZone TEE (HITB GSEC 2019 Singapore).<br />
<br />
<a href="https://zeronights.ru/wp-content/themes/zeronights-2019/public/materials/5_ZN2019_andrej_akimovLaunching_feedbackdriven_fuzzing_on_TrustZone_TEE.pdf">https://zeronights.ru/wp-content/themes/zeronights-2019/public/materials/5_ZN2019_andrej_akimovLaunching_feedbackdriven_fuzzing_on_TrustZone_TEE.pdf</a><br />
Andrey Akimov : Launching Feedback-Driven Fuzzing on TrustZone TEE (ZeroNights 2019)<br />
<br />
<a href="https://blog.quarkslab.com/a-deep-dive-into-samsungs-trustzone-part-1.html">https://blog.quarkslab.com/a-deep-dive-into-samsungs-trustzone-part-1.html</a><br />
Alexandre Adamski, Joffrey Guilbon, Maxime Peterlin of Quarkslab : A Deep Dive Into Samsung's TrustZone (Part 1)<br />
<br />
<a href="https://www.usenix.org/system/files/sec20summer_harrison_prepub.pdf">https://www.usenix.org/system/files/sec20summer_harrison_prepub.pdf</a><br />
Lee Harrison, Hayawardh Vijayakumar, Rohan Padhye , Koushik Sen , and Michael Grace: PARTEMU: Enabling Dynamic Analysis of Real-World TrustZone Software Using Emulation<br />
<br />
<a href="https://www.ndss-symposium.org/wp-content/uploads/2020/04/bar2020-23014.pdf">https://www.ndss-symposium.org/wp-content/uploads/2020/04/bar2020-23014.pdf</a><br />
Marcel Busch and Kalle Dirsch : Finding 1-Day Vulnerabilities in Trusted Applications using Selective Symbolic Execution<br />
<h3>
Follow-up on reverse-engineering and security research.</h3>
I also found a few bugs in TEEGRIS and S-Boot that got assigned CVEs by Samsung (check 2019/2020 <a href="https://security.samsungmobile.com/securityUpdate.smsb">here</a>).<br />
I'm somewhat happy about this achievement. Prior to that, I mostly worked on<br />
the defense side both implementing mitigations/OS kernels and then debugging<br />
security issues submitted by other researchers. So I was glad to receive this<br />
external validation of my ability to find bugs on my own, although a little<br />
bit surprised at how easy it was to find them with the code review of the<br />
code decompiled with Ghidra.<br />
<br />
I have not really found any bugs with fuzzing using the QEMU emulators for<br />
S-Boot and TEEGRIS described in my previous blog post. However, these came<br />
handy for debugging proof-of-concepts as I could use GDB and dump memory as if<br />
it was just a regular Linux app on the PC.<br />
<br />
I would also like to point your attention to this paper on Phrack<br />
about emulating RKP (Samsung Hypervisor) with QEMU by Aris Thallas.<br />
<a href="http://phrack.org/papers/emulating_hypervisors_samsung_rkp.html">http://phrack.org/papers/emulating_hypervisors_samsung_rkp.html</a><br />
<br />
I have used a similar approach with full-system QEMU emulation for debugging some RKP bugs.<br />
However, after having spent so much effort on emulating S-Boot and TEEGRIS,<br />
I was not in the mood to boot Linux in EL1 and put all the pieces together.<br />
I used a different approach for testing Hypervisor Calls (HVCs). Instead<br />
of having a proper EL1 client, I wrote a piece of C code that invoked the<br />
EL2 exception handler directly. I then linked it to the address of some<br />
uninteresting function in RKP and used GDB to overwrite the code in QEMU<br />
memory and jump to my stub.<br />
<br />
I especially like the part about using QEMU instrumentation to provide<br />
coverage information to AFL.<br />
I have also implemented a similar approach (based on the QEMU and Unicorn modes<br />
from the AFL source tree) for my TEEGRIS QEMU emulator.<br />
<a href="https://github.com/astarasikov/qemu/commits/teegris_usermode_persist_rewriteafl">https://github.com/astarasikov/qemu/commits/teegris_usermode_persist_rewriteafl</a><br />
<a href="https://twitter.com/astarasikov/status/1187902865710428160">https://twitter.com/astarasikov/status/1187902865710428160</a><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpJpAVmSEgZnveYZhDbBf6690Izu78IWqwq9ZiGMJbeETSecVqzq1KqHY-qiHU23chUDNK0bEAd0RHNKUlnfPL0RnPP6xgxnf4va2YEjKYOIRIT5koNo6cH-9WM4qgTxXV_12sxwWdvHI/s1600/teegris_afl_persistent.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1489" data-original-width="1213" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpJpAVmSEgZnveYZhDbBf6690Izu78IWqwq9ZiGMJbeETSecVqzq1KqHY-qiHU23chUDNK0bEAd0RHNKUlnfPL0RnPP6xgxnf4va2YEjKYOIRIT5koNo6cH-9WM4qgTxXV_12sxwWdvHI/s320/teegris_afl_persistent.png" width="260" /></a></div>
<br />
<br />
Unfortunately, I have not found any bugs with fuzzing (although I have with code review).<br />
I believe better results could be achieved with the CompareCoverage plugin which<br />
would prevent the fuzzer from getting stuck on magic values/constants.<br />
<a href="https://andreafioraldi.github.io/articles/2019/07/20/aflpp-qemu-compcov.html">https://andreafioraldi.github.io/articles/2019/07/20/aflpp-qemu-compcov.html</a><br />
Additionally, please check out this blog about implementing ASAN (Address Santiizer)<br />
for binary-mode QEMU within the TCG interpreter/JIT.<br />
<a href="https://andreafioraldi.github.io/articles/2019/12/20/sanitized-emulation-with-qasan.html">https://andreafioraldi.github.io/articles/2019/12/20/sanitized-emulation-with-qasan.html</a><br />
<br />
Finally, if you're interested in fuzzing at the source-code level and are<br />
getting stuck with magic values/constants, please check out this<br />
post from 2016 about a strategy for splitting comparisons (which is related<br />
to CompareCoverage).<br />
<a href="https://lafintel.wordpress.com/">https://lafintel.wordpress.com/</a><br />
<br />
This is already implemented in libFuzzer, but<br />
if you have to use AFL, consider using AFL++ which maintains LLVM plugins<br />
for these strategies. In any case, check out AFL++ because it attempts to unify<br />
most of the forks developed in academia.<br />
<a href="https://github.com/AFLplusplus/AFLplusplus">https://github.com/AFLplusplus/AFLplusplus</a><br />
<h2>
Other interesting news.</h2>
<h3>
I9100 (Samsung Galaxy S2) upstream work.</h3>
I was surprised when I got a GitHub notification in 2020 about a project I have<br />
not worked on before. Turns out, people have been resurrecting the work I've<br />
done in back 2012 which was a nice surprise.<br />
<br />
In 2012 I was doing some work on getting FOSS software to run on<br />
the Samsung Galaxy S2 phone. It was a hobby project, I got this phone<br />
after completing my work on porting Linux and Android to Sony Xperia X1 and<br />
hoped that starting with a device which ran Linux out of the box would be<br />
advantageous for this goal.<br />
<br />
So the first problem that I solved was getting multi-boot working.<br />
I solved it by porting the U-Boot bootloader.<br />
This eventually related in a weird chain of events that landed me several interesting<br />
jobs and gigs.<br />
<br />
Anyway, the u-boot.<br />
<br />
<ul>
<li><a href="https://forum.xda-developers.com/galaxy-s2/general/uboot-bootloader-true-multiboot-t1680898">https://forum.xda-developers.com/galaxy-s2/general/uboot-bootloader-true-multiboot-t1680898</a></li>
<li><a href="https://forum.xda-developers.com/galaxy-nexus/development/bootloader-boot-multi-boot-support-t2201146">https://forum.xda-developers.com/galaxy-nexus/development/bootloader-boot-multi-boot-support-t2201146</a></li>
<li><a href="https://github.com/astarasikov/i9100-uboot">https://github.com/astarasikov/i9100-uboot</a></li>
<li><a href="https://github.com/astarasikov/uboot-tuna">https://github.com/astarasikov/uboot-tuna</a></li>
</ul>
<br />
I then attempted porting the Galaxy S2 board support to the mainline kernel tree.<br />
I was using the latest Linaro tree. I had some limited success in getting most<br />
hardware working with upstream drivers (WIFI, Camera with V4L2) and by porting<br />
some non-upstream ones (Sound, Modem).<br />
<a href="https://github.com/astarasikov/i9100-proper-linux-kernel/commits/i9100_linaro_33">https://github.com/astarasikov/i9100-proper-linux-kernel/commits/i9100_linaro_33</a><br />
<br />
Eventually I had to resort to using the Android kernel with some changes<br />
but I got dual-boot working with Ubuntu on the SD Card.<br />
<br />
Native Ubuntu (with X11) on Samsung Galaxy S2 (2012)<br />
<a href="https://www.youtube.com/watch?v=VHl8PytVt50">https://www.youtube.com/watch?v=VHl8PytVt50</a><br />
<br />
Back in 2012 I made a post to summarize my efforts related to S2.<br />
<a href="https://www.mail-archive.com/smartphones-userland@linuxtogo.org/msg02865.html">https://www.mail-archive.com/smartphones-userland@linuxtogo.org/msg02865.html</a><br />
<h3>
Mainline linux port by Sekil</h3>
Fast-forward to 2020, I was surprised to learn that not only people are still<br />
using the device, they are also using my U-Boot port and one developer even<br />
went as far as resurrecting the attempts to run mainline linux tree.<br />
They made great progress and independently authored patches for the mainline<br />
tree which have a high chance of being accepted.<br />
<br />
See this port by Evgeniy Stenkin.<br />
<br />
<ul>
<li><a href="https://forum.xda-developers.com/galaxy-s2/general/port-run-mainline-linux-kernel-t3901190">https://forum.xda-developers.com/galaxy-s2/general/port-run-mainline-linux-kernel-t3901190</a></li>
<li><a href="https://github.com/Sekilsgs2/i9100_kernel_mainline_port">https://github.com/Sekilsgs2/i9100_kernel_mainline_port</a></li>
<li><a href="https://github.com/Sekilsgs2/i9100-uboot">https://github.com/Sekilsgs2/i9100-uboot</a></li>
<li><a href="https://lkml.org/lkml/2020/3/18/1160">https://lkml.org/lkml/2020/3/18/1160</a></li>
</ul>
<br />
This effort is acknowledged and is used by the PostmarketOS project.<br />
<a href="https://wiki.postmarketos.org/wiki/Samsung_Galaxy_SII_(samsung-i9100)">https://wiki.postmarketos.org/wiki/Samsung_Galaxy_SII_(samsung-i9100)</a><br />
<h3>
FOSS RIL for Samsung Galaxy S2, Galaxy Nexus</h3>
Later, my focus switched to reverse-engineering the userspace libraries<br />
in order to provide a fully open-source build of Android for Samsung Galaxy Nexus,<br />
a device which shared the modem with Galaxy S2.<br />
<br />
For the previous-generation phone (Galaxy S1, I9000) an open-source implementation<br />
of the Radio Interface Layer (RIL) was provided by the engineers from the Replicant<br />
and OpenMoko projects (Paul Kocialkowski, Simon Busch and morphis).<br />
<br />
In 2012 I was asked by Ksys Labs to provide an open-source RIL for Samsung<br />
Galaxy Nexus which happened to have the same modem as Galaxy S2.<br />
So I have done the following:<br />
<br />
<ul>
<li>Firmware loader for these modems (based on reverse-engineering and a C++ implementation by another engineer)</li>
<li>Fixing SMS character encoding so that we could receive SMS in Russian</li>
<li>Fixing some edge cases for USSD support</li>
<li>Providing some rudimentary socket callback protocol so that a proprietary GPS library could be used by those who really wanted to.</li>
</ul>
<br />
These changes have been fully integrated into the Replicant project<br />
and served as the basis for supporting many more Samsung modems.<br />
Some builds of LineageOS for Galaxy S3 also use these libraries from the<br />
Replicant project to avoid the overhead of supporting the ABI for the<br />
proprietary driver libraries from 2012.<br />
<br />
<ul>
<li><a href="https://git.replicant.us/replicant/hardware_replicant_libsamsung-ipc/">https://git.replicant.us/replicant/hardware_replicant_libsamsung-ipc/</a></li>
<li><a href="https://github.com/astarasikov/libsamsung-ipc">https://github.com/astarasikov/libsamsung-ipc</a></li>
<li><a href="https://github.com/astarasikov/samsung-xmm6260-fw-loader">https://github.com/astarasikov/samsung-xmm6260-fw-loader</a></li>
</ul>
<br />
I even saw the Replicant stand at the CCC last year so these phones<br />
are living on.<br />
And the dream of supporting it in a non-Android setting such as Ofono<br />
seems to have never materialized. Oh well.<br />
<h3>
Summary</h3>
I am happy to see that my work on both U-Boot and RIL got reused by many projects.<br />
Back in 2012 having your phone run upstream software was a very ambitious goal,<br />
especially for a single developer. It usually took around a year and a half<br />
to get familiar with all the hardware and reverse-engineer it to a decent level<br />
in order to develop all the support by which time the device would get obsolete.<br />
However, if you're more interested in using upstream SW than using the latest<br />
HW, there is some hope.<br />
<br />
Oh, and Pinephone looks like a nice alternative these days. The hardware is similar to Galaxy S2, but the CPU is 64-bit and it's FOSS out of the box.<br />
<h3>
U-Boot without the proprietary bootloader.</h3>
Here's another interesting development that happened in those years to another<br />
related Exynos device (Galaxy S3 I9300).<br />
Simon Shields ported U-Boot to Galaxy S3, but unlike my port this one<br />
does not rely on the Samsung bootloader in any way and allows to boot the phone<br />
with even fewer proprietary components.<br />
https://blog.forkwhiletrue.me/posts/u-boot-on-galaxy-s3/<br />
<br />
Back when I was porting U-Boot to S2, I flashed it into the Linux kernel<br />
partition and made it so that it's loaded by the phone's original bootloader.<br />
My motivation was to avoid bricking the device (back when it was not known<br />
how to use Exynos USB recovery mode) and it was assumed that the bootloader<br />
needed to be signed. As it turned out later around 2014, on these early<br />
Exynos chips the initial bootloader shared the same signing key and device<br />
ID with development boards and it was possible to work around the signing<br />
requirement and replace the original bootloader by using the stage-1 bootloader<br />
from a development board.<br />
<h2>
KVM on the phone.</h2>
Ever since working on the ARM para-virtualization with L4/Genode I wanted<br />
to use real virtualization.<br />
I was very enthusiastic about the first (32-bit) ARM boards with the HYP extension<br />
when they arrived in 2013.<br />
<a href="http://allsoftwaresucks.blogspot.com/2013/11/kvm-on-arm-cortex-a15-omap5432-uevm.html">http://allsoftwaresucks.blogspot.com/2013/11/kvm-on-arm-cortex-a15-omap5432-uevm.html</a><br />
<br />
Since then, I've always wanted to get virtualization working on a mobile phone<br />
for the fun of running multiple operating systems.<br />
Unfortunately, most of them enable "secure" booting and require that the EL2<br />
hypervisor image is signed by the OEM.<br />
<br />
Some early phones did not implement a hypervisor or left it writable by the OS<br />
but I was wondering if I could do that on a fairly recent and powerful phone.<br />
<br />
Here's some small showcase of an attempt to run Windows 10 in KVM on a Samsung<br />
A50 phone with the Exynos9610 CPU.<br />
<br />
The bug I found works only on the unlocked phone (with KNOX tripped/fuse blown) before Linux MMU<br />
is on. In principle one might be able to find a variant that works with MMU on,<br />
but even passing arbitrary arguments to RKP would require compromising (rooting)<br />
Linux first. Therefore, this bug does not (IMHO) have a big security impact<br />
(because on older generation Exynos RKP/EL2 was only used for the kernel<br />
memory protection and ROPP/JOPP but not for IOMMU) but is interesting for research purposes.<br />
<br />
This is in no way a statement on the security of Samsung devices. I think<br />
their efforts are definitely above average for Android. However, given enough time<br />
any system can be broken, even the ones previously regarded as unbreakable such<br />
as PS4 or iPhone with PAC. Here, patching timely before the issues get disclosed<br />
is important and looks like things have improved a lot in the Android world recently.<br />
<a href="https://www.zdnet.com/article/android-oem-patch-rates-have-improved-with-nokia-and-google-leading-the-charge/">https://www.zdnet.com/article/android-oem-patch-rates-have-improved-with-nokia-and-google-leading-the-charge/</a><br />
<br />
The bug has been patched in October 2019 anyway so users with the latest updates<br />
should not be affected (SVE-2019-15221, SVE-2019-15143).<br />
<br />
What I've also learnt from watching a lot of talks and following the discussions<br />
by other researchers is that security issues often concentrate in two areas:<br />
where no one has looked before, and where many people have looked and then gave<br />
up because they decided that they found all the low-hanging fruits. So RKP seemed<br />
like an interesting target given the previous research from Google Project Zero<br />
in 2017 (<a href="https://googleprojectzero.blogspot.com/2017/02/lifting-hyper-visor-bypassing-samsungs.html">https://googleprojectzero.blogspot.com/2017/02/lifting-hyper-visor-bypassing-samsungs.html</a>).<br />
<br />
I will not be providing additional details on that bug but here are some nice<br />
screenshots and videos:<br />
<br />
Ubuntu X11 running on Samsung Galaxy A50. KVM guest runs Windows 10.<br />
Here, we can see that the colors are swapped as the framebuffer driver is confiruged<br />
to output BGR instead of RGB by default in Android.<br />
<br />
Video of UEFI booting Windows 10 installer in KVM.<br />
<a href="https://twitter.com/astarasikov/status/1249904283098796033">https://twitter.com/astarasikov/status/1249904283098796033</a><br />
<br />
A mysterious BSOD (yes it's actually supposed to be blue) in the USB controller<br />
driver, possibly related to how the controller is emulated in QEMU.<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEyP1jtxHY09xTTLqAiJQVCrb6gvEXbI0wjku5Kjj_h9R5tt29k109e0NXRvnVLZxWWAS8NsLMlentc-az3XMRDbkQcg2UUbD9GAPU0juuLemviO2Mg6NZDoBJca7SSxL7LXZhXshvD1U/s1600/a505f_kvm_w10_bsod.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1594" data-original-width="1196" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEyP1jtxHY09xTTLqAiJQVCrb6gvEXbI0wjku5Kjj_h9R5tt29k109e0NXRvnVLZxWWAS8NsLMlentc-az3XMRDbkQcg2UUbD9GAPU0juuLemviO2Mg6NZDoBJca7SSxL7LXZhXshvD1U/s320/a505f_kvm_w10_bsod.jpg" width="240" /></a></div>
<br />
<br />
Unfortunately for now I had to stop further work on this project because I accidentally<br />
upgraded the phone to the latest firmware revision and now due to the rollback protection<br />
I can no longer install the vulnerable RKP image.<br />
<br />
If you're interested in this kind of stuff, there are good news.<br />
Recently a few open-source phones have appeared which do not enforce secure boot/<br />
signature verification and you can run KVM (or any other hypervisor) out of the box.<br />
<br />
For example, multiple people have reported getting KVM and Windows 10 working<br />
on the Pinephone and Pinebook.<br />
Pinephone has a Cortex-A7 CPU with an old Mali GPU so in terms of hardware<br />
it's almost an exact copy of the Galaxy S2 discussed above, but it's more<br />
FOSS-friendly.<br />
<br />
<a href="https://twitter.com/RealDanct12/status/1231607283412426756">https://twitter.com/RealDanct12/status/1231607283412426756</a><br />
<a href="https://twitter.com/Manawyrm/status/1197981073101271040">https://twitter.com/Manawyrm/status/1197981073101271040</a><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiy8sM_awRBCW6lJE-RlSODgXvKHMtXg1ZkPPbMZCOzu-V7Qa46-EHU4e_Ngw89W3tUh6MSQ6woUlH_8LGFina16hcdMNMeWzWPWz8gTnwm7uOMlWvPtMuqVKxWM0VCdO63YEjHLPJAd0c/s1600/pinephone1.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="900" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiy8sM_awRBCW6lJE-RlSODgXvKHMtXg1ZkPPbMZCOzu-V7Qa46-EHU4e_Ngw89W3tUh6MSQ6woUlH_8LGFina16hcdMNMeWzWPWz8gTnwm7uOMlWvPtMuqVKxWM0VCdO63YEjHLPJAd0c/s320/pinephone1.jpeg" width="180" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLTqndzhvSWw2Hjrp6_HsYlVqJCrmiNCUp6UkbvEW2JGzdWOZIu_hZ5scAUX06V013ayM3rtLjmQ0elda95sxSKf6KRKDs_MBQo7LxAYTiqn9NcK2wMBCMV5GYYAvKn5t1r5yvXIvJhM0/s1600/w10_pinephone.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1248" data-original-width="1200" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLTqndzhvSWw2Hjrp6_HsYlVqJCrmiNCUp6UkbvEW2JGzdWOZIu_hZ5scAUX06V013ayM3rtLjmQ0elda95sxSKf6KRKDs_MBQo7LxAYTiqn9NcK2wMBCMV5GYYAvKn5t1r5yvXIvJhM0/s320/w10_pinephone.jpg" width="307" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiczGazZJj9fe7SFkm8L-hJpBHPDTBWgLTFdPvyHkiQy3uG2f2dsg3G6T-ePYVyznHPIPEMkZZRbfb90EnCJAhRRviC9q9axz-qdT62XtNIUGKhNYzc5hLmR1zcUWdqXjbK0ccVTQ4sqCY/s1600/w10_pinebook.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1408" data-original-width="1204" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiczGazZJj9fe7SFkm8L-hJpBHPDTBWgLTFdPvyHkiQy3uG2f2dsg3G6T-ePYVyznHPIPEMkZZRbfb90EnCJAhRRviC9q9axz-qdT62XtNIUGKhNYzc5hLmR1zcUWdqXjbK0ccVTQ4sqCY/s320/w10_pinebook.jpg" width="273" /></a></div>
I hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com12tag:blogger.com,1999:blog-1416969656167659289.post-6592891377134371572019-05-29T05:04:00.002+03:002019-05-29T05:04:26.376+03:00Reverse-engineering Samsung Exynos 9820 bootloader and TZ<h2>
Reverse-engineering Samsung S10 TEEGRIS TrustZone OS</h2>
It's been a while since my last post, huh?<br />
Even though I have quite a lot of stuff I'm planning to write about, time is very limited.<br />
<br />
Lately I've been working on reverse engineering and documenting<br />
the S-Boot bootloader and TrustZone OS from the Exynos version<br />
of Samsung Galaxy S10.<br />
TLDR: I can now run S-Boot and TEEGRIS TrustZone TAs in QEMU but too lazy to find bugs.<br />
<br />
It's been a while since I had a Samsung phone, my last was Galaxy S2.<br />
It's also been a while since I last looked into bootloader binaries.<br />
<br />
Last year I got an Exynos S9 model, mostly because I was impressed by its<br />
CPU benchmark scores and wanted to run my own code to measure it.<br />
This year I got some spare time but since S10 came out and a lot of people<br />
have already looked at S9 software, I've decided to start reverse engineering<br />
the software from S10.<br />
<br />
<h3>
S-Boot bootloader image layout.</h3>
<a href="https://gist.github.com/astarasikov/f47cb7f46b5193872f376fa0ea842e4b#file-exynos9810_sboot_layout-txt">github gist</a><br />
<br />
<br />
<ul>
<li>0x0: probably EPBL (early primitive bootloader) with some USB support</li>
<li>0x13C00: ACPM (Access Control and Power Management?)</li>
<li>0x27800: some PM-related code</li>
<li>0x4CC00: some tables with PM parameters</li>
<li>... -> either charger mode code or PMIC firmware</li>
<li>0xA4000: BL2, the actual s-boot</li>
<li>0x19E000: TEEGRIS SPKG (CSMC)</li>
<li>0x19E02B: TEEGRIS SPKG ELF start (cut from here to load into the dissasembler). This probably stands for "Crypto SMC" or "Checkpoint SMC". This handles some SMC calls from the bootloader as part of Secure Boot for Linux.</li>
<li>0x1ACE00: TEEGRIS SPKG (FP_CSMC)</li>
<li>0x1ACE2B: TEEGRIS FP_CSMC (ELF header). My guess is that it's related to the Fingerprint sensor because all it does is set some registers in the GPIO block and USI block (whatever it is).</li>
<li>0x264000: TEEGRIS kernel, relocate to 0xfffffffff0000000 to resolve relocations</li>
<li>0x29e000: EL1 VBAR for TEEGRIS kernel. fffffffff0041630: syscall table, first entry is zero.</li>
<li>0x2D4000: startup_loader package</li>
<li>0x2D4028: startup_loader ELF start. This one's invoked by S-Boot to read the TEEGRIS kernel either from Linux kernel via shared memory or from the LZ4 archive compiled into S-Boot.</li>
</ul>
<br />
There's also one encrypted region containing ARM Trusted Firmware which is EL3 monitor code. It's right after the bunch of Rijndael substitution box constants.<br />
<h3>
Running S-Boot in QEMU.</h3>
I've long wanted to run S-Boot in QEMU for reverse engineering it.<br />
I think I've mentioned this idea to my colleague Fred 2 years ago which kind of motivated him to write this great post about Exynos4210 early bootloader in SROM.<br />
Check out his blog if you're interested in Samsung, btw.<br />
https://fredericb.info/2018/03/emulating-exynos-4210-bootrom-in-qemu.html<br />
<br />
Long story short, with a bit of hacks to emulate some MMIO peripherals I've prepared the patch for QEMU to run S-Boot from Exynos9820.<br />
<a href="https://github.com/astarasikov/qemu/commit/a2d3780055a37d07105c43deec97c59d38106c39#diff-8a8e18df006d4278f34711f0ef6a4041R1900">QEMU Support for Exynos9820 S-Boot</a><br />
<h4>
SCTLR_EL3 register</h4>
According to ARM ARM, top half of SCTLR is Undefined.<br />
Samsung reused them to store the base address for the S-Boot bootloader.<br />
When running in EL3, part of SCTLR is used when computing the value to write to VBAR registers which point to the Exception Table.<br />
I initially attempted running S-Boot in EL3 but it checks EL at runtime and I believe it's actually running at EL1 but the binary supports EL1, EL2 and EL3.<br />
<h4>
Re-enabling debugging prints</h4>
Turns out, early in the boot process the bootloader disables most of the debugging logging.<br />
I've prepared the GDB script to work around that.<br />
gdbscript<br />
set *(int*)0x8f16403c = 0<br />
<h4>
UART</h4>
<a href="https://github.com/astarasikov/qemu/blob/exynos9820/hw/arm/virt.c#L1900">https://github.com/astarasikov/qemu/blob/exynos9820/hw/arm/virt.c#L1900</a><br />
As usual (WM5 blog [http://allsoftwaresucks.blogspot.com/2016/10/running-without-arm-and-leg-windows.html]), we can solve it by making the MMIO Read request return different data on subsequent reads.<br />
We simply invert the value in cache on each invokation.<br />
Using this trick we can bypass busy loops which wait for some bits to be set or cleared.<br />
<br />
In fact, emulating two UART registers, status and TX, is enough to get debugging output from the bootloader.<br />
<br />
<h4>
Peripherals</h4>
We can identify some peripherals either by looking up their addresses in Linux Device Tree files<br />
or by analysing what is done by the code that accesses them.<br />
For example, we can easily identify Timer registers.<br />
<br />
<h4>
EL3 Monitor emulation.</h4>
<br />
S-Boot calls into the Monitor code (ARM Trusted Firmware) to do some crypto and EFUSE-related operations.<br />
These calls have argument numbers starting with a lot of FFFFFF.<br />
It was necessary to enable the "PSCI conduit" in QEMU which intercepts some SMC calls and add a simple<br />
handler to allow S-Boot to properly start without crashing.<br />
<a href="https://github.com/astarasikov/qemu/blob/exynos9820/target/arm/psci.c#L55">arm_is_psci_call</a><br />
<span style="background-color: black; color: lime;"><span style="white-space: pre;"> </span>if ((param & 0xfffff000) == 0xfffff000) {</span><br />
<span style="background-color: black; color: lime;"><span style="white-space: pre;"> </span>//Exynos SROM </span><br />
<span style="background-color: black; color: lime;"><span style="white-space: pre;"> </span>return true;</span><br />
<br />
<h4>
Putting all the pieces together: running it.</h4>
<span style="background-color: black; color: lime;">./aarch64-softmmu/qemu-system-aarch64 -m 2048 -M virt -serial stdio -bios ~/Downloads/sw/pda/s10/BL/sboot_bl2.bin -s -S 2>/dev/null</span><br />
<br />
At this point, we're not emulating most peripherals like I2C, PMIC, USB.<br />
However, the bootloader gets to the point where memory allocator and printing subsystem is initialized which should be enough<br />
to fuzz-test some parsers if we hook UFS/MMC access functions.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyjc6LDOrK9VzT6q2E4mAY8_u12bxjYUc3wi-ub750-IQR746bMv5ca3spYdsQ-0J4pG8k4GX3Z1a4xZvUBRApjidfV3IwBbZHSfOWdmJmJi3zsAa68jAPIIwLDxehN868qNu0lASKpmU/s1600/sboot_run.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1000" data-original-width="1600" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyjc6LDOrK9VzT6q2E4mAY8_u12bxjYUc3wi-ub750-IQR746bMv5ca3spYdsQ-0J4pG8k4GX3Z1a4xZvUBRApjidfV3IwBbZHSfOWdmJmJi3zsAa68jAPIIwLDxehN868qNu0lASKpmU/s640/sboot_run.jpg" width="640" /></a></div>
<br />
<br />
<h2>
General approach to reverse-engineering</h2>
Samsung leaves a lot of debugging prints in their binaries.<br />
Even in the RKP hypervisor, although most strings are obfuscated by getting replaced with their hashes,<br />
some strings in the exception handler are not obfuscated at all.<br />
With this knowledge, it's easy to identify the logging function, snprintf<br />
and then strcpy, memcpy. Memcpy and strcpy are often near malloc and free.<br />
Knowing this functions it's trivial to reverse-engineer the rest.<br />
<br />
<h3>
TEEGRIS intro</h3>
In the Exynos version of Galaxy S10, Samsung have replaced<br />
the TrustZone OS from MobiCore with their solution called TEEGRIS.<br />
<br />
As we've seen before, TEEGRIS kernel and loader are located inside<br />
the BL image along with S-Boot.<br />
Userspace portion - dynamic libraries and TAs (Trusted Applications)<br />
reside in two locations:<br />
<br />
<ul>
<li>System partition ("/system/tee"):</li>
<li>A TAR-like archive linked into the Linux Kernel</li>
</ul>
<div>
Here is what we can find:</div>
<br />
<ul>
<li>00000000-0000-0000-0000-4b45594d5354 (notice how 4b 45 49 4d 53 54 are ASCII codes for "KEYMST" (Key Master))</li>
<li>00000000-0000-0000-0000-564c544b5052 VLTKPR (Vault Keeper)</li>
<li>00000005-0005-0005-0505-050505050505 - TSS (TEE Shared Memory Server?)</li>
<li>00000007-0007-0007-0707-070707070707 - ACSD (Access Control and Signing Driver?) basically the loader for TAs with a built-in X.509 parser </li>
</ul>
<br />
<br />
I wrote a <a href="https://gist.github.com/astarasikov/f47cb7f46b5193872f376fa0ea842e4b#file-startup_tzar_file_list-txt">Python script</a> to unpack the (uncompressed) TZAR files.<br />
https://gist.github.com/astarasikov/f47cb7f46b5193872f376fa0ea842e4b#file-unpack_startup_tzar-py<br />
After unpacking the file "startup.tzar" from S10 kernel tree (LINK)<br />
we can see that it contains a bunch of libraries as well as two TEE applications<br />
which can be identified by their file names resembling GUIDs.<br />
<h3>
Security mechanisms</h3>
<br />
<ul>
<li>Boot Time: TEEGRIS kernel and startup_loader reside in the same partition as S-Boot so their integrity should be checked by the early bootloader (in SROM).</li>
<li>Run Time: TrustZone applets (TAs) are authenticated using either built-in hashes or X.509 certificates.</li>
<li>Trustlets and TEEGRIS kernel has stack cookies and they are randomized.</li>
</ul>
<br />
All TAs are ELF files which export the symbol "TA_InvokeCommandEntryPoint" which<br />
is where requests from Non-Secure EL1 (and other Secure EL0 TAs) are processed.<br />
Additionally, some extra TZ applets can be found in the "system" partition.<br />
<br />
<h3>
Indentifying TEEGRIS syscalls</h3>
<h4>
Attempt 1 (stupid)</h4>
Look for the syscall number and a compare instruction.<br />
For example, for the "recv" syscall, let's search for 0x38, filter results by "cmp".<br />
No Luck. Ok, it's probably using a jump table or a function pointer array instead.<br />
<br />
<h4>
Attempt 2</h4>
Let's locate AArch64 exception table and go from there.<br />
We can find it by a bunch of NOPs (1f 20 03 d5) immediately after a block of zero-filled memory.<br />
We can then find the actual exception handler for EL0 by knowing the offset from the ARM ARM.<br />
https://developer.arm.com/docs/100933/latest/aarch64-exception-vector-table<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgak391vWbIGe51vQ7-weEd7yZgt2TnBrdDFUZiOA0Uhiox4oM1kSzgfXVWT2Y9vKTw4FHcMuvxdT_LQOHmaECrGk2stMkaMUp_3TTbmIsMQKszkamHCIDOfGI8qUOcOKSX_ljDUMgR_KM/s1600/TEEGRIS_Syscall_handler_exc.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="499" data-original-width="1212" height="163" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgak391vWbIGe51vQ7-weEd7yZgt2TnBrdDFUZiOA0Uhiox4oM1kSzgfXVWT2Y9vKTw4FHcMuvxdT_LQOHmaECrGk2stMkaMUp_3TTbmIsMQKszkamHCIDOfGI8qUOcOKSX_ljDUMgR_KM/s400/TEEGRIS_Syscall_handler_exc.png" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2ool0pdetDrF298LM_d3xCLyhvx_pKeeQnaJffy4hf009pTxAdGCJ80jUxeRmTYUn0m8eHSXmsCcioqP3r6Z9xU9R78SGcxaBZZEQy0jUBkmf8R-p8E4yJ8hzKlrMhJRonGsKREbWeK8/s1600/TEEGRIS_Syscall_Range_Check.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1000" data-original-width="1600" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2ool0pdetDrF298LM_d3xCLyhvx_pKeeQnaJffy4hf009pTxAdGCJ80jUxeRmTYUn0m8eHSXmsCcioqP3r6Z9xU9R78SGcxaBZZEQy0jUBkmf8R-p8E4yJ8hzKlrMhJRonGsKREbWeK8/s640/TEEGRIS_Syscall_Range_Check.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />
<br />
P.S.<br />
In fact, the code which launches "startup_loader" sets VBAR_EL1 to the same<br />
address which we've identified before.<br />
<br />
<h4>
Syscalls</h4>
Luckily for us, Samsung put wrappers for each syscall into the library called "Libtzsl.so"<br />
so we can easily recover the syscall names from the index in the table.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7LLT_uWDdRsXsu7OoWsYLRSvvhaGbKXEH7b64oLXOcSVDYRZ6KJfA5yLFxYjfz-t8Bad7jpAK088aHJ7TplpL_ryNtVf9F8XUyHqeSiD9YK4MVmZcxP3NrZBQ-s47XO_sonA6QL3QwkY/s1600/teegris_syscall_socket.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1000" data-original-width="1600" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7LLT_uWDdRsXsu7OoWsYLRSvvhaGbKXEH7b64oLXOcSVDYRZ6KJfA5yLFxYjfz-t8Bad7jpAK088aHJ7TplpL_ryNtVf9F8XUyHqeSiD9YK4MVmZcxP3NrZBQ-s47XO_sonA6QL3QwkY/s640/teegris_syscall_socket.png" width="640" /></a></div>
<br />
<h4>
TEEGRIS IPC</h4>
Curiously, Samsung chose to implement two popular POSIX APIs to communicate<br />
between TAs as well between TAs and REE (Linux): "epoll" and "sendmsg/recvmsg".<br />
<br />
Peripherals such as I2C and RPMB are of course handled by file paths with magic<br />
names, like on most UNIX-like kernels.<br />
<h4>
List of (most) TEEGRIS syscalls</h4>
<a href="https://github.com/astarasikov/qemu/blob/teegris_usermode/linux-user/syscall.c#L11590">https://github.com/astarasikov/qemu/blob/teegris_usermode/linux-user/syscall.c#L11590</a><br />
<h3>
TEEGRIS emulator</h3>
Since I'm better at reverse engineering than at exploitation<br />
and I like writing emulators but hate code review, I decided to<br />
find a way to run TAs on the Linux laptop instead of the actual<br />
device.<br />
<br />
Besides doing full-system emulation, QEMU supports the "user" target.<br />
In this case it loads the target ELF binary into memory and translates<br />
instructions to the host architecture, but instead of blindly passing<br />
syscall arguments to real syscalls it can patch them and do any kind of<br />
emulation.<br />
<br />
Here are the changes that I needed to make in order to run TEEGRIS binaries instead of Linux ones:<br />
<br />
<ul>
<li>ELF Entrypoint: setup AUXVALs in a specific order that "bin/libtzld.so" expects</li>
<li>Slightly different ABI: register X7 is used for the syscall number for both ARM32 and ARM64</li>
<li>https://github.com/astarasikov/qemu/blob/teegris_usermode/linux-user/syscall.c#L11785</li>
<li>TLS handling (QEMU bug?)</li>
</ul>
<br />
<h4>
Current Status.</h4>
<br />
<ul>
<li>Boots TAs, both 32-bit and 64-bit</li>
<li>Currently does not support launching TAs from TAs (thread_create)</li>
<li>Currently only invalid command handler is reached. Need to improve</li>
<li>recvmsg or patch the library code as a workaround.</li>
<li>But overall it should be possible to build a fuzzer for TAs in less than a week of work now.</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrXxuX0nelgkNLdW2Xe3ZL_Ulsff8cMF1o5idJemj6DS0wTrNyp8HoEl6syR-CNHIKEsBhhw2v_HMH9ID4HUtqLK5PcZy6KeefkZ8u0U2G70xpwqFyOzJPDdLsBfi3M3qyeLyLofX-bQE/s1600/teegris_mmap.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1000" data-original-width="1600" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrXxuX0nelgkNLdW2Xe3ZL_Ulsff8cMF1o5idJemj6DS0wTrNyp8HoEl6syR-CNHIKEsBhhw2v_HMH9ID4HUtqLK5PcZy6KeefkZ8u0U2G70xpwqFyOzJPDdLsBfi3M3qyeLyLofX-bQE/s640/teegris_mmap.png" width="640" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<br />
Here's one idea: now that we know we've emulated enough of syscalls<br />
for a TA to boot and start message processing, we can just override the<br />
return address and arguments for one of the syscalls which are invoked<br />
in the message processing loop and redirect the execution directly<br />
to TA_InvokeCommandEntryPoint.<br />
<br />
For this proof of concept I've manually identified the entry point address<br />
and adjusted it according to the ELF base load address and QEMU-specific load<br />
offset. Of course it would be better to automate this part so that TA loader<br />
is more generic but as every software engineer knows, those who write<br />
good code get scooped by those who don't.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimihdvzRp3R7fovbhVOhrRD8R4YPYL7Z4fI100bBCGsxeKdkxigEdpRawLC4iNphYBYV4vaLnbWAoWKKbgrI7QZN5HY24RIwT8ynUMa-sTHVPFnfFBjhFQ54xtkyYWhyphenhyphenKaGdAzOSmFzKw/s1600/KEYMASTER_Entry.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="554" data-original-width="1329" height="265" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimihdvzRp3R7fovbhVOhrRD8R4YPYL7Z4fI100bBCGsxeKdkxigEdpRawLC4iNphYBYV4vaLnbWAoWKKbgrI7QZN5HY24RIwT8ynUMa-sTHVPFnfFBjhFQ54xtkyYWhyphenhyphenKaGdAzOSmFzKw/s640/KEYMASTER_Entry.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhh1HHEilIyolnJAvKFVHXLcIjN51OIBKH2zUJuIgQ9y5eWdknObak_U7ThO4VKkV9p9HMmVHKBHyIQPM0lqhPWJ8I2tiXig3h2vMploS9d7svze9h2k6LzQK1CKuj1SKVr8uZaJyxJnT0/s1600/KEYMASTER_handler2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1000" data-original-width="1600" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhh1HHEilIyolnJAvKFVHXLcIjN51OIBKH2zUJuIgQ9y5eWdknObak_U7ThO4VKkV9p9HMmVHKBHyIQPM0lqhPWJ8I2tiXig3h2vMploS9d7svze9h2k6LzQK1CKuj1SKVr8uZaJyxJnT0/s640/KEYMASTER_handler2.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhnTUo2R21xMLi9UDh_k7No_vkIfzU-YeplidvTBqEiOxiqapIpVk_TAWDwJel1mF8chIFQ5acjaC1ZHUtN9i_NE63_1Oi0SEajgqaoxEM8q8MWjhmsMfmw1MJaW8Y_kicBj7nAkzOazDc/s1600/KEYMASTER_swd.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="311" data-original-width="1046" height="190" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhnTUo2R21xMLi9UDh_k7No_vkIfzU-YeplidvTBqEiOxiqapIpVk_TAWDwJel1mF8chIFQ5acjaC1ZHUtN9i_NE63_1Oi0SEajgqaoxEM8q8MWjhmsMfmw1MJaW8Y_kicBj7nAkzOazDc/s640/KEYMASTER_swd.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgAB_-NvtPPn9K04JSfXg93kdge-wB8CMaHW8mJ6uvmzxnJnsOHrp9qrogzPYWTy8Q1-UvurshNLMfeEjnwHQqrEJDAHjqpOagkXKweisPhpqlSmWGTjXw7CgxRqtJckGAGZCCpJYEsoRM/s1600/TEEGRIS_ASN1_Templates.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1000" data-original-width="1600" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgAB_-NvtPPn9K04JSfXg93kdge-wB8CMaHW8mJ6uvmzxnJnsOHrp9qrogzPYWTy8Q1-UvurshNLMfeEjnwHQqrEJDAHjqpOagkXKweisPhpqlSmWGTjXw7CgxRqtJckGAGZCCpJYEsoRM/s640/TEEGRIS_ASN1_Templates.png" width="640" /></a></div>
<br />
<br />
This kind of works in that we're getting messages from inside the TA: check the full log at <a href="https://pastebin.com/sVtWk5CD">https://pastebin.com/sVtWk5CD</a> and search for "<span style="background-color: white; color: #333333; font-family: Consolas, Menlo, Monaco, "Lucida Console", "Liberation Mono", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", monospace, serif; font-size: 12px;">keymaster [ERR]".</span><br />
However, it fails early when validating the message contents.<br />
We need to generate the correct ASN.1 payload which should be doable<br />
since ASN.1 grammar templates are compiled into the binary.<br />
<h3>
Ideas for future research</h3>
<br />
<ul>
<li>Hook malloc/free and some other functions and invoke native system C library calls.</li>
<li>Hook QEMU JIT (TCG) or interpreter to check memory accesses against ASAN shadow memory. This way we can enable Address Sanitizer for binary blobs, similarly to how Valgrind does memory debugging. Since QEMU Usermode runs TAs in the same address space as itself, we can use ASAN allocator or libdislocator to detect OOB memory access. Unicorn is kind of hard to use because for this because it does not allow to easily set up MMIO traps, it only allows to register chunks of normal memory.</li>
<li>Finish reverse-engineering ASN.1 format for Keymaster and fuzz this TA.</li>
<li>Run TEEGRIS kernel in QEMU as well to fuzz syscalls.</li>
<li>A Ghidra script to rename functions according to the debug strings passed to invokations of print callees</li>
<li>Look at the ring buffer implementation in the shared memory.</li>
</ul>
<h3>
Running TEEGRIS Emulator</h3>
export TEE_CMD=777<br />
qemu/teegris$ ../arm-linux-user/qemu-arm -z fuzz_keymaster/in/test0.bin -cpu max ./00000000-0000-0000-0000-4b45594d5354.elf<br />
<br />
Debugging panics with GDB<br />
<br />
<ul>
<li><a href="https://github.com/astarasikov/qemu/blob/teegris_usermode/linux-user/syscall.c#L11929">https://github.com/astarasikov/qemu/blob/teegris_usermode/linux-user/syscall.c#L11929</a></li>
<li>Uncomment the code in the "panic" syscall which sets PC to 0x41414141 so that the exception is delivered to GDB.</li>
<li>Add -g 1234 to qemu-arm arguments to listen for GDB.</li>
<li>Run the GDB script in root folder: "qemu$ arm-none-eabi-gdb -x gdbinit"</li>
</ul>
<br />
<h3>
Related Projects</h3>
Post from Daniel Komaromy on reverse-engineering Galaxy S8 which mostly focuses on the other part of the picture: getting from Linux into Secure EL0.<br />
<br />
<ul>
<li><a href="https://medium.com/taszksec/unbox-your-phone-part-i-331bbf44c30c">https://medium.com/taszksec/unbox-your-phone-part-i-331bbf44c30c</a></li>
</ul>
<br />
Blog from Blue Frost Security on reverse-engineering S9 TrustZone. The OS kernel is different but actual TAs are the same.<br />
<br />
<ul>
<li><a href="https://labs.bluefrostsecurity.de/blog/2019/05/27/tee-exploitation-on-samsung-exynos-devices-introduction/">https://labs.bluefrostsecurity.de/blog/2019/05/27/tee-exploitation-on-samsung-exynos-devices-introduction/</a></li>
<li><a href="https://labs.bluefrostsecurity.de/files/TEE.pdf">https://labs.bluefrostsecurity.de/files/TEE.pdf</a></li>
</ul>
I hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com9tag:blogger.com,1999:blog-1416969656167659289.post-63965103913518670182016-10-24T14:11:00.002+03:002016-10-24T14:11:35.892+03:00running without an ARM and a leg: Windows Mobile in QEMU<h2>
Introduction.</h2>
One project I had in mind long time ago was getting Windows Mobile to run in QEMU.<br />
I think it's a lovely OS with a long history and the project seemed like a nice tecnhical challenge.<br />
<br />
Initially I started working on it two years ago back in 2014 and the plan was to later run it in KVM on Cortex-A15 with Virtualization Extensions. However, I had to suspend it because I started working on two other challenges - running XNU on Xen (aka Virtu.al LLC) and later doing GSoC (running FreeBSD in ARM emulator).<br />
<br />
Now since I've got some free time on my hands, I decided to finally get back to this project and cross it off my TODO list.<br />
<h3>
Choosing the emulation target.</h3>
In order to run Windows CE on QEMU (or any OS for that matter) it would be necessary to either develop a Board Support Package with all the drivers for a specific virtual machine or take the opposite approach and emulate some machine for which there already exists a ROM image.<br />
<br />
For Windows, there is the emulator developed by Microsoft which is unsurprisingly called just Device Emulator. It emulates a real board - MINI2440 based on Samsung S3C2440 SoC which is an ancient ARMv4 CPU. Turns out, this is the same SoC that's used in OpenMoko so there is an old fork of QEMU with the support for most of the peripherals. So the choice of the platform seemed a no-brainer.<br />
<br />
So I took the QEMU fork supporting MINI2440 and tried to adapt it to running the unmodified Windows Mobile images from Microsoft. Needless to say, I made sure the images are placed into memory at the correct addresses but the code seemed to crash spontaneously and never got past enabling MMU.<br />
<br />
The first idea that comes to mind is of course to take the latest QEMU and see if it fixes anything. However, trying random changes until something works is actually quite a crappy approach.<br />
<br />
So, I decided to single-step the execution and see what happens. QEMU provides the very useful GDB interface (which can be activated with the "-s -S" switches) for this purpose.<br />
<h3>
Caches are hard.</h3>
The first surprise came from the bootloader code. Before launching the kernel, Windows CE bootloader disables the MMU. At this moment QEMU crashes spectacularly. Initially I tried hacking around the issue by adding the code to translate the addresses into the "exec-all.h". However, it didn't solve the problem and the heuristic started looking too complex which suggested I'm on the wrong way.<br />
<br />
I started thinking of and realized that disabling MMU is a tricky thing because on ARM Program Counter is usually ahead of the current instruction so the CPU has to fetch a couple instructions ahead. So we have a caching problem. In this Windows CE code, there is a NOP instruction between disabling the MMU and jumping to the new PA from a register. The hypothesis was that QEMU did not fetch the needed instructions correctly and it was necessary to add a translation cache for them. The reality was more funny.<br />
<br />
As it turned out, QEMU contained a hack to cache a couple instructions because... because Linux kernel for PXA270 had the same problem in the opposite scenario - when the MMU was enabled. I decided to comment out the hack and it made the boot-up progress further. (<a href="https://github.com/astarasikov/qemu/blob/52667dc0fab56b0b2f9f317f9d706b6c1c723034/target-arm/translate.c#L2637">PXA Cache Hack</a>).<br />
<h3>
Stacks are not easy either.</h3>
Next thing I know is that soon after enabling the MMU the code crashes when trying to access the stack at a very peculiar virtual address of 0xffffce70. I examined the QEMU translation code and found out that it correctly locates the physical address but the permission bits are incorrect. I decided to force it to return RW access for the particular address range and voila - Windows Mobile boots to Home Screen successfully. (<a href="https://github.com/astarasikov/qemu/blob/wm5_with_hacks/target-arm/helper.c#L1234">Hack to force RW stack</a>).<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgH0yY4llGSzFQdge5Nff0B-NqoQU3nbTl1Gk3HyDx-fzEXKbtKNqkE4jCoIgZ-gaa_l-cMauU8nXuOjRFNbfAfjxYxN1GRsaSj-JtmWbo3i5tYHHWLJwFYQHQumBe1jk13xdlPPVEt_yE/s1600/sp.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="258" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgH0yY4llGSzFQdge5Nff0B-NqoQU3nbTl1Gk3HyDx-fzEXKbtKNqkE4jCoIgZ-gaa_l-cMauU8nXuOjRFNbfAfjxYxN1GRsaSj-JtmWbo3i5tYHHWLJwFYQHQumBe1jk13xdlPPVEt_yE/s400/sp.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Windows Mobile 5.0 Smartphone Edition on QEMU on Mac OS X.</td></tr>
</tbody></table>
Now that everything seemed to boot, I decided to take another look at the MMU issue and fix it properly. The first idea I had was to compare ARM920T (ARMv4) and ARM1136 (ARMv5) page table formats. It is worth noting that ARMv4 did not have TEX remap bits, and also last level page tables had different type (last two bits). It turned out that QEMU (probably as real ARM CPUs) supported all types of pages, even ARMv4 on ARMv6 target, and TEX/caching bits were simply ignored. After careful examination of the code and all the page table parsing I found a typo that was already fixed in the upstream QEMU (<a href="https://github.com/astarasikov/qemu/commit/52667dc0fab56b0b2f9f317f9d706b6c1c723034#diff-94d702870edadd5b54b9c045b93ad0b8R1000">MMU Typo</a>).<br />
<br />
You can grab the cleaner tree without the ugly intermediate hacks: <a href="https://github.com/astarasikov/qemu/tree/wm5">https://github.com/astarasikov/qemu/tree/wm5</a> . If you need to have a look at the older WIP stuff, it's at <a href="https://github.com/astarasikov/qemu/tree/wm5_with_hacks">https://github.com/astarasikov/qemu/tree/wm5_with_hacks</a> .<br />
<h3>
Running Pocket PC version.</h3>
<h4>
Preparing PocketPC Image.</h4>
Windows Mobile Standard aka Smartphone comes shipped as the NOR flash image and it is XIP (Execute-in-Place). Pocket PC Image comes in a different form. It comes as an image already relocated to the RAM so we need to launch it directly via the "-kernel" parameter. We need to create the NOR memory image for it. We can use the smartphone image as a base, but an empty 32MB file should work as well.<br />
<br />
<ul>
<li>Grab the "romtools" scripts at github: <span style="background-color: white; font-family: "arial" , sans-serif; line-height: 16px; white-space: nowrap;"><a href="https://github.com/pinkavaj/romtools">https://github.com/pinkavaj/romtools</a></span></li>
<li>python b000ff-to-bin.py ../ppc_50/_208PPC_USA_bin</li>
<li>dd if=_208PPC_USA_bin-80070000-81491ed0.bin of=SPHIDPI_BIN bs=$((0x30000)) seek=1 conv=notrunc</li>
<li>./arm-softmmu/qemu-system-arm -show-cursor -m 256 -M mini2440 -serial stdio -kernel ../wm5/50_ppc/_208PPC_USA_bin-80070000-81491ed0.bin -pflash ../wm5/50_ppc/SPHIDPI_BIN</li>
</ul>
<br />
This one did not boot and kept hanging at the boot splash screen. The host CPU usage was high and it indicated there was some busy activity inside the VM like an IRQ storm. After hooking up the debugger it turned out that the VM was hanging in two busy loops.<br />
<br />
One of them was in the Audio driver - during the splash screen Windows plays a welcome sound. This was worked around by setting the "TX FIFO Ready" status in the sound codec. The second freeze was in the touchscreen driver but that looks like a MINI2440 emulation bug - once the touchscreen is disabled, the workqueue timer in qemu is permanently disabled and the status bits are not updated properly. I commented out the routine which disabled the touchscren controller since it's a VM anyway (not a real device where it would have a power-saving impact).<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBAQNEg0s_vOTiGRRUs1jMzB4PGWAuloHofreHtDhW0qCmMFS_uqRFSagIX9uW2_-8Dg3GTai4g-TjAx1sWNSqOIdnfkUcSB-bxQRM_0Ls4PKD_owNG_p-XaYYxcquLqzkIves9llhtSk/s1600/ppc.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="216" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBAQNEg0s_vOTiGRRUs1jMzB4PGWAuloHofreHtDhW0qCmMFS_uqRFSagIX9uW2_-8Dg3GTai4g-TjAx1sWNSqOIdnfkUcSB-bxQRM_0Ls4PKD_owNG_p-XaYYxcquLqzkIves9llhtSk/s320/ppc.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Obligatory screensot: Windows Mobile 5.0 Pocket PC</td></tr>
</tbody></table>
<h4>
Retarded idea.</h4>
I was fantasizing about adding a logic to QEMU which would detect if emulation was stuck invoking MMIO device handler in a loop and fuzzing the returned register value until it was unstuck. While not very practical, one could reverse-engineer the register bits blindly this way.<br />
<h3>
Further ideas.</h3>
I don't think I'll invest more time into this project because there's little value but I'm considering developing an Android port of this QEMU fork just for the fun of it. Perhaps a better option would be to emulate a newer SoC such as Qualcomm 8250 and run an ARMv7 image from HTC HD2.<br />
<br />
It's quite sad that most if not all Android phones ship with HYP mode (Hypervisor) disabled so KVM is a no-go. On the other hand, allowing to run custom hypervisors opens up a pandora box of security issues so it might be a good decision. Luckily for us who like to tinker with hardware, most TrustZone implementations also contain exploitable bugs so there are some possibilities to uncover the potential :)<br />
<br />
Today, many companies working with embedded SoCs, seek GPU virtualization solutions. The demand is particularly high in the automotive sector where people are forced to use eclectic mixes of relatively old SoCs and retaining binary compatibility. So it would be interesting to prototype a solution similar to Intel's GVT.<br />
<br />
I think it would be nice to use Qualcomm Adreno GPU as an emulation target. It is relatively well reverse-engineered - there exists Mesa FreeDreno driver for it, Qualcomm commits patches to the upstream KMS driver and most importantly the GPU ISA is similar to older AMD Radeon GPUs which is extensively documented by AMD. Besides, a similar GPU is used in Xbox 360 so one more place to learn is Xenia emulator which simulates the Xbox GPU.I hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com4tag:blogger.com,1999:blog-1416969656167659289.post-74772572653023102232016-09-28T00:55:00.000+03:002016-09-28T00:55:28.962+03:00Trying out and debugging Open-Source OpenCL on AMD Radeon RX480<h2>
Introduction.</h2>
I have decided to check out the state of OpenCL support for AMD GPUs in Linux Open-Source stack. In this post I will describe the experience of debugging issues across the whole stack trying to get a few basic apps up and running.<br />
<br />
I have the AMD RX480 GPU which supports GCN 8.0.1 instruction set and has the code name "polaris10". At the time of writing, both Linux kernel and the Mesa in Debian were too old and did not support this GPU. Besides, as we will see later, the OpenCL in Mesa does not work out of the box and we would need to learn to build it from source anyway.<br />
<h2>
Building the software.</h2>
A page at FreeDesktop website has some relatively outdated instructions but the general approach is the same. <a href="https://dri.freedesktop.org/wiki/GalliumCompute/">https://dri.freedesktop.org/wiki/GalliumCompute/</a><br />
You will need to install a relatively fresh kernel (I built Linux 4.7.0-rc7). I also installed the polaris10 firmware manually but now it seems to be shipped in Linux/Debian.<br />
<br />
I'm posting the steps I went through to build LLVM and Mesa with OpenCL support. After writing it I realized that p<span style="background-color: white;">erhaps everything here is redundant I should write a repo manifest to clone everything with one command.</span><br />
<h3>
Getting the sources.</h3>
I'm also posting the folder tree and git hashes just in case.<br />
<h4>
Build CLC</h4>
<div>
CLC is the runtime library for the OpenCL. It contains code that is compiled to LLVM bitcode and linked against your apps. It provides intrinsics and functions for the functions defined by the OpenCL standard such as <span style="background-color: #cfe2f3;">"get_global_id"</span> and <span style="background-color: #cfe2f3;">"mad"</span>. In case you're wondering, CUDA works exactly the same way and the binary SDK from NVIDIA ships the relevant bitcodes (and you can disassemble them with llvm-dis if you're interested).</div>
<div>
<br /></div>
<div>
Actually, some of the definitions failed to compile with latest LLVM and I needed to add the explicit type casting. You can use the patch (from this github gist <a href="https://gist.github.com/astarasikov/9f00dee718f217c6a9715510dc09d300">https://gist.github.com/astarasikov/9f00dee718f217c6a9715510dc09d300</a>) and apply it on top of <b>88b82a6f70012a903b10dfc1e2304d3ef2e76dbc </b>to fix it.</div>
<div>
<br /></div>
<div>
<span style="background-color: #d9ead3;">git clone https://github.com/llvm-mirror/libclc.git</span></div>
<div>
<span style="background-color: #d9ead3;">./configure.py -g ninja --with-llvm-config=/home/alexander/Documents/workspace/builds/llvm/build/bin/llvm-config</span></div>
<div>
<span style="background-color: #d9ead3;">ninja</span></div>
<div>
<span style="background-color: #d9ead3;">ninja install</span></div>
<h4>
Get LLVM Sources.</h4>
<div>
<span style="background-color: #d9ead3;">mkdir -p ~/Documents/workspace/builds/llvm/</span></div>
<div>
<span style="background-color: white;">I have a useful script to check out the latest llvm, clang and libcxx. </span><a href="https://gist.github.com/astarasikov/a2e9287a34381f680d58">https://gist.github.com/astarasikov/a2e9287a34381f680d58</a></div>
<div>
<h4>
Get LLVM ROC branch.</h4>
</div>
<div>
This is optional. I used the ROC branch initially because I thought it would fix the issue with codegen (FLAT instructions) but it did not and otherwise it seems to behave identical to vanilla LLVM.</div>
<div>
<br /></div>
<div>
<span style="background-color: #d9ead3;">git remote add roc https://github.com/RadeonOpenCompute/llvm.git</span></div>
<div>
<span style="background-color: #d9ead3;">git checkout roc/amd-common</span></div>
<div>
<span style="background-color: #d9ead3;">git fetch roc</span></div>
<div>
<span style="background-color: #d9ead3;">cd tools/clang/</span></div>
<div>
<span style="background-color: #d9ead3;">git remote add roc https://github.com/RadeonOpenCompute/clang.git</span></div>
<div>
<span style="background-color: #d9ead3;">git fetch roc</span></div>
<div>
<span style="background-color: #d9ead3;">git checkout roc/amd-common</span></div>
<h4>
<div style="font-weight: normal;">
List of git hashes:</div>
<div style="font-weight: normal;">
<div class="p1">
</div>
<ul>
<li>llvm - 1819637 Merge branch amd-master into amd-common</li>
<li>llvm/tools/clang/tools/extra - 079dd6a [clang-rename] Add comment after namespace closing</li>
<li>llvm/tools/clang - f779a93 Merge branch amd-master into amd-common</li>
<li>llvm/projects/libcxx - d979eed Fix Bug 30240 - std::string: append(first, last) error when aliasing.</li>
<li>llvm/projects/compiler-rt - 5a27c81 asan: allow __asan_{before,after}_dynamic_init without registered globals</li>
<li>llvm/projects/libcxxabi - 9f08403 [lit] Replace print with lit_config.note().</li>
</ul>
</div>
</h4>
<h4>
<span style="background-color: white;">Build LLVM.</span></h4>
<span style="white-space: pre;">#</span>create the build directory<br />
<span style="background-color: #d9ead3;">mkdir -p ~Documents/workspace/builds/llvm/build/</span><br />
<span style="background-color: #d9ead3;">cd cd ~Documents/workspace/builds/llvm/build/</span><br />
<br />
<div class="p1">
<span class="s1" style="background-color: #d9ead3;">cmake -G Ninja -DCMAKE_BUILD_TYPE=Debug -DLLVM_TARGETS_TO_BUILD="AMDGPU;X86" -DLLVM_INCLUDE_TESTS=OFF -DLLVM_VERSION_SUFFIX="" ../llvm/ -DBUILD_SHARED_LIBS=ON</span></div>
<span style="background-color: #d9ead3;">ninja</span><br />
<br />
#add the symlink to make LLVM pick up internal headers when building Mesa<br />
<span style="background-color: #d9ead3;">cd ~Documents/workspace/builds/llvm/build/include</span><br />
<span style="background-color: #d9ead3;">ln -s $(echo $PWD/../tools/clang/include/clang) clang</span><br />
<br />
In principle, it is not necessary to install llvm, it's enough to add it to the PATH and clang will pick up the necessary libraries itself.<br />
<h4>
Build MESA</h4>
Before building Mesa, we need to prepend the path to the "bin" directory of our custom LLVM build to the PATH variable so that clang is picked up as the compiler. I also had to add a symlink to the source code in the build directory because some headers were not getting picked up but I think there's a cleaner way to add it to CFLAGS.<br />
<br />
I was using Mesa git <b>0d7ec8b7d0554382d5af6c59a69ca9672d2583cd.</b><br />
<br />
<span style="background-color: #d9ead3;">git clone git://anongit.freedesktop.org/mesa/mesa</span><br />
<br />
The configure.ac seems to have the incorrect regex for getting LLVM version which causes compilation to fail with the latest LLVM (4.0.0). Here is a patch to fix it and also force the radeonsi chip class to VI (Volcanic Islands). The latter is not strictly necessary but I used it during debugging to ensure the corect code path is always hit. Grab the diff at <a href="https://gist.github.com/astarasikov/6146dbbd07d0dc3bea2ee6a8b979eaa8">https://gist.github.com/astarasikov/6146dbbd07d0dc3bea2ee6a8b979eaa8</a><br />
<br />
<span style="background-color: #d9ead3;">export PATH=~/Documents/workspace/builds/llvm/build/bin:$PATH</span><br />
<span style="background-color: #d9ead3;">cd ~/Documents/workspace/builds/mesa/mesa/</span><br />
<span style="background-color: #d9ead3;">make clean</span><br />
<br />
<div class="p1">
<span class="s1" style="background-color: #d9ead3;">./autogen.sh --enable-texture-float --enable-dri3 --enable-opencl --enable-opencl-icd --enable-sysfs --enable-gallium-llvm --with-gallium-drivers=radeonsi --prefix=/opt/my_mesa --with-egl-platforms=drm --enable-glx-tls</span></div>
<span style="background-color: #d9ead3;"><span class="Apple-tab-span" style="white-space: pre;"> </span>make install</span><br />
<span style="background-color: #d9ead3;"><br /></span>
<span style="background-color: white;">Now, before running any OpenCL application, we'll need to override the library path to point to our custom Mesa.</span><br />
<span style="background-color: #d9ead3;">export LD_LIBRARY_PATH=/opt/my_mesa/lib:/home/alexander/Documents/workspace/builds/llvm/build/lib</span><br />
<h3>
Useful Links</h3>
<h3>
AMD Presentations about GCN ISA.</h3>
<ul>
<li><a href="http://www.slideshare.net/DevCentralAMD/gs4106-the-amd-gcn-architecture-a-crash-course-by-layla-mah">http://www.slideshare.net/DevCentralAMD/gs4106-the-amd-gcn-architecture-a-crash-course-by-layla-mah</a></li>
<li><a href="http://www.slideshare.net/DevCentralAMD/gs4152-michael-mantor">http://www.slideshare.net/DevCentralAMD/gs4152-michael-mantor</a></li>
</ul>
<h3>
GCN ISA Manual</h3>
<ul>
<li>Southern Islands <a href="http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf">http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf</a></li>
<li>Volcanic Islands <a href="http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf">http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf</a></li>
</ul>
<h3>
Intel OpenCL Samples</h3>
<ul>
<li><a href="https://software.intel.com/en-us/intel-opencl-support/code-samples">https://software.intel.com/en-us/intel-opencl-support/code-samples</a></li>
<li><a href="https://software.intel.com/sites/default/files/managed/43/b8/intel_ocl_bitonic_sort.zip">https://software.intel.com/sites/default/files/managed/43/b8/intel_ocl_bitonic_sort.zip</a></li>
<li><a href="https://software.intel.com/sites/default/files/managed/db/51/intel_ocl_montecarlo.zip">https://software.intel.com/sites/default/files/managed/db/51/intel_ocl_montecarlo.zip</a></li>
<li><a href="https://software.intel.com/sites/default/files/managed/b3/1d/intel_ocl_god_rays.zip">https://software.intel.com/sites/default/files/managed/b3/1d/intel_ocl_god_rays.zip</a></li>
</ul>
<h3>
AMD App SDK</h3>
I've used the older version because I thought it was in the tarball and the latest one seemed to be an executable file (though actually it was a tarball with an executable script).<br />
<ul>
<li><a href="http://developer.amd.com/tools-and-sdks/opencl-zone/amd-accelerated-parallel-processing-app-sdk/">http://developer.amd.com/tools-and-sdks/opencl-zone/amd-accelerated-parallel-processing-app-sdk/</a> - <span style="font-family: inherit;">Look for "<span style="background-color: white; font-size: 14px;">AMD-APP-SDK-linux-v2.9-1.599.381-GA-x64.tar.bz2".</span></span></li>
</ul>
<h2>
Trying it out.</h2>
<div>
<h4>
Assertion in LLVM codegen.</h4>
So we can try running any OpenCL application now and we'll hit the assertion.<br />
./BitonicSort -p Clover<br />
<br />
After reading the source code and LLVM git log it turns out.<br />
So what can we do? Let's try to see if we can force the LLVM to emit the abovementioned "FLAT Atomics".<br />
Turns out the code is already there and there is a flag, which is enabled by default when the LLVM target is "AMD HSA".<br />
<br />
Now, let's think what could the possible limitations of this approach be?<br />
<h4>
FLAT Instructions</h4>
Let's see the description from the GCN Manual. "Flat memory instructions let the kernel read or write data in memory, or perform atomic operations on data already in memory. These operations occur through the texture L2 cache.".<br />
<br />
I have not fully understood the difference between these new FLAT instructions and older MUBUF/MTBUF. As it seems to me, before GCN3 it used to be the case that different address spaces (global, private etc) could only be accessed through different instructions and FLAT instructions allow accessing any piece of GPU (and host) memory by the virtual address (hence the name since virtual address space is flat). So it seems that as long as the kernel driver sets up GPU page tables correctly and we're only using the memory allocated through OpenCL API we should be fine.<br />
<br />
FWIW, let's try running the code and see if it works.</div>
<div>
<h3>
LLVM Flat Instructions hack</h3>
As it has been mentioned before, we need to force LLVM to generate the "FLAT" instructions to access memory in GPU code.<br />
A proper way to fix it would be to add the options to the Mesa source code into the location where it instantiates the LLVM compiler (clang).<br />
To save us some time we can hack the relevant piece of the code generator in the LLVM directly (see the patch at the end of the article).</div>
<h2>
Trying it out again.</h2>
I have tried the following samples from the AMD App SDK and Intel Samples. I didn't want to write sample code myself and besides OpenCL claims to be portable so running the code from other IHVs or SDKs should be a good stress test of the toolchain.<br />
<h4>
AMD</h4>
<ul>
<li>BlackScholesDP</li>
<li>BinominalOption</li>
<li>BitonicSort</li>
<li>BinarySearch</li>
<li>BufferBandwidth</li>
</ul>
<h4>
Intel</h4>
<ul>
<li>Bitonic Sort</li>
<li>Montecarlo</li>
<li>God Rays</li>
</ul>
<div>
All of them worked and the "verification" step which computes the data on the host and compares to the GPGPU result has passed! You can take a look at the screenshots and logs at the end of the post.</div>
<br />
The "God Rays" demo even produces the convinceable picture.<br />
<h2>
Running Protonect (libfreenect2)</h2>
<div>
One of the apps I'm particularly interested in running is Protonect which is the demo app for libfreenect2, the open-source driver for Microsoft Kinect v2 RGBD ToF camera. Let's build it with OpenCL support and invoke it from the shell via "./Protonect cl".</div>
<div>
<br /></div>
<div>
And we're hitting an assertion!</div>
<div>
<span style="background-color: #f4cccc;">'llvm::AsmPrinter::~AsmPrinter(): Assertion `!DD && Handlers.empty() && "Debug/EH info didn't get finalized"'</span>.</div>
<div>
Since it happens in a destructor for the purposes of testing we can simply comment it out because the worst thing that could happen is a memory leak.</div>
<div>
<br /></div>
<div>
Let's try running it again!</div>
<div>
And we're hitting another error. This time it is <span style="background-color: #f4cccc;">"unsupported initializer for address space"</span>. Okay, let's debug it. First, let's grep the string verbatim.</div>
<div>
<br /></div>
<div>
Good, we're hitting in only one place. Now, debugging message is not particularly helpful because it does not give us the precise location or the name of the variable which caused this error, it only shows the prototype of the function. Let's try to just print the address space type and try to find out what might be causing it. (see the patch at the end of the article).</div>
<div>
<br /></div>
<div>
What's this? Good, good, we see that the address space enum value is "0". Looking it up reveals that it's a private address space. Okay, what could cause the private address space to be used? Function-local arrays! Let's look for one! Great, here it is, see the "const float gaussian[9]" array in the "filterPixelStage1" function? Let's try commenting it out and replacing the "gaussian[j]" access with some constant (let it be 1.0f since it's a weighed average, otherwise if we choose 0.0f we'll see nothing in the output buffer). Yay, it worked!</div>
<div>
<br /></div>
<div>
Since we could not longer use the private address space we would need to find a way to get rid of the locally-declared array.OpenCL 1.1 does not support the static storage class.<br />
One option would be to add another kernel argument and just pass the array there.<br />
It might be slower depending though because it would get placed into the slower region of cache-coherent memory.<br />
<br />
Another option would be to compute the values in-place and since it's just a 3x3 convolution kernel for a gaussian blur it's easy to come up with a crude approximation formula which is what I've done (see the patch at the end of the post).<br />
<br />
Sor far, Protonect works. Performance is subpar but not completely awful. It's around 4 times slower than the NVIDIA GTX970 with the binary driver (in most real-world scenarios GTX970 and RX480 are quite close). I think that with a bit of profiling it can be sped up drastically. In fact, one of the contributing factors might be that my display is connected to the Intel iGPU and PCIE bandwidth is saturated by the framebuffer blitting (it's 4K after all). I'll try with OpenGL running on Radeon next time.</div>
<h2>
RadeonOpenCompute Initiative.</h2>
<div>
Since OpenCL is crap and CUDA is all the hype, it makes now wonder many people want to run CUDA on all GPUs.</div>
<div>
In principle, to run CUDA on non-NVIDIA hardware, one needs to implement the CUDA runtime and the lowering from the NVPTX intermediate language to the native ISA. The challenging part is actually building the source code because one would need the proprietary headers from the NVIDIA toolchain. One could create the fake headers to mimic the CUDA toolchain but it is a shady legal area especially when it comes down to predefined macros and constants.</div>
<div>
<br /></div>
<div>
In fact, there's not much need to emit NVPTX at all and you can lower straight to the ISA. What AMD have done to work around legal issues is they've come up with their own language called "HIP" which mimics most of CUDA design but keywords and predefined macros are named differently. Therefore, porting is straightforward even with a search/replace function, but there's an automated translator based on clang.</div>
<h3>
GCN old RX480 vs R390</h3>
It looks interesting that Polaris10 (RX480) seems to use an older version of the ISA (8.0.1) while the older R390 uses 8.0.3. Not sure if it's a bug in documentation. However, it's interesting that AMD GPUs consist of multiple units (such as video encoder/decoder) and they seem to be picked up in arbitrary order when designing a new chip.<br />
<h3>
HIP without Mesa.</h3>
HIP/ROC ships its own runtime, and since all memory access is done through DRM via ioctls to <span style="background-color: #cfe2f3;">"/dev/drm/cardX"</span>, Mesa is tecnhically not needed to implement OpenCL or whatever compute API.<br />
<br />
However, the open question is buffer sharing between different APIs. I have came across this issue before when dealing with an Intel GPU. The good news is that on Linux there's an EGL extension to export the DRM buffer object (BO) from the GLES context (but not from GLX). You can read the old article about it here: <a href="https://allsoftwaresucks.blogspot.com/2014/10/abusing-mesa-by-hooking-elfs-and-ioctl.html">https://allsoftwaresucks.blogspot.com/2014/10/abusing-mesa-by-hooking-elfs-and-ioctl.html</a><br />
<h2>
Outstanding issues.</h2>
<h3>
Slow Compilation time for Kernels.</h3>
While running OpenCL code samples and Protonect, I've noticed that they take several seconds to start up compared to immediate start when using Intel OpenCL driver (Beignet). I think it might get caused by LLVM. It would be a good idea to profile everything using the Linux "perf" tool.<br />
<h3>
Private Address Spaces.</h3>
As we've seen with libfreenect2 kernels, private address spaces do not work. It is necessary to figure out if they are not supported at all or it is just a simple bug.<br />
Until this is resolved it effectively renders a lot of GPGPU applications unusable.<br />
<h3>
LLVM Assertions (Bugs).</h3>
As mentioned before, running Protonect and compiling libfreenect2 kernels yields the following assertion:<br />
<span style="background-color: #f4cccc;">'llvm::AsmPrinter::~AsmPrinter(): Assertion `!DD && Handlers.empty() && "Debug/EH info didn't get finalized"'</span>.<br />
<br />
While trying to run one of the OpenCL samples I hit yet another assertion:<br />
<span style="background-color: #f4cccc;">'clang::Sema::~Sema(): Assertion `DelayedTypos.empty() && "Uncorrected typos!"'</span> failed.<br />
<h2>
Conslusions.</h2>
As we can see, the OpenCL support is still not mature with the Mesa/Clover stack.<br />
However given the RadeonOpenCompute initiative I'm sure most of the missing features will be in place soon.<br />
So far I'm glad that the just-releases GPU support is on par with older models and working around the issues was not too hard for me.<br />
I've also satisfied part of my desire to understand the interaction between different components involved in the OpenCL/Compute pipeline.<br />
<br />
I think for a start I will look at the LLVM assertions and see if I can debug them or prepare the test cases to submit upstream.<br />
Next up I'll be trying out HIP to build some CUDA samples.<br />
<br />
One idea I had in mind for quite some time was virtualizing a mobile GPU. I think Qualcomm Adreno is a good target because it's relatively well supported by the FreeDreno driver and the ISA is similar to other AMD chips. The plan is to add the ISA decoder and MMIO space emulation to QEMU so that it can be used both in KVM on ARM and in emulation mode on Intel. Of course, the most nerdy way to do it would be to make a translator from the guest ISA to the host ISA. But for a start we could reuse the Virgil driver as a target.<br />
I think it would be a very useful thing for running legacy applications in a virtualized environment (such as Windows RT or automotive IVI systems) and could aid in security engineering.<br />
Hopefully I will have enough motivation and time to do it before I'm bounded by an NDA :)<br />
<h2>
Latest update!</h2>
Also, check out the latest news! Looks like Mesa has now switched to using the HSA ABI by default which means that the hack for the FLAT instructions will not be needed with more recent versions and they will be enabled automagically! <a href="https://www.phoronix.com/scan.php?page=news_item&px=RadeonSI-HSA-Compute-Shaders">https://www.phoronix.com/scan.php?page=news_item&px=RadeonSI-HSA-Compute-Shaders</a><br />
<br />
I started trying OpenCL on RX480 around 2 weeks ago, then I spent 1 week debugging and 1 week I was away. Meanwhile some changes seem to have landed upstream and some of the hacking described here may be redundant. I urge you to check with the latest source code but I decided to keep this post just to describe the process of debugging I went through.<br />
<h2>
Extra: Logs and Screenshots.</h2>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCNH5HqC_agOO4ZMqPak089SggwlVyZ52JL5Kp9eDyK1g3D_a9eEOYhl1a5vigPFn5WDAZ2HrMKQsY3tTSsboWlrfOShpfy2cV_kHCTiQW2HMd9cKhTBVS8p9OQy3PVuNFS2XQowzMKKg/s1600/GodRaysOutput.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCNH5HqC_agOO4ZMqPak089SggwlVyZ52JL5Kp9eDyK1g3D_a9eEOYhl1a5vigPFn5WDAZ2HrMKQsY3tTSsboWlrfOShpfy2cV_kHCTiQW2HMd9cKhTBVS8p9OQy3PVuNFS2XQowzMKKg/s400/GodRaysOutput.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">God Rays from Intel OpenCL samples.</td></tr>
</tbody></table>
<div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtjiiSGvravwQTfnYHUHHBW_mZhz-K41UlQ3OlZdgKHz1joMZ1SIwHpWhLJbwcbJjJOXKjl1XOmOcH4HZ5i8Kp-HNI82jmrvhtSirf51QWQcYAukW5kDCJSSV8nZQXs3ab8QJKcxZpttw/s1600/protonect_amd_radeon.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="475" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtjiiSGvravwQTfnYHUHHBW_mZhz-K41UlQ3OlZdgKHz1joMZ1SIwHpWhLJbwcbJjJOXKjl1XOmOcH4HZ5i8Kp-HNI82jmrvhtSirf51QWQcYAukW5kDCJSSV8nZQXs3ab8QJKcxZpttw/s640/protonect_amd_radeon.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Protonect running on AMD FOSS OpenCL stack.</td></tr>
</tbody></table>
<h4>
LLVM Force FLAT instructions hack.</h4>
<div>
<span style="font-size: xx-small;">diff --git a/lib/Target/AMDGPU/AMDGPUSubtarget.cpp b/lib/Target/AMDGPU/AMDGPUSubtarget.cpp</span><br />
<span style="font-size: xx-small;">index 3c4b5e7..f6d500c 100644</span><br />
<span style="font-size: xx-small;">--- a/lib/Target/AMDGPU/AMDGPUSubtarget.cpp</span><br />
<span style="font-size: xx-small;">+++ b/lib/Target/AMDGPU/AMDGPUSubtarget.cpp</span><br />
<span style="font-size: xx-small;">@@ -46,7 +46,7 @@ AMDGPUSubtarget::initializeSubtargetDependencies(const Triple &TT,</span><br />
<span style="font-size: xx-small;"> // disable it.</span><br />
<br />
<span style="font-size: xx-small;"> SmallString<256> FullFS("+promote-alloca,+fp64-denormals,+load-store-opt,");</span><br />
<span style="font-size: xx-small;">- if (isAmdHsaOS()) // Turn on FlatForGlobal for HSA.</span><br />
<span style="font-size: xx-small;">+ if (1 || isAmdHsaOS()) // Turn on FlatForGlobal for HSA.</span><br />
<span style="font-size: xx-small;"> FullFS += "+flat-for-global,+unaligned-buffer-access,";</span><br />
<span style="font-size: xx-small;"> FullFS += FS;</span></div>
<h4>
Patch for Libfreenect2</h4>
<span style="font-size: xx-small;">@@ -102,8 +102,8 @@ void kernel processPixelStage1(global const short *lut11to16, global const float</span><br />
<span style="font-size: xx-small;"> /*******************************************************************************</span><br />
<span style="font-size: xx-small;"> * Filter pixel stage 1</span><br />
<span style="font-size: xx-small;"> ******************************************************************************/</span><br />
<span style="font-size: xx-small;">-void kernel filterPixelStage1(global const float3 *a, global const float3 *b, global const float3 *n,</span><br />
<span style="font-size: xx-small;">- global float3 *a_out, global float3 *b_out, global uchar *max_edge_test)</span><br />
<span style="font-size: xx-small;">+void kernel filterPixelStage1(__global const float3 *a, __global const float3 *b, __global const float3 *n,</span><br />
<span style="font-size: xx-small;">+ __global float3 *a_out, __global float3 *b_out, __global uchar *max_edge_test)</span><br />
<span style="font-size: xx-small;"> {</span><br />
<span style="font-size: xx-small;"> const uint i = get_global_id(0);</span><br />
<br />
<span style="font-size: xx-small;">@@ -113,7 +113,7 @@ void kernel filterPixelStage1(global const float3 *a, global const float3 *b, gl</span><br />
<span style="font-size: xx-small;"> const float3 self_a = a[i];</span><br />
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;"> const float3 self_b = b[i];</span><br />
<br />
<span style="font-size: xx-small;">- const float gaussian[9] = {GAUSSIAN_KERNEL_0, GAUSSIAN_KERNEL_1, GAUSSIAN_KERNEL_2, GAUSSIAN_KERNEL_3, GAUSSIAN_KERNEL_4, GAUSSIAN_KERNEL_5, GAUSSIAN_KERNEL_6, GAUSSIAN_KERNEL_7, GAUSSIAN_KERNEL_8};</span><br />
<span style="font-size: xx-small;">+ //const float gaussian[9] = {GAUSSIAN_KERNEL_0, GAUSSIAN_KERNEL_1, GAUSSIAN_KERNEL_2, GAUSSIAN_KERNEL_3, GAUSSIAN_KERNEL_4, GAUSSIAN_KERNEL_5, GAUSSIAN_KERNEL_6, GAUSSIAN_KERNEL_7, GAUSSIAN_KERNEL_8};</span><br />
<br />
<span style="font-size: xx-small;"> if(x < 1 || y < 1 || x > 510 || y > 422)</span><br />
<span style="font-size: xx-small;"> {</span><br />
<span style="font-size: xx-small;">@@ -155,7 +155,9 @@ void kernel filterPixelStage1(global const float3 *a, global const float3 *b, gl</span><br />
<span style="font-size: xx-small;"> const int3 c1 = isless(other_norm * other_norm, threshold);</span><br />
<br />
<span style="font-size: xx-small;"> const float3 dist = 0.5f * (1.0f - (self_normalized_a * other_normalized_a + self_normalized_b * other_normalized_b));</span><br />
<span style="font-size: xx-small;">- const float3 weight = select(gaussian[j] * exp(-1.442695f * joint_bilateral_exp * dist), (float3)(0.0f), c1);</span><br />
<span style="font-size: xx-small;">+ //const float3 weight = 1.0f;//select(gaussian[j] * exp(-1.442695f * joint_bilateral_exp * dist), (float3)(0.0f), c1);</span><br />
<span style="font-size: xx-small;">+ const float gj = exp(0.6 - (0.3 * (abs(yi) + abs(xi))));</span><br />
<span style="font-size: xx-small;">+ const float3 weight = select(gj * exp(-1.442695f * joint_bilateral_exp * dist), (float3)(0.0f), c1);</span><br />
<br />
<h4>
LLVM Patch for assertion in AsmPrinter destructor.</h4>
<span style="font-size: xx-small;">diff --git a/lib/CodeGen/AsmPrinter/AsmPrinter.cpp b/lib/CodeGen/AsmPrinter/AsmPrinter.cpp</span><br />
<span style="font-size: xx-small;">index 0fed4e9..0d63a2a 100644</span><br />
<span style="font-size: xx-small;">--- a/lib/CodeGen/AsmPrinter/AsmPrinter.cpp</span><br />
<span style="font-size: xx-small;">+++ b/lib/CodeGen/AsmPrinter/AsmPrinter.cpp</span><br />
<span style="font-size: xx-small;">@@ -114,6 +114,7 @@ AsmPrinter::AsmPrinter(TargetMachine &tm, std::unique_ptr<MCStreamer> Streamer)</span><br />
<span style="font-size: xx-small;"> }</span><br />
<br />
<span style="font-size: xx-small;"> AsmPrinter::~AsmPrinter() {</span><br />
<span style="font-size: xx-small;">+ return;</span><br />
<span style="font-size: xx-small;"> assert(!DD && Handlers.empty() && "Debug/EH info didn't get finalized");</span><br />
<br />
<span style="font-size: xx-small;"> if (GCMetadataPrinters) {</span><br />
<br />
<h4>
Patch for debugging Address Space issues.</h4>
<span style="font-size: xx-small;">diff --git a/lib/Target/AMDGPU/AMDGPUISelLowering.cpp b/lib/Target/AMDGPU/AMDGPUISelLowering.cpp</span><br />
<span style="font-size: xx-small;">index 682157b..d2a5c4a 100644</span><br />
<span style="font-size: xx-small;">--- a/lib/Target/AMDGPU/AMDGPUISelLowering.cpp</span><br />
<span style="font-size: xx-small;">+++ b/lib/Target/AMDGPU/AMDGPUISelLowering.cpp</span><br />
<span style="font-size: xx-small;">@@ -766,6 +766,8 @@ SDValue AMDGPUTargetLowering::LowerGlobalAddress(AMDGPUMachineFunction* MFI,</span><br />
<span style="font-size: xx-small;"> unsigned Offset = MFI->allocateLDSGlobal(DL, *GV);</span><br />
<span style="font-size: xx-small;"> return DAG.getConstant(Offset, SDLoc(Op), Op.getValueType());</span><br />
<span style="font-size: xx-small;"> }</span><br />
<span style="font-size: xx-small;">+ default:</span><br />
<span style="font-size: xx-small;">+ printf("%s: address space type=%d\n", __func__, G->getAddressSpace());</span><br />
<span style="font-size: xx-small;"> }</span><br />
<br />
<span style="font-size: xx-small;"> const Function &Fn = *DAG.getMachineFunction().getFunction();</span></div>
<h3>
OpenCL Bandwidth Test (AMD App SDK)</h3>
<h3>
Intel (Beignet GPGPU Driver)</h3>
<span style="font-size: xx-small;">Platform found : Intel</span><br />
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;">Device 0 Intel(R) HD Graphics Haswell GT2 Desktop</span><br />
<span style="font-size: xx-small;">Build: release</span><br />
<span style="font-size: xx-small;">GPU work items: 32768</span><br />
<span style="font-size: xx-small;">Buffer size: 33554432</span><br />
<span style="font-size: xx-small;">CPU workers: 1</span><br />
<span style="font-size: xx-small;">Timing loops: 20</span><br />
<span style="font-size: xx-small;">Repeats: 1</span><br />
<span style="font-size: xx-small;">Kernel loops: 20</span><br />
<span style="font-size: xx-small;">inputBuffer: CL_MEM_READ_ONLY </span><br />
<span style="font-size: xx-small;">outputBuffer: CL_MEM_WRITE_ONLY </span><br />
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;">Host baseline (naive):</span><br />
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;">Timer resolution 256.11 ns</span><br />
<span style="font-size: xx-small;">Page fault 531.44 ns</span><br />
<span style="font-size: xx-small;">CPU read 15.31 GB/s</span><br />
<span style="font-size: xx-small;">memcpy() 15.54 GB/s</span><br />
<span style="font-size: xx-small;">memset(,1,) 26.54 GB/s</span><br />
<span style="font-size: xx-small;">memset(,0,) 27.06 GB/s</span><br />
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;">AVERAGES (over loops 2 - 19, use -l for complete log)</span><br />
<span style="font-size: xx-small;">--------</span><br />
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;">1. Host mapped write to inputBuffer</span><br />
<span style="font-size: xx-small;"> ---------------------------------------|---------------</span><br />
<span style="font-size: xx-small;"> clEnqueueMapBuffer -- WRITE (GBPS) | 9513.290</span><br />
<span style="font-size: xx-small;"> ---------------------------------------|---------------</span><br />
<span style="font-size: xx-small;"> memset() (GBPS) | 24.746</span><br />
<span style="font-size: xx-small;"> ---------------------------------------|---------------</span><br />
<span style="font-size: xx-small;"> clEnqueueUnmapMemObject() (GBPS) | 6176.168</span><br />
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;">2. GPU kernel read of inputBuffer</span><br />
<span style="font-size: xx-small;"> ---------------------------------------|---------------</span><br />
<span style="font-size: xx-small;"> clEnqueueNDRangeKernel() (GBPS) | 38.225</span><br />
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;"> Verification Passed!</span><br />
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;">3. GPU kernel write to outputBuffer</span><br />
<span style="font-size: xx-small;"> ---------------------------------------|---------------</span><br />
<span style="font-size: xx-small;"> clEnqueueNDRangeKernel() (GBPS) | 26.198</span><br />
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;">4. Host mapped read of outputBuffer</span><br />
<span style="font-size: xx-small;"> ---------------------------------------|---------------</span><br />
<span style="font-size: xx-small;"> clEnqueueMapBuffer -- READ (GBPS) | 9830.400</span><br />
<span style="font-size: xx-small;"> ---------------------------------------|---------------</span><br />
<span style="font-size: xx-small;"> CPU read (GBPS) | 15.431</span><br />
<span style="font-size: xx-small;"> ---------------------------------------|---------------</span><br />
<span style="font-size: xx-small;"> clEnqueueUnmapMemObject() (GBPS) | 10485.760</span><br />
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;"> Verification Passed!</span><br />
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;">Passed!</span><br />
<h4>
AMD Radeon (OpenCL) </h4>
<span style="font-size: xx-small;">Platform found : Mesa</span><br />
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;">Device 0 AMD POLARIS10 (DRM 3.2.0 / 4.7.0-rc7-meow+, LLVM 4.0.0)</span><br />
<span style="font-size: xx-small;">Build: release</span><br />
<span style="font-size: xx-small;">GPU work items: 32768</span><br />
<span style="font-size: xx-small;">Buffer size: 33554432</span><br />
<span style="font-size: xx-small;">CPU workers: 1</span><br />
<span style="font-size: xx-small;">Timing loops: 20</span><br />
<span style="font-size: xx-small;">Repeats: 1</span><br />
<span style="font-size: xx-small;">Kernel loops: 20</span><br />
<span style="font-size: xx-small;">inputBuffer: CL_MEM_READ_ONLY </span><br />
<span style="font-size: xx-small;">outputBuffer: CL_MEM_WRITE_ONLY </span><br />
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;">Host baseline (naive):</span><br />
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;">Timer resolution 256.12 ns</span><br />
<span style="font-size: xx-small;">Page fault 538.31 ns</span><br />
<span style="font-size: xx-small;">CPU read 12.19 GB/s</span><br />
<span style="font-size: xx-small;">memcpy() 11.38 GB/s</span><br />
<span style="font-size: xx-small;">memset(,1,) 20.93 GB/s</span><br />
<span style="font-size: xx-small;">memset(,0,) 22.98 GB/s</span><br />
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;">AVERAGES (over loops 2 - 19, use -l for complete log)</span><br />
<span style="font-size: xx-small;">--------</span><br />
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;">1. Host mapped write to inputBuffer</span><br />
<span style="font-size: xx-small;"> ---------------------------------------|---------------</span><br />
<span style="font-size: xx-small;"> clEnqueueMapBuffer -- WRITE (GBPS) | 7586.161</span><br />
<span style="font-size: xx-small;"> ---------------------------------------|---------------</span><br />
<span style="font-size: xx-small;"> memset() (GBPS) | 6.369</span><br />
<span style="font-size: xx-small;"> ---------------------------------------|---------------</span><br />
<span style="font-size: xx-small;"> clEnqueueUnmapMemObject() (GBPS) | 12822.261</span><br />
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;">2. GPU kernel read of inputBuffer</span><br />
<span style="font-size: xx-small;"> ---------------------------------------|---------------</span><br />
<span style="font-size: xx-small;"> clEnqueueNDRangeKernel() (GBPS) | 113.481</span><br />
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;"> Verification Passed!</span><br />
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;">3. GPU kernel write to outputBuffer</span><br />
<span style="font-size: xx-small;"> ---------------------------------------|---------------</span><br />
<span style="font-size: xx-small;"> clEnqueueNDRangeKernel() (GBPS) | 105.898</span><br />
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;">4. Host mapped read of outputBuffer</span><br />
<span style="font-size: xx-small;"> ---------------------------------------|---------------</span><br />
<span style="font-size: xx-small;"> clEnqueueMapBuffer -- READ (GBPS) | 9.559</span><br />
<span style="font-size: xx-small;"> ---------------------------------------|---------------</span><br />
<span style="font-size: xx-small;"> CPU read (GBPS) | 17.179</span><br />
<span style="font-size: xx-small;"> ---------------------------------------|---------------</span><br />
<span style="font-size: xx-small;"> clEnqueueUnmapMemObject() (GBPS) | 4060.750</span><br />
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;"> Verification Passed!</span><br />
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;"><br /></span>
<span style="font-size: xx-small;">Passed!</span>I hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com1tag:blogger.com,1999:blog-1416969656167659289.post-73650929734108551932016-03-10T06:24:00.003+03:002016-03-10T06:32:16.767+03:00Fuzzing Vulkans, how do they work?<h2>
Introduction</h2>
<br />
Disclaimer: I have not yet fully read the specs on SPIR-V or Vulkan.<br />
<br />
I decided to find out how hard it is to crash code working with SPIR-V.<br />
Initially I wanted to crash the actual GPU drivers but for a start I decided<br />
to experiment with GLSLang.<br />
<br />
<h2>
What I got</h2>
<br />
I used the "afl-fuzz" fuzzer to generate test cases that could crash<br />
the parser. I have briefly examined the generated cases.<br />
I have uploaded the results (which contain the SPIR-V binaries causing<br />
the "spirv-remap" to crash) to the [following location](<a href="https://drive.google.com/file/d/0B7wcN-tOkdeRTGItSDhFM0JYUEk/view?usp=sharing">https://drive.google.com/file/d/0B7wcN-tOkdeRTGItSDhFM0JYUEk/view?usp=sharing</a>)<br />
<br />
Some of them trigger assertions in the code (which is not bad, but perhaps<br />
returning an error code and shutting down cleanly would be better).<br />
Some of them cause the code to hang for long time or indefinitely (which is worse<br />
especially if someone intends to use the SPIR-V parser code online in the app).<br />
<br />
Perhaps some of the results marked as "hangs" just cause too long compilation<br />
time and could produce more interesting results if the timeout in "afl-fuzz"<br />
is increased.<br />
<br />
Two notable examples causing long compilation time are:<br />
"out/crashes/id:000000,sig:06,src:000000,op:flip1,pos:15"<br />
"out/hangs/id:000011,src:000000,op:flip1,pos:538" - for this one I waited for<br />
a minute but it stil did not complete the compilation while causing 100% CPU load.<br />
<br />
A log output of "glslang" indicating that most of the error cases found are handled, but with "abort" instead of graceful shutdown.<br />
<a href="http://pastebin.com/BnZ63tKJ">http://pastebin.com/BnZ63tKJ</a><br />
<br />
<h3>
NVIDIA</h3>
I have also tried using these shaders with the NVIDIA driver (since it was the only hardware I could run a real Vulkan driver on).<br />
<br />
I have used the "instancing" demo from [SaschaWillems Repository](<a href="https://github.com/SaschaWillems/Vulkan">https://github.com/SaschaWillems/Vulkan</a>) .<br />
I patched it to accept the path to the binary shader via the command line.<br />
Next, I fed it with the generated<br />
test cases. Some of them triggered segfaults inside the NVIDIA driver.<br />
What is curious is that when i used the "hangs" examples, they also caused<br />
the NVIDIA driver to take extra long time to compile and eventually crash<br />
at random places.<br />
<br />
I think it indicates either that there is some common code between the driver<br />
and GLSLang (the reference implementation) or the specification is missing<br />
some sanity check somewhere and the compiler can get stuck optimizing certain<br />
code.<br />
Is there a place in specification that mandates that all the values are<br />
checked to be within the allowed range, and all complex structures (such as<br />
function calls) are checked recursively?<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQ-uUyqIQT-i66ji5T8CZhr4kSlvf10JZSa8Eqpqx-79XH1_rfb-erll7L41lBrKnVmFVFpIbHLOAI-bSwxErJvrQDBsq9iVySts48Ch8vt85FgLfnteH1K79pYqO7s33kn00bkKAVLeI/s1600/Screen+Shot+2016-03-10+at+05.59.05.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="60" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQ-uUyqIQT-i66ji5T8CZhr4kSlvf10JZSa8Eqpqx-79XH1_rfb-erll7L41lBrKnVmFVFpIbHLOAI-bSwxErJvrQDBsq9iVySts48Ch8vt85FgLfnteH1K79pYqO7s33kn00bkKAVLeI/s400/Screen+Shot+2016-03-10+at+05.59.05.png" width="400" /></a></div>
<br />
<br />
Perhaps I should have a look at other drivers (Mali anyone?).<br />
<br />
<span style="background-color: #cccccc;">```</span><br />
<span style="background-color: #cccccc;">[ 3672.137509] instancing[26631]: segfault at f ip 00007fb4624adebf sp 00007ffefd72e100 error 4 in libnvidia-glcore.so.355.00.29[7fb462169000+1303000]</span><br />
<span style="background-color: #cccccc;">[ 3914.294222] instancing[26894]: segfault at f ip 00007f00b28fcebf sp 00007ffdb9bab980 error 4 in libnvidia-glcore.so.355.00.29[7f00b25b8000+1303000]</span><br />
<span style="background-color: #cccccc;">[ 4032.430179] instancing[27017]: segfault at f ip 00007f7682747ebf sp 00007fff46679bf0 error 4 in libnvidia-glcore.so.355.00.29[7f7682403000+1303000]</span><br />
<span style="background-color: #cccccc;">[ 4032.915849] instancing[27022]: segfault at f ip 00007fb4e4099ebf sp 00007fff3c1ac0f0 error 4 in libnvidia-glcore.so.355.00.29[7fb4e3d55000+1303000]</span><br />
<span style="background-color: #cccccc;">[ 4033.011699] instancing[27023]: segfault at f ip 00007f7272900ebf sp 00007ffdb54261e0 error 4 in libnvidia-glcore.so.355.00.29[7f72725bc000+1303000]</span><br />
<span style="background-color: #cccccc;">[ 4033.107939] instancing[27025]: segfault at f ip 00007fbf0353debf sp 00007ffde4387750 error 4 in libnvidia-glcore.so.355.00.29[7fbf031f9000+1303000]</span><br />
<span style="background-color: #cccccc;">[ 4033.203924] instancing[27026]: segfault at f ip 00007f0f9a6f0ebf sp 00007ffff85a9dd0 error 4 in libnvidia-glcore.so.355.00.29[7f0f9a3ac000+1303000]</span><br />
<span style="background-color: #cccccc;">[ 4033.299138] instancing[27027]: segfault at 2967000 ip 00007fcb42cab720 sp 00007ffcbad45228 error 6 in libc-2.19.so[7fcb42c26000+19f000]</span><br />
<span style="background-color: #cccccc;">[ 4033.394667] instancing[27028]: segfault at 36d2000 ip 00007efc789eb720 sp 00007fff26c636d8 error 6 in libc-2.19.so[7efc78966000+19f000]</span><br />
<span style="background-color: #cccccc;">[ 4033.490918] instancing[27029]: segfault at 167b15e170 ip 00007f3b02095ec3 sp 00007ffd768cbf68 error 4 in libnvidia-glcore.so.355.00.29[7f3b01cbe000+1303000]</span><br />
<span style="background-color: #cccccc;">[ 4033.586699] instancing[27030]: segfault at 2ffc000 ip 00007fdebcc06720 sp 00007fff4fe59bd8 error 6 in libc-2.19.so[7fdebcb81000+19f000]</span><br />
<span style="background-color: #cccccc;">[ 4033.682939] instancing[27031]: segfault at 8 ip 00007fb80e7eed50 sp 00007ffe9cd21de0 error 4 in libnvidia-glcore.so.355.00.29[7fb80e410000+1303000]</span><br />
<span style="background-color: #cccccc;">[ 4374.480872] show_signal_msg: 27 callbacks suppressed</span><br />
<span style="background-color: #cccccc;">[ 4374.480876] instancing[27402]: segfault at f ip 00007fd1fc3cdebf sp 00007ffe483ff520 error 4 in libnvidia-glcore.so.355.00.29[7fd1fc089000+1303000]</span><br />
<span style="background-color: #cccccc;">[ 4374.809621] instancing[27417]: segfault at 2e0c3910 ip 00007f39af846e96 sp 00007ffe1c6d8f10 error 6 in libnvidia-glcore.so.355.00.29[7f39af46f000+1303000]</span><br />
<span style="background-color: #cccccc;">[ 4374.905112] instancing[27418]: segfault at 2dc46a68 ip 00007f7b9ff7af32 sp 00007fff290edf00 error 6 in libnvidia-glcore.so.355.00.29[7f7b9fba2000+1303000]</span><br />
<span style="background-color: #cccccc;">[ 4375.001019] instancing[27419]: segfault at f ip 00007f5a4e066ebf sp 00007ffe0b775d70 error 4 in libnvidia-glcore.so.355.00.29[7f5a4dd22000+1303000]</span><br />
<span style="background-color: #cccccc;">[ 4375.096894] instancing[27420]: segfault at f ip 00007f7274d49ebf sp 00007ffe96fdea10 error 4 in libnvidia-glcore.so.355.00.29[7f7274a05000+1303000]</span><br />
<span style="background-color: #cccccc;">[ 4375.193165] instancing[27421]: segfault at f ip 00007fa3bf3c8ebf sp 00007ffc4117e8d0 error 4 in libnvidia-glcore.so.355.00.29[7fa3bf084000+1303000]</span><br />
<span style="background-color: #cccccc;">[ 4375.288969] instancing[27423]: segfault at f ip 00007f50e0327ebf sp 00007ffc02aa1d50 error 4 in libnvidia-glcore.so.355.00.29[7f50dffe3000+1303000]</span><br />
<span style="background-color: #cccccc;">[ 4375.385530] instancing[27424]: segfault at f ip 00007f0d9a32eebf sp 00007ffd0298eb40 error 4 in libnvidia-glcore.so.355.00.29[7f0d99fea000+1303000]</span><br />
<span style="background-color: #cccccc;">[ 4375.481829] instancing[27425]: segfault at f ip 00007f8400bc5ebf sp 00007ffef0334240 error 4 in libnvidia-glcore.so.355.00.29[7f8400881000+1303000]</span><br />
<span style="background-color: #cccccc;">[ 4375.576983] instancing[27426]: segfault at 2dec2bc8 ip 00007f52260afec3 sp 00007fffd2bd1728 error 4 in libnvidia-glcore.so.355.00.29[7f5225cd8000+1303000]</span><br />
<span style="background-color: #cccccc;">```</span><br />
<br />
<h2>
How to reproduce</h2>
<br />
Below are the steps I have taken to crash the "spirv-remap" tool.<br />
I believe this issue is worth looking at because some vendors may<br />
choose to build their driver internals based on the reference implementations<br />
which may lead to bugs directly crippling into the software as-is.<br />
<br />
0. I have used the Debian Linux box. I have installed the "afl-fuzz" tool,<br />
and also manually copied the Vulkan headers to "/usr/include".<br />
<br />
1. cloned the GLSLang repository<br />
<span style="background-color: yellow;">```</span><br />
<span style="background-color: yellow;">git clone git@github.com:KhronosGroup/glslang.git</span><br />
<span style="background-color: yellow;">cd glslang</span><br />
<span style="background-color: yellow;">```</span><br />
<br />
2. Compiled it with afl-fuzz<br />
<span style="background-color: yellow;">```</span><br />
<span style="background-color: yellow;"><span class="Apple-tab-span" style="white-space: pre;"> </span>mkdir build</span><br />
<span style="background-color: yellow;"><span class="Apple-tab-span" style="white-space: pre;"> </span>cat SetupLinux.sh </span><br />
<span style="background-color: yellow;"><span class="Apple-tab-span" style="white-space: pre;"> </span>cd build/</span><br />
<span style="background-color: yellow;"><span class="Apple-tab-span" style="white-space: pre;"> </span>cmake -DCMAKE_C_COMPILER=afl-gcc -DCMAKE_CXX_COMPILER=afl-g++ ..</span><br />
<span style="background-color: yellow;"><span class="Apple-tab-span" style="white-space: pre;"> </span>cd ..</span><br />
<span style="background-color: yellow;">```</span><br />
<br />
3. Compiled a sample shader from the GLSL to the SPIR-V format using<br />
<span style="background-color: yellow;">```</span><br />
<span style="background-color: yellow;"><span class="Apple-tab-span" style="white-space: pre;"> </span>./build/install/bin/glslangValidator -V -i Test/spv.130.frag</span><br />
<span style="background-color: yellow;">```</span><br />
<br />
4. Verified that the "spirv-remap" tool works on the binary<br />
<span style="background-color: yellow;">```</span><br />
<span style="background-color: yellow;"><span class="Apple-tab-span" style="white-space: pre;"> </span>./build/install/bin/spirv-remap -v -i frag.spv -o /tmp/</span><br />
<span style="background-color: yellow;">```</span><br />
<br />
5. Fed the SPIR-V binary to the afl-fuzz<br />
<span style="background-color: yellow;">```</span><br />
<span style="background-color: yellow;"><span class="Apple-tab-span" style="white-space: pre;"> </span>afl-fuzz -i in -o out ./build/install/bin/spirv-remap -v -i @@ -o /tmp</span><br />
<span style="background-color: yellow;">```</span><br />
<br />
6. Quickly discovered several crashes. I attach the screenshot of afl-fuzz<br />
in the work.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2untV_nlTtvum3Wcg__vNYeaxAmRTn7UQ9aKqXkKfsK-d0zJ5RXFHri4bWoHXMnykYWEPWBFQsqAvKB1pyCQgSny8gxgfUI3X_PQgG8y4WfXd88d-l3Hd6XbOpo-CQ4cxUFuS5l17fwI/s1600/Screen+Shot+2016-03-10+at+03.36.11.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="204" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2untV_nlTtvum3Wcg__vNYeaxAmRTn7UQ9aKqXkKfsK-d0zJ5RXFHri4bWoHXMnykYWEPWBFQsqAvKB1pyCQgSny8gxgfUI3X_PQgG8y4WfXd88d-l3Hd6XbOpo-CQ4cxUFuS5l17fwI/s320/Screen+Shot+2016-03-10+at+03.36.11.png" width="320" /></a></div>
<br />
<br />
7. Examined them.<br />
<br />
First, I made a hex diff of the good and bad files. The command to generate<br />
the diff is the following:<br />
<span style="background-color: yellow;">```</span><br />
<span style="background-color: yellow;"><span class="Apple-tab-span" style="white-space: pre;"> </span>for i in out/crashes/*; do hexdump in/frag.spv > in.hex && hexdump $i > out.hex && diff -Naur in.hex out.hex; done > hex.diff</span><br />
<span style="background-color: yellow;">```</span><br />
<br />
Next, I just ran the tool on all cases and generated the log of crash messages.<br />
<span style="background-color: yellow;">```</span><br />
<span style="background-color: yellow;"><span class="Apple-tab-span" style="white-space: pre;"> </span>for i in out/crashes/*; do echo $i && ./build/install/bin/spirv-remap -i $i -o /tmp/ 2>&1 ;done > abort.log</span><br />
<span style="background-color: yellow;">```</span><br />
<div>
<br /></div>
<h2>
Conclusions</h2>
<div>
Well, there are two obvious conclusions:</div>
<div>
1. Vulkan/SPIR-V is still a WIP and drivers are not yet perfect</div>
<div>
2. GPU drivers have always been notorious for poor compilers - not only codegen, but parsers and validators. Maybe part of the reason is that CPU compilers simply handle more complex code and therefore more edge cases have been hit already.</div>
I hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com0tag:blogger.com,1999:blog-1416969656167659289.post-85402014337263707342016-02-25T06:07:00.001+03:002016-02-25T06:11:08.784+03:00Notes on firmware and tools<h4>
Introduction.</h4>
In this blog post I mainly want to summarize my latest experiments at using clang's static analyzer and some thoughts on what could be further done at the analyzer, and open-source software quality in general. It's mostly some notes I've decided to put up.<br />
<br />
Here are the references to the people involved in developing clang static analyzer at Apple. I recommend following them on twitter and also reading their presentation on developing custom checkers for the analyzer.<br />
Ted Kremenek - <a href="https://twitter.com/tkremenek">https://twitter.com/tkremenek</a><br />
Anna Zaks - <a href="https://twitter.com/zaks_anna">https://twitter.com/zaks_anna</a><br />
Jordan Rose - <a href="https://twitter.com/UINT_MIN">https://twitter.com/UINT_MIN</a><br />
<br />
"How to Write a Checker in 24 Hours" - <a href="http://llvm.org/devmtg/2012-11/Zaks-Rose-Checker24Hours.pdf">http://llvm.org/devmtg/2012-11/Zaks-Rose-Checker24Hours.pdf</a><br />
"Checker Developer Manual" - <a href="http://clang-analyzer.llvm.org/checker_dev_manual.html">http://clang-analyzer.llvm.org/checker_dev_manual.html</a> - This one requires some understanding of LLVM, so I recommend to get comfortable with using the analyzer and looking at AST dumps first.<br />
<br />
There are not so many analyzer plugins which not made at Apple.<br />
<br />
<ul>
<li>The "Tartan" which is intended to check the GLib conventions used in GNOME projects. <a href="https://www.freedesktop.org/software/tartan/">https://www.freedesktop.org/software/tartan/</a></li>
<li>A presentation called "MPI-Checker – Static Analysis for MPI" <a href="https://llvm-hpc2-workshop.github.io/slides/Droste.pdf">https://llvm-hpc2-workshop.github.io/slides/Droste.pdf</a></li>
</ul>
<br />
<h4>
GLibc.</h4>
Out of curiosity I ran the clang analyzer on glibc. Setting it up was not a big deal - in fact, all that was needed was to just run the scan-build. It did a good job of intercepting the gcc calls and most of the sources were successfully analyzed. In fact, I did not do anything special, but even running the analyzer with the default configuration revealed a few true bugs, like the one showed in the screenshot.<br />
<br />
For example, the "iconv" function has a check to see if the argument "outbuf" is NULL, which indicates that it is a valid case expected by the authors. The manpage for the function also says that passing a NULL argument as "outbuf" is valid. However, we can see that one of the branches is missing the similar check for NULL pointer, which probably resulted from a copy-paste in the past. So, passing valid pointers to "inbuf" and "inbytesleft" and a NULL pointer for "outbuf" leads to the NULL pointer dereference and consequently a SIGSEGV.<br />
<br />
Fun fact: my example is also not quite correct, as pointed out by my Twitter followers. The third argument to iconv must be a pointer to the integer, not the integer. However, on my machine my example crashes at dereferencing the "outbuf" pointer and not the "inbytesleft", because the argument evaluation order in C is not specified, and on x86_64 arguments are usually pushed to the stack (and therefore evaluated) in the reverse order.<br />
<br />
Is it a big deal? Who knows. On the one hand, it's a userspace library, and unlike OpenSSL, no one is likely to embed it into kernel-level firmware. On the other, I may very well imagine a device such as a router or a web kiosk where this NULL pointer dereference could be triggered, because internalization and text manipulation is always a complex issue.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhoumfstWAm29sMrPfR1-FkYk04zKqcyjc50ya6QTYs__TsC4y4AentA0reJoPGD37Dq-yrF4OSazxu8qjByrO4aWPYMArbv_XCdtlWG8wHTEJmCV8QXWuCaDSNgR1mswDxu4ma1uNpLKs/s1600/screenshot_glibc_clang.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="555" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhoumfstWAm29sMrPfR1-FkYk04zKqcyjc50ya6QTYs__TsC4y4AentA0reJoPGD37Dq-yrF4OSazxu8qjByrO4aWPYMArbv_XCdtlWG8wHTEJmCV8QXWuCaDSNgR1mswDxu4ma1uNpLKs/s640/screenshot_glibc_clang.png" width="640" /></a></div>
<br />
<h4>
Linux Kernel.</h4>
I had this idea of trying to build LLVMLinux with clang for quite a while, but never really had the time to do it. My main interest in doing so was using the clang static analyzer.<br />
<br />
Currently, some parts of Linux Kernel fail to build with clang, so I had to use the patches from the LLVMLinux project. They failed to apply cleanly though. I had to manually edit several patches. Another problem is that "git am" does not support the "fuzzy" strategy when applying the patches, so I had to use a certain script found on GitHub which uses "patch" and "git commit" to do the same thing.<br />
<a href="https://gist.github.com/kfish/7425248">https://gist.github.com/kfish/7425248</a><br />
<br />
I have pushed my tree to github. I based it off the latest upstream at the time when I worked on it (which was the start of February 2016). The branch is called "4.5_with_llvmlinux".<br />
<a href="https://github.com/astarasikov/linux/tree/4.5_with_llvmlinux">https://github.com/astarasikov/linux/tree/4.5_with_llvmlinux</a><br />
<br />
I've used the following commands to get the analysis results. Note that some files have failed to compile, and I had to manually stop the compilation job for one file that took over 40 minutes.<br />
<span style="background-color: #666666;"><span style="color: cyan;"><br /></span></span>
<span style="background-color: #666666; color: lime;">export CCC_CXX=clang++</span><br />
<span style="background-color: #666666; color: lime;">export CCC_CC=clang</span><br />
<div class="p1">
<span style="background-color: #666666; color: lime;"><span class="s1">scan-build</span><span class="s2"> make CC=clang HOSTCC=clang -j10 -k</span></span></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHSzvfPGz3O3EhlGf_k8YL71XDO7Np8s7vItj9ocMrORnA3L0H0OiJKmLFuY6oHNaIXPe0HtxdRthrumlEO-8yqoegSCe0TL0Mg5Z11UEAyTGrJ1S9uA8EiX-DFP1AVs1-9dOenLxHKJU/s1600/scan_build_linux_kernel.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="127" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHSzvfPGz3O3EhlGf_k8YL71XDO7Np8s7vItj9ocMrORnA3L0H0OiJKmLFuY6oHNaIXPe0HtxdRthrumlEO-8yqoegSCe0TL0Mg5Z11UEAyTGrJ1S9uA8EiX-DFP1AVs1-9dOenLxHKJU/s320/scan_build_linux_kernel.png" width="320" /></a></div>
<br />
<h3>
Ideas.</h3>
<h4>
Porting clang instrumentations to major OpenSource projects.</h4>
<div>
Clang has a number of instrumentation plugins called "Sanitizers" which were largely developed by Konstantin Serebryany and Dmitry Vyukov at Google.</div>
<div>
<br /></div>
<div>
Arguably the most useful tool for C code is AddressSanitizer which allows to catch Use-After-Free and Out-of-Bounds access on arrays. There exists a port of the tool for the Linux Kernel, called Kernel AddressSanitizer, or KASAN, and it has been used to uncover a lot of memory corruption bugs, leading to potential vulnerabilities or kernel panics.</div>
<div>
Another tool, also based on compile-time instrumentation, is the ThreadSanitizer for catching data races, and it has also been ported to the Linux Kernel.</div>
<div>
<br /></div>
<div>
<a href="https://github.com/google/kasan/wiki">https://github.com/google/kasan/wiki</a></div>
<div>
<a href="https://github.com/google/kasan/wiki/Found-Bugs">https://github.com/google/kasan/wiki/Found-Bugs</a></div>
<div>
<br /></div>
<div>
I think it would be very useful to port these tools to the other projects. There are a lot of system-level software driving the critical aspects of system initialization process. To name a few:</div>
<div>
<ul>
<li>EDK2 UEFI Development Kit</li>
<li>LittleKernel bootloader by Travis Geiselbrecht. It has been extensively used in the embedded world instead of u-boot lately. Qualcomm (and Codeaurora, its Open-Source department) is using a customized version of LK for its SoCs, and nearly every mobile phone with Android has LK inside it. NVIDIA have recently started shipping a version of LK called "TLK" or "Trusted Little Kernel" in its K1 processors, with the largest deployment yet being the Google Nexus 9 tablet.</li>
<li>U-boot bootloader, which is still used in a lot of devices.</li>
<li>FreeBSD and other kernels. While I don't know how widely they are deployed today, it would still be useful, at least for junior and intermediate kernel hackers.</li>
<li>XNU Kernel powering Apple iOS and OS X. I have some feeling that Apple might be using some of the tools internally, though the public snapshots of the source are heavily stripped of newest features.</li>
</ul>
</div>
<div>
Btw, if anyone is struggling at getting userspace AddressSanitizer working with NVIDIA drivers, take a look at this thread where I posted a workaround. Long story short, NVIDIA mmaps memory at large chunks up to a certain range, and you need to change the virtual address which ASan claims for itself.</div>
<div>
<a href="http://marc.info/?l=llvm-dev&m=141168078303757&w=2">http://marc.info/?l=llvm-dev&m=141168078303757&w=2</a><br />
<a href="http://marc.info/?l=llvm-dev&m=141169586008101&w=2">http://marc.info/?l=llvm-dev&m=141169586008101&w=2</a></div>
<div>
<br /></div>
<div>
Note that you will likely get problems if you build a library with a customized ASan and link it to something with unpatched ASan, so I recommend using a distro where you can rebuild all packages simultaneously if you want to experiment with custom instrumentation, such as NixOS, Gentoo or *BSD.</div>
<div>
<br /></div>
<div>
As I have <a href="http://allsoftwaresucks.blogspot.ru/2015/11/porting-legacy-rtos-software-to-custom.html">mentioned before</a>, there is a caveat with all these new instrumentation tools - most of them are made for the 64-bit architecture while the majority of the embedded code stays 32-bit. There are several ways to address it, such as running memory checkers in a separate process or using a simulator such as a modified QEMU, but it's an open problem.</div>
<div>
<div>
<h4>
Other techniques for firmware quality</h4>
<div>
In one of my previous posts I have drawn attention to gathering code coverage in low-level software, such as bootloaders. With GCOV being quite easy to port, I think more projects should be exposed to it. Also, AddressSanitizer now comes with a custom infrastructure for gathering code coverage, which is more extensive than GCOV.</div>
</div>
<div>
<br /></div>
<div>
Here's a link to the post which shows how one can easily embed GCOV to get coverage using Little Kernel as an example.</div>
<div>
<a href="http://allsoftwaresucks.blogspot.ru/2015/05/gcov-is-amazing-yet-undocumented.html">http://allsoftwaresucks.blogspot.ru/2015/05/gcov-is-amazing-yet-undocumented.html</a></div>
<div>
<br /></div>
<div>
While userspace applications have received quite a lot of fuzzing recently, especially with the introduction of the <a href="http://lcamtuf.coredump.cx/afl/">AFL fuzzer tool</a>, kernel-side received little attention. For Linux, there is a syscall fuzzer called Trinity.</div>
<div>
<a href="http://codemonkey.org.uk/projects/trinity/">http://codemonkey.org.uk/projects/trinity/</a></div>
<div>
<a href="https://github.com/kernelslacker/trinity">https://github.com/kernelslacker/trinity</a></div>
<div>
<br /></div>
<div>
There is also an interesting project at fuzzing kernel through USB stack (vUSBf).</div>
<div>
<a href="https://github.com/schumilo/vUSBf">https://github.com/schumilo/vUSBf</a></div>
<div>
<a href="https://www.blackhat.com/docs/eu-14/materials/eu-14-Schumilo-Dont-Trust-Your-USB-How-To-Find-Bugs-In-USB-Device-Drivers.pdf">https://www.blackhat.com/docs/eu-14/materials/eu-14-Schumilo-Dont-Trust-Your-USB-How-To-Find-Bugs-In-USB-Device-Drivers.pdf</a></div>
<div>
<br /></div>
<div>
What should be done is adapting this techniques for other firmware projects. On the one hand, it might get tricky due to the limited debugging facilities available in some kernels. On the other one, the capabilities provided by simulators such as QEMU are virtually unlimited (pun unintended).</div>
<div>
<br /></div>
<div>
An interesting observation might be that there are limited sources of external data input on a system. They include processor interrupt vectors/handlers and MMIO hardware. As for the latter, in Linux and most other firmwares, there are certain facilities for working with MMIO - such as functions like "readb()", "readw()" and "ioremap". Also, if we're speaking of a simulator such as QEMU, we can identify memory regions of interest by walking the page table and checking the physical address against external devices, and also checking the access type bits - for example, the data marked as executable is more likely to be the firmware code, while the uncached contiguous regions of memory are likely DMA windows.</div>
<div>
<br /></div>
<div>
ARM mbed TLS library is a good example of a project that tries to integrate as many dynamic and static tools into its testing process. However, it can be built as a userspace library on desktop, which makes it less interesting in the context of Firmware security.</div>
<div>
<a href="https://blog.gdssecurity.com/labs/2015/9/21/fuzzing-the-mbed-tls-library.html">https://blog.gdssecurity.com/labs/2015/9/21/fuzzing-the-mbed-tls-library.html</a></div>
<div>
<a href="https://tls.mbed.org/kb/generic/what-tests-and-checks-are-run-for-mbedtls">https://tls.mbed.org/kb/generic/what-tests-and-checks-are-run-for-mbedtls</a></div>
<div>
<a href="https://www.mbed.com/en/development/software/mbed-tls/">https://www.mbed.com/en/development/software/mbed-tls/</a></div>
<div>
<br /></div>
<div>
Another technique that has been catching my attention for a lot of time is symbolic execution. In many ways, it is a similar problem to the static analysis - you need to find a set of input values constrained by certain equations to trigger a specific execution path leading to the incorrect state (say, a NULL pointer dereference).</div>
<div>
<br /></div>
<div>
Rumors are, a tool based on this technique called SAGE is actively used at Microsoft Research to analyze internal code, but sadly there are not so many open-source and well-documented tools one can easily play with and directly plug into an existing project.</div>
<div>
<a href="https://www.blogger.com/goog_927574502"><br /></a></div>
<div>
An interesting example of applying this idea to the practical problem at a large and established corporation (Intel) is presented in the paper called "Symbolic execution for BIOS security" which tries to utilize symbolic execution with KLEE for attacking firmware - the SMM handler. You can find more interesting details in the blog of one of the authors of the paper - Vincent Zimmer (<a href="https://www.blogger.com/goog_927574502">https://twitter.com/vincentzimmer</a> and <a href="http://vzimmer.blogspot.com/">http://vzimmer.blogspot.com/</a>).</div>
</div>
<div>
<br /></div>
<div>
Also, not directly related, but an interesting presentation about bug hunting on Android.</div>
<div>
<a href="http://www.slideshare.net/jiahongfang5/qualcomm2015-jfang-nforest">http://www.slideshare.net/jiahongfang5/qualcomm2015-jfang-nforest</a><br />
<br />
I guess now I will have to focus on studying recent exploitation techniques used for hacking XNU, Linux and Chromium to have a more clear picture of what's needed to achieve :)</div>
<h4>
Further developing clang analyzer.</h4>
<div>
One feature missing from clang is the ability to perform the analysis across the translation units. A Translation Unit (or TU) in clang/LLVM roughly represents a file on the disk being parsed. Clang analyzer is implemented as a pass which traverses the AST and is limited to the one translation unit. Which is not quite true - it will go and analyze the includes recursively. But if you have two separate "C" files, which do not include one another, and a function from one file is calling the function from another one, the analysis will not work.</div>
<div>
<br /></div>
<div>
Implementing a checker across multiple sources is not a trivial thing, as pointed out by the analyzer authors in the mailing list, though this topic is often brought up by different people.</div>
<div>
<br /></div>
<div>
<a href="http://lists.llvm.org/pipermail/cfe-dev/2013-January/027234.html">http://lists.llvm.org/pipermail/cfe-dev/2013-January/027234.html</a></div>
<div>
<a href="http://lists.llvm.org/pipermail/cfe-dev/2016-January/046905.html">http://lists.llvm.org/pipermail/cfe-dev/2016-January/046905.html</a></div>
<h4>
<div style="font-weight: normal;">
Often, it is possible to come up with a workarounds, especially if one aims at implementing ad-hoc checks for their project. The simplest one would be creating a top-level "umbrella" C file which would include all files with implementations of the functions. I have seen some projects do exactly this. The most obvious shortcomings of this approach is that it will require redesigning the whole structure of the project, and will not work if some of the translation units need custom C compiler options.</div>
<div style="font-weight: normal;">
<br /></div>
<div style="font-weight: normal;">
Another option would be to dump/serialize the AST and any additional information during the compilation of each TU and process it after the whole project is built. It looks like this approach has been proposed multiple times on the mailing list, and there exist at least one paper which claims doing that.</div>
<div style="font-weight: normal;">
<br /></div>
<span style="font-weight: normal;">"Towards Vulnerability Discovery Using Extended
Compile-time Analysis" - <a href="http://arxiv.org/pdf/1508.04627.pdf">http://arxiv.org/pdf/1508.04627.pdf</a></span><br />
<br />
<span style="font-weight: normal;">In fact, the analyzer part itself might very well be independent of the actual parser, and could be reused, for example, to analyze the data obtained by the disassembly, but it's a topic for another research area.</span></h4>
<h4>
Developing ad-hoc analyzers for certain projects.</h4>
<div>
While it is very difficult to statically rule out many kinds of errors in an arbitrary project, either due to state explosion problem or the dynamic nature of the code, quite often we can design tools to verify some contracts specific to a certain project.</div>
<div>
<br /></div>
<div>
An example of such tool would be a tool called "sparse" from the Linux Kernel. It effectively works as a parser for the C code and can be made to run on every C file compiled by the GCC while building the kernel. It allows to specify some annotations to the declarations in the C code. It works similar to how attributes were implemented in GCC and Clang later.</div>
<div>
<a href="https://en.wikipedia.org/wiki/Sparse.">https://en.wikipedia.org/wiki/Sparse.</a></div>
<div>
<a href="http://linuxwireless.org/en/developers/Documentation/using-sparse/">http://linuxwireless.org/en/developers/Documentation/using-sparse/</a></div>
<div>
<br /></div>
<div>
One notable example of the code in the Linux Kernel, which deals with passing void pointers around and relying on pointer trickery via the "container_of" macro is the workqueue library.</div>
<div>
<br /></div>
<div>
While <a href="http://allsoftwaresucks.blogspot.ru/2014/09/gsoc-results-or-how-i-nearly-wasted.html">working on the FreeBSD kernel during the GSoC in 2014</a>, I faced a similar problem while developing device drivers - at certain places pointers were cast to void, and casted back to typed ones where needed. Certainly it's easy to make a mistake while coding these things.</div>
<div>
<br /></div>
<div>
Now, if we dump enough information during compilation, we can implement advanced checks. For example, when facing a function pointer which is initialized dynamically, we can do two things. First, find all places where it can potentially be initialized. Second, find all functions with a matching prototype. While checking all of them might be time consuming and generate false positives, it will also allow to check more code statically at compilation time.</div>
<div>
<br /></div>
<div>
A notable source of problems when working with C code is that linking stage is traditionally separated from the compilation stage. The linker usually manipulates the abstract "symbols" which are just void pointers. Even though it could be possible to store enough information about the types in a section of the ELF (in fact, DWARF debugging data contains information about the types) and use it to type-check symbols when linking, it's not usually done.</div>
<div>
<br /></div>
<div>
It leads to certain funky problems. For example, aliases (weak aliases) are a linker-time feature. If one defines an alias to some function, where the type of the alias does not match the type of the original function, the compiler cannot check it (well, it could if someone wrote a checker, but it does not), and you will get a silent memory corruption at runtime. I once ran into this issue when porting the RUMP library which ships a custom LIBC, and our C library had different size for "off_t".</div>
<h4>
Refactoring</h4>
<div>
There are two ideas I had floating around for quite a while.</div>
<div>
<br /></div>
<div>
Microkernelizing Linux.</div>
<div>
<br /></div>
<div>
An interesting research problem would be coming up with a tool to automatically convert/rewrite linux-kernel code into a microkernel with device drivers and other complex pieces of code staying in separate protection domains (processes).<br />
<br />
It is interesting for several reasons. One is that a lot of microkernels, such as L4, rely on DDE-Kits to run pieces of Linux and other systems (such as NetBSD in case of RUMP) to provide drivers. Another is that there's a lot of tested code with large functionality, which could possibly made more secure by minimizing the impact of memory corruption.<br />
<br />
Besides obvious performance concerns there are a lot of research questions to this.<br />
<br />
<ul>
<li>Converting access to global variables to IPC accesses. Most challenging part would be dealing with function pointers and callbacks.</li>
<li>Parsing KConfig files and "#ifdef" statements to ensure all conditional compilation cases are covered when refactoring. This in itself is a huge problem for every C codebase - if you change something in one branch of an "#ifdef" statement, you cannot guarantee you didn't break it for another branch. To get whole coverage, it would be useful to come up with some automated way to ensure all configurations of "#ifdef" are built.</li>
<li>Deciding which pieces of code should be linked statically and reside in the same process. For example, one might want to make sound subsystem, drm and vfs to run as separate processes, but going as far as having each TU converted to a process would be an overkill and a performance disaster.</li>
</ul>
<br />
Translating C code to another language. Not really sure if it could be really useful. It is likely that any complex code involving pointers and arrays would require a language with similar capabilities as a target (if we're speaking of generating a useful and readable idiomatic code, and not just a translator such as emscripten). Therefore, the target language might very well have the same areas with unspecified behavior. Some people have proposed for a stricter and a more well-defined dialect of C.<br />
<br />
One may note that it is not necessary to use clang for any of these tasks. In fact, one can get away with writing a custom parser or hacking GCC. These options are perfectly legal. I had some of these ideas floating around for at least three years, but back then I didn't have skills to build GCC and hack on it, and now I've been mostly using clang, but it's not a big deal.</div>
<h4>
Conclusion</h4>
<div>
Overall, neither improving the compiler nor switching to another language alone would not save us from errors. What's often overlooked is the process of the continuous integration, with regression tests to show what became broken, with gathering coverage, with doing integration tests.</div>
<div>
There are a lot of non-obvious parameters which are difficult to measure. For example, how to set up a regression test that would detect a software breakage due to a new compiler optimization.</div>
<div>
<br /></div>
<div>
Besides, we could possibly borrow a lot of practices from the HDL/Digital Design industry. Besides coverage, an interesting idea is simulating the same design in different languages with different semantics to hope that if there are mistakes at the implementation stage, they are not at the same places, and testing will show where outputs of two systems diverge.</div>
<h4>
P.S.</h4>
Ping me if you're interested in working on the topics mentioned above :)I hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com0tag:blogger.com,1999:blog-1416969656167659289.post-56264797547634221442016-01-25T04:42:00.000+03:002016-01-25T04:42:06.370+03:00On GPU ISAs and hackingRecently I've been reading several interesting posts on hacking Windows OpenGL drivers to exploit the "<span style="white-space: pre-wrap;">GL_ARB_get_program_binary" extension to access raw GPU instructions.</span><br />
<span style="white-space: pre-wrap;"><br /></span>
<br />
<ul>
<li><span style="white-space: pre-wrap;">"Hacking GCN via OpenGL" by Tomasz Stachowiak (<a href="https://twitter.com/h3r2tic">@h3r2tic</a>) - </span><a href="https://onedrive.live.com/view.aspx?resid=EBE7DEDA70D06DA0!107&app=PowerPoint&authkey=!AD-O3oq3Ung7pzk" style="white-space: pre-wrap;">Presentation on OneDrive</a></li>
<li><span style="white-space: pre-wrap;"><span style="font-family: inherit;">"</span></span><span style="background-color: white; font-family: inherit; line-height: 1.2;">You Compiled This, Driver. Trust Me…." about hacking Intel Haswell GPU - </span><span style="white-space: pre-wrap;"><span style="font-family: inherit;"><a href="http://www.joshbarczak.com/blog/?p=1028">The blog</a> by </span></span><span style="background: rgb(245, 248, 250); color: black; font-family: inherit; line-height: 19.25px; text-decoration: none;"><a class="account-group js-account-group js-action-profile js-user-profile-link js-nav" data-user-id="2435632202" href="https://twitter.com/JoshuaBarczak" style="background: rgb(245, 248, 250); line-height: 19.25px; text-decoration: none;"><span class="fullname js-action-profile-name show-popup-with-id" data-aria-label-part="">Joshua Barczak</span> <span class="username js-action-profile-name" data-aria-label-part="" style="direction: ltr; unicode-bidi: embed;">@JoshuaBarczak</span> </a></span></li>
</ul>
<br />
<span style="white-space: pre-wrap;">Reverse engineering is always cool and interesting. However, a lot of these efforts have been previously done not once because people directed their efforts at making open-source drivers for linux, and I believe these projects are a good way to jump-start into GPU architecture without reading huge vendor datasheets.</span><br />
<span style="white-space: pre-wrap;"><br /></span>
<span style="white-space: pre-wrap;">Since I'm interested in the GPU drivers, I decided to put up links to several open-source projects that could be interesting to fellow hackers and serve as a reference, potentially more useful than vendor documentation (which is not available for some GPUs. you know, the green ones).</span><br />
<span style="white-space: pre-wrap;"><br /></span>
<span style="white-space: pre-wrap;">Please also take a look at this interesting blog post for the detailed list of GPU datasheets - <a href="http://renderingpipeline.com/graphics-literature/low-level-gpu-documentation/">http://renderingpipeline.com/graphics-literature/low-level-gpu-documentation/</a></span><br />
<br />
* <span style="background-color: yellow;">AMD R600 GPU Instruction Set Manual</span>. It is an uber-useful artifact because many modern GPUs (older AMD desktop GPUs, Qualcomm Adreno GPUs and Xbox 360 GPU) bear a lot of similarities with AMD R300/R600 GPU series.<br />
<a href="http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf">http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf</a><br />
<br />
* <span style="background-color: yellow;">Xenia Xbox 360 Emulator</span>. It features a Dynamic Binary Translation module that translates the AMD instruction stream into GLSL shaders at runtime. Xbox 360 has an AMD Radeon GPU, similar to the one in R600 series and Adreno<br />
https://github.com/benvanik/xenia<br />
<br />
Let's look at "src/xenia/gpu". Warning: the code at this project seems to be changing very often and the links may quickly become stale, so take a look yourself.<br />
One interesting file is "register_table.inc" which has register definitions for MMIO space allowing you to figure out a lot about how the GPU is configured.<br />
The most interesting file is "<a href="https://github.com/benvanik/xenia/blob/master/src/xenia/gpu/ucode.h%22">src/xenia/gpu/ucode.h</a> which contains the structures to describe ISA.<br />
<a href="https://github.com/benvanik/xenia/blob/master/src/xenia/gpu/packet_disassembler.cc">src/xenia/gpu/packet_disassembler.cc</a> contains the logic to disassemble the actual GPU packet stream at binary level<br />
<a href="https://github.com/benvanik/xenia/blob/master/src/xenia/gpu/shader_translator.cc">src/xenia/gpu/shader_translator.cc</a> contains the logic to translate the GPU bytecode to the IR<br />
<a href="https://github.com/benvanik/xenia/blob/master/src/xenia/gpu/glsl_shader_translator.cc">src/xenia/gpu/glsl_shader_translator.cc</a> the code to translate Xenia IR to GLSL<br />
Part of the Xbox->GLSL logic is also contained in "<a href="https://github.com/benvanik/xenia/blob/master/src/xenia/gpu/gl4/gl4_command_processor.cc">src/xenia/gpu/gl4/gl4_command_processor.cc</a>" file.<br />
<br />
* <span style="background-color: yellow;">FreeDreno - the Open-Source Driver for the Qualcomm/ATI Adreno GPUs</span>. These GPUs are based on the same architecture that the ATI R300 and consequently AMD R600 series, and both ISAs and register spaces are quite similar. This driver was done by Rob Clark, who is a GPU driver engineer with a lot of experience who (unlike most FOSS GPU hackers) previously worked on another GPU at a vendor and thus had enough experience to successfully reverse-engineer a GPU.<br />
<br />
<a href="https://github.com/freedreno/mesa">https://github.com/freedreno/mesa</a><br />
<br />
Take a look at "<a href="https://github.com/freedreno/mesa/blob/master/src/gallium/drivers/freedreno/a3xx/fd3_program.c#L87">src/gallium/drivers/freedreno/a3xx/fd3_program.c</a>" to have an idea how the GPU command stream packet is formed on the A3XX series GPU which is used in most Android smartphones today. Well, it's the same for most GPUs - there is a ring-buffer to which the CPU submits the commands, and from where the GPU reads them out, but the actual format of the buffer is vendor and model specific.<br />
<br />
I really love this work and this driver because it was the first open-source driver for mobile ARM SoCs that provided full support for OpenGL ES and non-ES enough to run Gnome Shell and other window managers with GPU accelleration and also supports most Qualcomm GPUs.<br />
<br />
* <span style="background-color: yellow;">Mesa. Mesa is the open-source stack for several compute and graphic APIs for Linux: OpenGL, OpenCL, OpenVG, VAAPI and OpenMAX</span>.<br />
<br />Contrary to the popular belief, <span style="background-color: lime;">it is not a "slow" and "software-rendering" thing</span>. Mesa has several backends. "Softpipe" is indeed a slow and CPU-side implementation of OpenGL, "llvmpipe" is a faster one using LLVM for vectorizing. But the most interesting part are the drivers for the actual hardware. Currently, Mesa supports most popular GPUS (with the notable exception of ARM Mali). You can peek into Mesa source code to get insight into how to control both the 2D/ROP engines of the GPUs and how to upload the actual instructions (the ISA bytecode) to the GPU. Example links are below. It's a holy grail for GPU hacking.<br />
<br />
"<a href="http://cgit.freedesktop.org/mesa/mesa/tree/src/mesa/drivers/dri/i915/i915_program.c">http://cgit.freedesktop.org/mesa/mesa/tree/src/mesa/drivers/dri/i915/i915_program.c</a>" contains the code to emit I915 instructions.<br />
The code for the latest GPUs - Broadwell - is in the "i965" driver.<br />
<br />
<ul>
<li>The code to set up the compute unit - the number of threads/groups - <a href="http://cgit.freedesktop.org/mesa/mesa/tree/src/mesa/drivers/dri/i965/brw_compute.c#L76">http://cgit.freedesktop.org/mesa/mesa/tree/src/mesa/drivers/dri/i965/brw_compute.c#L76</a></li>
<li>The dreaded CURBE described in the Joshua's post - <a href="http://cgit.freedesktop.org/mesa/mesa/tree/src/mesa/drivers/dri/i965/brw_curbe.c#L295">http://cgit.freedesktop.org/mesa/mesa/tree/src/mesa/drivers/dri/i965/brw_curbe.c#L295</a></li>
<li>The code to encode the instructions for the Broadwell EU (Execution Unit) - <a href="http://cgit.freedesktop.org/mesa/mesa/tree/src/mesa/drivers/dri/i965/brw_eu_emit.c">http://cgit.freedesktop.org/mesa/mesa/tree/src/mesa/drivers/dri/i965/brw_eu_emit.c</a></li>
<li>Broadwell Instruction Disassembler - <a href="http://cgit.freedesktop.org/mesa/mesa/tree/src/mesa/drivers/dri/i965/brw_disasm.c">http://cgit.freedesktop.org/mesa/mesa/tree/src/mesa/drivers/dri/i965/brw_disasm.c</a></li>
</ul>
<br />
<br />
<br />
<div>
While browsing Mesa code, one will notice the mysterious "bo" name. It stands for "Buffer Object" and is an abstraction for the GPU memory region. It is handled by the userpace library called "libdrm" - <a href="http://cgit.freedesktop.org/mesa/drm/tree/">http://cgit.freedesktop.org/mesa/drm/tree/</a></div>
<br />
* <span style="background-color: yellow;">Linux Kernel "DRM/KMS" drivers</span>. KMS/DRM is a facility of Linux Kernel to handle GPU buffer management and basic GPU initialization and power management in kernel space. The interesting thing about these drivers is that you can peek at the places where the instructions are actually transferred to the GPU - the registers containing physical addresses of the code.<br />
There are other artifacts. For example, I like this code in the Intel GPU driver (i915) which does runtime patching of GPU instruction stream to relax security requirements.<br />
<br />
<a href="https://github.com/torvalds/linux/blob/master/drivers/gpu/drm/i915/i915_cmd_parser.c">https://github.com/torvalds/linux/blob/master/drivers/gpu/drm/i915/i915_cmd_parser.c</a><br />
<br />
* <span style="background-color: yellow;">Beignet - an OpenCL implementation for Intel Ivy-Bridge and later GPUs</span>. Well, it actually works.<br />
<a href="http://cgit.freedesktop.org/beignet/">http://cgit.freedesktop.org/beignet/</a><br />
The interesting code to emit the actual command stream is at <a href="http://cgit.freedesktop.org/beignet/tree/src/intel/intel_gpgpu.c">http://cgit.freedesktop.org/beignet/tree/src/intel/intel_gpgpu.c</a><br />
<br />
* LibVA Intel H.264 Encoder driver. Note that the proprietary MFX SDK also uses the LibVA internally, albeit a patched version with newer code which typically gets released and merged into upstream LibVA after it goes through Intel internal testing.<br />
<a href="https://www.blogger.com/goog_1959131072"><br /></a>
<a href="http://cgit.freedesktop.org/vaapi/intel-driver/tree/">http://cgit.freedesktop.org/vaapi/intel-driver/tree/</a><br />
An interesting artifact are the sources with the encoder pseudo-assembly.<br />
<br />
<a href="http://cgit.freedesktop.org/vaapi/intel-driver/tree/src/shaders/h264/mc/interpolate_Y_8x8.asm">http://cgit.freedesktop.org/vaapi/intel-driver/tree/src/shaders/h264/mc/interpolate_Y_8x8.asm</a><br />
<a href="http://cgit.freedesktop.org/vaapi/intel-driver/tree/src/shaders/h264/mc/AVCMCInter.asm">http://cgit.freedesktop.org/vaapi/intel-driver/tree/src/shaders/h264/mc/AVCMCInter.asm</a><br />
<br />
Btw, If you want zero-copy texture sharing to pass OpenGL rendering directly to the H.264 encoder with either OpenGL ES or non-ES, please refer to my earlier post - <a href="http://allsoftwaresucks.blogspot.ru/2014/10/abusing-mesa-by-hooking-elfs-and-ioctl.html">http://allsoftwaresucks.blogspot.ru/2014/10/abusing-mesa-by-hooking-elfs-and-ioctl.html</a> and to <a href="http://cgit.freedesktop.org/wayland/weston/tree/src/vaapi-recorder.c">Weston VAAPI recorder</a>.<br />
<br />
* <span style="background-color: yellow;">Intel KVM-GT and Xen-GT. These are GPU virtualization solutions from Intel</span>. They emulate the PCI GPU in the VM. They expose the MMIO register space fully compatible with the real GPU so that the guest OS uses the vanilla or barely modified driver. The host contains the code to reinterpret the GPU instruction stream and also to set up the host GPU memory space in such a way that it's partitioned equally between the clients (guest VMs).<br />
<br />
There are useful papers, and interesting pieces of code.<br />
<br />
<ul>
<li><a href="http://www.linux-kvm.org/images/f/f3/01x08b-KVMGT-a.pdf">http://www.linux-kvm.org/images/f/f3/01x08b-KVMGT-a.pdf</a></li>
<li><a href="https://github.com/01org/KVMGT-qemu/blob/master/hw/display/vga-vgt.c">https://github.com/01org/KVMGT-qemu/blob/master/hw/display/vga-vgt.c</a> - the QEMU PCI device code.</li>
<li>Linux Kernel Patch - <a href="https://github.com/01org/KVMGT-kernel/commit/a17514b9696754f21d37c112139064a91a34a73e">https://github.com/01org/KVMGT-kernel/commit/a17514b9696754f21d37c112139064a91a34a73e</a></li>
<li>The LKML message with the detailed announcement - <a href="https://lwn.net/Articles/624516/">https://lwn.net/Articles/624516/</a></li>
</ul>
<div>
* <span style="background-color: yellow;">Intel GPU Tools</span></div>
<div>
A lot of useful tools. Most useful one IMHO is the "intel_perf_counters" which shows compute unit, shader and video encoder utilization. There are other interesting tools like register dumper.</div>
<div>
<br /></div>
<div>
<a href="http://cgit.freedesktop.org/xorg/app/intel-gpu-tools/tree/tools">http://cgit.freedesktop.org/xorg/app/intel-gpu-tools/tree/tools</a></div>
<div>
<br /></div>
<div>
It also has the Broadwell GPU assembler</div>
<div>
<a href="http://cgit.freedesktop.org/xorg/app/intel-gpu-tools/tree/assembler">http://cgit.freedesktop.org/xorg/app/intel-gpu-tools/tree/assembler</a></div>
<div>
<br /></div>
<div>
And the ugly Shaders in ugly assembly similar to those in LibVA</div>
<div>
<a href="http://cgit.freedesktop.org/xorg/app/intel-gpu-tools/tree/shaders/gpgpu/gpgpu_fill.gxa">http://cgit.freedesktop.org/xorg/app/intel-gpu-tools/tree/shaders/gpgpu/gpgpu_fill.gxa</a></div>
<div>
<br /></div>
<div>
I once wrote the tool complimentary to the register dumper - to upload the register dump back to the GPU registers. I used it to debug some issues with the eDP panel initialization in my laptop</div>
<div>
<br /></div>
<div>
<a href="https://github.com/astarasikov/intel-gpu-dump-upload">https://github.com/astarasikov/intel-gpu-dump-upload</a></div>
<div>
<br /></div>
* <span style="background-color: yellow;">LunarGLASS is a collection of experimental GPU middleware</span>. As far as I understand it, it is an ongoing initiative to build prototype implementations of upcoming Khronos standards before trying to get them merged to mesa. And also to allow the code to be used with any proprietary GPU SW stack, not tied to Mesa. Apparently it also has the "GLSL" backend which should allow you to try out SPIR-V on most GPU stacks.<br />
<br />
The main site - <a href="http://lunarg.com/lunar-glass/">http://lunarg.com/lunar-glass/</a><br />
The SPIR-V frontend source code - <a href="https://github.com/LunarG/LunarGLASS/tree/master/Frontends/SPIRV">https://github.com/LunarG/LunarGLASS/tree/master/Frontends/SPIRV</a><br />
The code to convert LunarG IR to Mesa IR - <a href="https://github.com/LunarG/LunarGLASS/blob/master/mesa/src/mesa/program/ir_to_mesa.cpp">https://github.com/LunarG/LunarGLASS/blob/master/mesa/src/mesa/program/ir_to_mesa.cpp</a><br />
<br />
* <span style="background-color: yellow;">Lima driver project</span>. It was a project for reverse-engineering the ARM Mali driver. The authors (<a href="http://libv.livejournal.com/">Luc Verhaegen aka "libv"</a> and Connor Abbot) have decoded some part of the initialization routine and ISA. Unfortunately, the project seems to have stopped due to legal issues. However, it has significant impact on the open-source GPU drivers, because Connor Abbot went on to implement some of the ideas (namely, the NIR - "new IR") in the Mesa driver stack.<br />
<br />
Some interesting artifacts:<br />
The place where the Geometry Processor data (attributes and uniforms) are written to the GPU physical memory:<br />
<a href="https://github.com/limadriver/lima/blob/master/limare/lib/gp.c#L333">https://github.com/limadriver/lima/blob/master/limare/lib/gp.c#L333</a><br />
"Render State" - a small structure describing the GPU memory, including the shader program address.<br />
<a href="https://github.com/limadriver/lima/blob/master/limare/lib/render_state.h">https://github.com/limadriver/lima/blob/master/limare/lib/render_state.h</a><br />
<br />
* <span style="background-color: yellow;">Etnaviv driver project</span>. It was a project to reverse-engineer the Vivante GPU used by Freescale I.MX SoC series. It was done by <a href="https://github.com/laanwj">Wladimir J. van der Laan</a>. The driver has reached the mature state, successfully running games like Quake. Now, there is a version of Mesa with this driver built-in.<br />
<br />
<a href="https://github.com/etnaviv/etna_viv">https://github.com/etnaviv/etna_viv</a><br />
<a href="https://github.com/laanwj/etna_viv">https://github.com/laanwj/etna_viv</a><br />
<a href="https://github.com/etnaviv/mesa">https://github.com/etnaviv/mesa</a><br />
<a href="http://wiki.wandboard.org/index.php/Etna_viv">http://wiki.wandboard.org/index.php/Etna_viv</a><br />
<br />I hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com0tag:blogger.com,1999:blog-1416969656167659289.post-65614307448030281652016-01-18T22:16:00.000+03:002016-01-18T22:30:14.503+03:00Abusing QEMU and mmap() or Poor Man's PCI Pass-Through<h4>
The problem.</h4>
Another tale from the endless firmware endeavours.<br />
The other day I ended up in a weird situation. I was locked in a room with three firmwares - one of them was built by myself from source, while the other two came in binary form from the vendor. Naturally the one I built was not fully working, and one of the binaries worked fine. So I set out on a quest to trace all PCI MMIO accesses.<br />
<br />
When one thinks about it, several solutions come to mind. The first idea is to run a QEMU/XEN VM and pass-through the PCI device. Unfortunately the board we had did not have IOMMU (or VT-D in Intel terminology). The board had the Intel BayTrail SoC which is basically a CPU for phones.<br />
Another option would be to binary-patch the prebuilt firmware and introduce a set of wrapper functions around MMIO access routines and add corresponding calls to printf() for debugging but this approach is very tedious and error-prone, and I try to avoid patching binaries because I'm extremely lazy.<br />
<br />
So I figured the peripherals of interest (namely, the I2C controller and a NIC card) were simple devices configured by simple register writes. The host did not pass any shared memory to the device, so I used a simple technique. I patched QEMU and introduced "fake" PCI devices with VID and PID of the devices I wanted to "pass-through". By default the MMIO read access return zero (or some other default value) while writes are ignored. Then I went on and added the real device access. Since I was running QEMU on Linux, I just blacklisted and rmmod'ed the corresponding drivers. Then, I used "<span style="background-color: yellow;">lspci</span>" to find the addresses of the PCI BARs and <span style="background-color: yellow;">mmap() </span>syscall on <span style="background-color: yellow;">"/dev/mem"</span> to access the device memory.<br />
<br />
<h4>
A picture is worth a thousand words</h4>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgrZdDk6WyA6dNHGG9VgqAs_eYCf4F68cFAuomQSMMNtYc3hV75oavq9uCaQANSqonacXB8ZX5aXwsphlItwKa4n_QeSO9Pisb0I-i3NwUSxJtvpOZFz1SdWOWH6neqDuFBFBesUgWiA18/s1600/DSC_0694.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="222" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgrZdDk6WyA6dNHGG9VgqAs_eYCf4F68cFAuomQSMMNtYc3hV75oavq9uCaQANSqonacXB8ZX5aXwsphlItwKa4n_QeSO9Pisb0I-i3NwUSxJtvpOZFz1SdWOWH6neqDuFBFBesUgWiA18/s400/DSC_0694.JPG" width="400" /></a></div>
<br />
<h4>
The code.</h4>
The code is quite funky, but thanks to the combination of many factors (X86 being a strongly ordered architecture and the code using only 4-byte transfers) the naive implementation worked fine.<br />
1. I can now trace all register reads and writes<br />
2. I have verified that the "good" firmware still boots and everything is working fine.<br />
So, after that I simply ran all three firmwares and compared the register traces only to find out that the problems might be unrelated to driver code (which is a good result because it allowed to narrow down the amount of code to explore).<br />
<br />
The branch with my customized QEMU is here. Also, I figured QEMU now has the ultra-useful option called "<span style="background-color: yellow;">readconfig</span>" which allows you to supply the board configuration in a text file and avoid messing with cryptic command line switches for enabling PCI devices or HDDs.<br />
<a href="https://github.com/astarasikov/qemu/tree/our_board">https://github.com/astarasikov/qemu/tree/our_board</a><br />
<a href="https://github.com/astarasikov/qemu/commit/2e60cad67c22dca750852346916d6da3eb6674e7">https://github.com/astarasikov/qemu/commit/2e60cad67c22dca750852346916d6da3eb6674e7</a><br />
<br />
The code for mmapping the PCI memory is here. It is a marvel of YOLO-based design, but hey, X11 does the same, so it is now industry standard and unix-way :)<br />
<a href="https://github.com/astarasikov/qemu/commit/2e60cad67c22dca750852346916d6da3eb6674e7#diff-febf7a335ad7cd658784899e875b5cc7R31">https://github.com/astarasikov/qemu/commit/2e60cad67c22dca750852346916d6da3eb6674e7#diff-febf7a335ad7cd658784899e875b5cc7R31</a><br />
<br />
<h4>
Limitations.</h4>
There is one obvious limitation, which is also the reason QEMU generally does not support passthrough of MMIO devices on systems without IOMMU (Like VT-D or SMMU).<br />
<br />
Imagine a case where the device is not only configured via simple register transfers (such as setting certain bits to enable IRQ masking or clock gating), but system memory is passed to the device via storing a pointer in a register.<br />
<br />
The problem is that the address coming from the guest VM (GPA, guest physical address) will not usually match the corresponding HPA (host physical address). The device however operates the bus addresses and is not aware of the MMU, let alone VMs. IOMMU solves this problem by translating the addresses coming from the device into virtual addresses, and the hypervisor can supply a custom page table to translate bus HPAs into GPAs.<br />
<br />
So if one would want to provide a pass-through solution for sharing memory in systems without IOMMU, one would either need to ensure that the VM and guest have 1:1 GPA/HPA mappings for the addresses used by the device or come up with an elaborate scheme which would detect register writes containing memory buffers and set up a corresponding buffer in the host memory space, but then one will need to deal with keeping the host and guest buffers consistent. Besides, that would require the knowledge of the device registers, so it would be non-trivial to build a completely generic, plug-and-play solution.I hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com0tag:blogger.com,1999:blog-1416969656167659289.post-18523445250069432542015-11-08T06:32:00.001+03:002015-11-10T21:19:45.483+03:00Porting legacy RTOS software to a custom OS.So lately I've been working on porting a large-sized application from a well-known RTOS to a custom kernel. The RTOS is VxWorks and the application is a router firmware. Unfortunately I will not be able to share any code of the porting layer. I'll just briefly go over the problems I've endeavoured and some thoughts I have come up with in the process.<br />
<br />
The VxWorks compiler (called "diab") is like many other proprietary C compilers based on the Edison frontend. Unlike GCC or clang, by default it is quite lax when it comes to following the standards. Besides, the original firmware project uses the VxWorks Workbench IDE (which is of course a fork of Eclipse) and runs on Windows.<br />
<br />
The first thing I had to do was to convert the project build system to Linux. The desirable way would be to write a parser that would go over the original build system and produce a Makefile. The advantage of this approach would be the ability to instantly obtain a new Makefile when someone changes the original project code. However, during the prototyping stage it makes a lot of sense to take shortcuts wherever possible, so I ended up writing a tiny wrapper program to replace the compiler and linker binaries. It would intercept its arguments, write them out to a log file and act as a proxy launching the original program. After that it was just a matter of making minor changes to the log file to convert it to a bash script and later on to a Makefile.<br />
<br />
As an effect of the development done on Windows, the majority of files have the same set of defects that prevent them from being compiled by a linux-based compiler: paths in the "#include" directive often have incorrect case, reverse slashes and so on. Worst is that some of the "#include" directives included multiple files in one set of brackets. Luckily, this was easy to fix with a script that parsed the compilation log for the "file not found" errors, looked for the corresponding file ignoring the case and fixed up the source code. In the end I was left with about a dozen places that had to be fixed manually.<br />
<h3>
Implementing the compatibility layer.</h3>
I have done a quick research into the available options and saw that the only up-to-date solution implementing the VxWorks API on other OSs is "Xenomai". However, it is quite intrusive because it relies on a loadable kernel module and some patches to the linux kernel to function. Since we were not interested in getting realtime behaviour but wanted to run on both our OS and Linux and entirely in userspace, I decided to write yet another VxWorks emulation layer.<br />
The original firmware comes as a single ELF file which is reasonable because in VxWorks all processes are implemented as threads in a shared address space. Besides, VxWorks provides a POSIX-compatible API for developers. So in order to identify which functions needed implementation it was enough to try linking the compiled code into a single executable.<br />
<br />
"One weird trick" useful for creating porting layers and DDEKits is the GCC/clang option "include" which allows you to prepend an include to absolutely all files compiled. This is super useful. You can use such an uber-header to place the definitions of the data types and function prototypes for the target platform. Besides, you can use it to hook library calls at compile-time.<br />
<br />
One of the problem that took a great amount of time was implementing synchronization primitives. Namely, mutexes and semaphores. The tricky semantic difference between semaphores and mutexes in VxWorks is that the latter are recursive. That means that once a thread has acquired a mutex, it is allowed to lock it any number of times as long as the lock/unlock count is balanced.<br />
Before I realized this semantic difference, I couldn't figure out why the software would always lock up, and disabling locking altogether led to totally random crashes.<br />
<br />
Ultimately I became frustrated and ended up with a simple implementation of a recursive mutex that has allowed me to move much further (<a href="https://gist.github.com/astarasikov/fd75fa7e706cb6337e60">Simple Recursive Mutex Implementation @ github</a>). Later for the purposes of debugging I also added the options to print backtrace indicating the previous lock owner when trying to enter the critical section or when the spinlock took too many attempts.<br />
<h3>
Hunting for code defects</h3>
<h4>
Uninitialized variables and what can we do about them</h4>
<div>
One problem I came across was that the code had a lot of uninitialized variables, hundreds of them. On the one hand, the proper solution is to inspect each case manually and write the correct initializer. On the other hand, the code works when compiled with the diab compiler so it must zero-initialize them.</div>
<div>
So I went ahead and wrote a clang rewriter plugin to add the initializers: zero for the primitive types and a pair of curly braces for structs. (<a href="https://github.com/astarasikov/my-llvm-plugins/tree/master/clang_add_initializers">clang rewriter to add initializers</a>). However, I realized that the biggest problem is that some functions use the convention of returning zero for indicating a failure while other return a non-zero value. This means, we cannot have a generic safe initializer that would make the code take the "fault" path when reaching the rewritten code. An alternative to manual inspection could be writing sophisticated rules for the rewriter to detect the convention used.</div>
<div>
<br /></div>
<div>
I ended up using valgrind and manually patching some warnings. AddressSanitizer was also useful. However fixing each warning and creating a blacklist is too tiresome. I ended up setting the breakpoint on the "__asan_report_error" function and a script that would make gdb print backtrace, return and continue execution.</div>
<h4>
Duplicate structures</h4>
<div>
One problem I supposed could be present in the code (due to the deep hirarchy of #ifdefs) is the presence of the structures with the same name but different content. I made up an example of a C program to demonstrate an effect where the compiler does not warn about the type mimatch but at runtime the code silently corrupts memory.</div>
<div>
<br /></div>
<div>
<a href="https://gist.github.com/astarasikov/d57f6b942da9a304ff05">An example of duplicate struct definition</a></div>
<div>
<br /></div>
<div>
I figured out an easy way of dealing with the problem. I ended up using clang and emitting LLVM bittecode for each file instead of the object files with machine code. Then I linked them together into a single bitcode file and disassembled with llvm-dis.</div>
<div>
<br /></div>
<div>
The nice thing about llvm is that when it finds two structures having the same name but declared differently, it would append a different numeric suffix to the struct name. Then one could just remove the suffixes and look for unique lines with different structure declarations.</div>
<div>
<br /></div>
<div>
Luckily for me, there was only one place where I supposed an incorrect definition, and it was not in the part of the code I was executing, so I ruled out this option as a source of incorrect behavior.</div>
<h3>
Further work</h3>
<h4>
Improving tooling for the 32-bit world</h4>
It is quite unfortunate but the project I have been working on is 32-bit. And one cannot simply convert it into a 64-bit one by a compiler flag. The problem is that the code has quite a lot of places where pointers are for some reason stored into packed structures and some other structures implicitly rely on the structure layout. So it is very difficult to modify this fragile mess.<br />
It is sad that two great tools, MemorySanitizer and ThreadSanitizer, are not available for the 32-bit applications. It is understandable because the 32-bit address space is too tiny to fit the shadow memory. I am thinking of ways to make them available for the 32-bit world. So far I see two ways of solving the problem.<br />
First, we can use the fragile and non-portable flag for the mmap (which is currently only supported on linux) to force allocations to below the 4gig limit. Then one could write an ELF loader that would load all the code below 4gigs, and use the upper memory range for shadows. Besides being non-portable, the other disadvantages of this approach include having to deal with 64-bit identifiers such as file handles.<br />
Alternatively we could store the shadow in a separate process and use shared memory or sockets for communication. That would likely be at least an order slower than using the corresponding sanitizer in a 64-bit world, but likely still faster than valgrind and besides it is a compile-time instrumentation with more internal data from the compiler.<br />
<h4>
Verifying the porting was done correctly</h4>
Now I am left with one challenging task: verify the ported SW is identical to what could be built using the original build system.<br />
<br />
One may notice that simply intercepting the calls to the compiler may not be enough because the build system may copy the files or export some shell variables during the build process. Besides, different compilers have different ways of handling "inline" and some other directives. It would be good to verify that the call graph of the original binary and the one produced by our Makefile is similar. (of course we will need to mark some library functions as leaf nodes and not analyze them). For a start I could try inspecting some of the unresolved symbols manually, but I'm thinking of automating the process. I think for this task I'll need a decompiler that can identify basic blocks. Probably Capstone engine should do the job. Any ideas on that?<br />
<br />
P.S. Oh, and I once tried visualizing the dependency graph of separate ".o" files (before I realized I could just link them altogether and get the list of missing symbols) and trust me those graphs grow really fast. I have found out that <a href="http://gephi.github.io/">a tool called "gephi"</a> does a decent job at visualizing really huge graphs and supports Graphviz's dot as the input format.<br />
<br />
EDIT 2015-11-10<br />
The challenge is complicated by the fact that there some subprojects have multiple copies (with various changes) and one should also ensure that the headers and sources are picked up from the correct copy. However, I've found an easy and acceptable solution. I just wrote a tool that parses the call graph generated by IDA and for every edge in the graph it looks up the function names of the corresponding vertices. Then it just prints a list of pairs "A -> B" for every function A calling function B. After that, we can sort the file alphabetically and remove the nodes that are uninteresting to us (the OS and library functions). Next, we can compare the files side-by-side with a tool like kdiff3 (or we can do it automatically). Whenever there is a significant difference (for example, 5 callees are different for the same caller), we can inspect manually and verify we're compiling the correct file with the correct options. Using this method I have identified several places where we chose the wrong object file for linking and now we're only concerned with the porting layer and OS kernel, without having to worry about the application itself.I hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com0tag:blogger.com,1999:blog-1416969656167659289.post-38283972283318307832015-05-20T02:24:00.001+03:002015-05-20T02:24:08.801+03:00on making a DDE kit, kindaSo I have this task of porting a huge piece of software running on a proprietary OS to another OS.
And I don't even have a clue how to compile it (well I do but it builds on windows so it's almost irrelevant).<br />
<br />
But luckily all code is linked into a single ELF file and the compilation produces intermediate object files.
The first thought I had was to visualize the dependency graph of object files to find out who calls what. You can find the script below that will recursively walk the supplied directory and try to parse the import/export table with objdump. There are some areas for improvement (for example, parsing also the dynsym table with -T or parsing .a archives) but it did the work for me.<br />
<br />
Unfortunately I realized that visualizing the graph with 30K edges was not even remotely a smart idea<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgb53Ue1CBbwLpOvJE5mJxM8xBn6RSII3DrHaR8Zl0LN0mFVYA2X-xy2PJbkE56zb-ME3NQYH0V9FFE3-9WJ67yVOb6XFXy2K8Ngf8pfmIdKKMO_v3UIG2lp-D6tCpu6atZTonMH7LL1VU/s1600/star.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgb53Ue1CBbwLpOvJE5mJxM8xBn6RSII3DrHaR8Zl0LN0mFVYA2X-xy2PJbkE56zb-ME3NQYH0V9FFE3-9WJ67yVOb6XFXy2K8Ngf8pfmIdKKMO_v3UIG2lp-D6tCpu6atZTonMH7LL1VU/s400/star.png" /></a></div>
https://gist.github.com/astarasikov/355bf825f130fe4b5633<br />
<br />
What I've also found out was that the OS-specific code and objects was stored in a separate location (since it was a part of an SDK). Even if it were not, we could just remove those object files that were both present in our project and in the SDK. After that, all the functions that the application was requiring from the OS unsurprisingly ended up in the "UNDEFINED" node and there were only 200 of them which gives me some hope.<br />
<br />
This approach can also be used for other use-cases. For example, porting drivers from Linux/FreeBSD to exotic platforms - first build the binaries, then pick as many of them as you can to minimize the required functions list. I find dealing with compiled code easier because build systems, C macros and ifdefs just drive me insane.I hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com0tag:blogger.com,1999:blog-1416969656167659289.post-44513926889935095802015-05-15T08:27:00.003+03:002015-05-15T08:27:57.537+03:00GCOV is amazing yet undocumentedOne useful technique for maintaining software quality is code coverage. While routinely used by high-level developers it is completely forgotten by many C hackers, especially when it comes to kernel. In fact, Linux is the only kernel which supports being compiled with the GCOV coverage tool.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDjFA-Hw00lpStrlaHdqqAcgHJMaMtzb7hhVzOlM95gB_MASzCVUJoUFF5f72fCwvaMuMgmEYXQ2D-Wna8OuRGmB6xp_JqbEb8CpQ-Zk0CxaJeqxfX5MSTWheYBi7D8sC2RtU6-lXW7Yo/s1600/kcov.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="355" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDjFA-Hw00lpStrlaHdqqAcgHJMaMtzb7hhVzOlM95gB_MASzCVUJoUFF5f72fCwvaMuMgmEYXQ2D-Wna8OuRGmB6xp_JqbEb8CpQ-Zk0CxaJeqxfX5MSTWheYBi7D8sC2RtU6-lXW7Yo/s1600/kcov.png" width="400" /></a></div>
<br />
<br />
GCOV works by instrumenting your code. It inserts some code to increment the stats counters around each basic block. These counters reside in a special section of your ELFs. During compilation, GCC generates a ".gcno" file for each ".c" file. These files allow the "gcov" tool to lookup function names and other info using the integer IDs (which are specific to each file).<br />
<br />
At runtime, an executable built with GCOV produces a file called ".gcda" which contains the values of the counters. During the executable initialization, constructors (which are function pointers in the ".ctors" section) are called. One of them is "__gcov_init" which registers a certain ".o" file inside libgcov.<br />
<br />
Since we're running in kernel or "bare-metal", we don't have neither libgcov nor file system to dump the ".gcda" files. But one should not fear GCOV! Adding it to your kernel is just a matter of adding one C file (which is mostly shamelessly copy-pasted from linux kernel and gcc sources) and a couple CFLAGS. In my example I'm using the LK kernel by Travis Geiselbrecht (https://github.com/travisg/lk). I've decided to just print out the contents of the ".gcda" files to the serial console (UART) as hex dump and then use an AWK script and the "xxd" tool to convert them to binaries on the host. This works completely fine since these files are typically below 2KB in size.<br />
<br />
An important thing to notice: if your kernel does not contain the ".ctors" section and does not call the constructors, be sure to add them to the ld script and add some code to invoke them. For example, here's how LK does that: https://github.com/travisg/lk/blob/master/top/main.c#L42<br />
<br />
You can see the whole patch below.<br />
https://github.com/astarasikov/lk/commit/2a4af09a894194dfaff3e05f6fd505241d54d074<br />
<br />
After running the "gcovr" tool you can get a nice HTML with summary and see which lines were executed and which were not and add the tests for the latter. Woot!<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlpnjS0Wo6a_jHXqxL0eWztAhvJRjCIKeZUj9Q3si3kUMUtMurE-bn0aYk_ZTVduBWSbk1AdYZGX3arfO7ijb9XzgW5n-9MSZKExiO7Hc42_zgdgEaPDSXOlQkCv5LK-gPgLjvs4PbGYk/s1600/Screen+Shot+2015-05-15+at+08.06.06.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlpnjS0Wo6a_jHXqxL0eWztAhvJRjCIKeZUj9Q3si3kUMUtMurE-bn0aYk_ZTVduBWSbk1AdYZGX3arfO7ijb9XzgW5n-9MSZKExiO7Hc42_zgdgEaPDSXOlQkCv5LK-gPgLjvs4PbGYk/s1600/Screen+Shot+2015-05-15+at+08.06.06.png" width="576" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />I hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com0tag:blogger.com,1999:blog-1416969656167659289.post-78045601397381462102014-10-24T23:16:00.001+04:002014-10-24T23:16:41.998+04:00abusing Mesa by hooking ELFs and ioctlAt work I had several occasions when I needed to hook a function on a Linux system. I will not go deep into technical details but the examples provided in the links contain the code which can be easily plugged into a project.<br />
<br />
<b><i>A simple case.</i></b><br />
First let us consider a case when we only need to replace a function in our application without replacing it in dynamic libraries loaded by the application.<br />
<br />
In Linux, functions in dynamic libraries are resolved using two tables: PLT (procedure linkage table) and GOT (global offset table). When you call a library function, you actually call a stub code in the PLT. What PLT essentially does is loading the function address from the GOT. On an X86_64, it does that with the "jmpq *" instruction (<span style="white-space: pre-wrap;">ff 25)</span>. Initially GOT contains the pointer back to PLT, which will invoke the resolver which will write the correct entry to GOT.<br />
<br />
So, to replace a library function, we must patch the entry in GOT. How to find it? We can decode the offset to GOT (which is the relative to PLT + JMPQ instruction length) from the JMPQ instruction. How do we find the PLT then? It's simply the function pointer - that is, in C, use the function name as a pointer.<br />
<br />
We should call "dlsym" to forcibly resolve the entry in GOT before replacing it. For example, if we want to swap two dynamic functions. A complete example is available as a github gist:<br />
<a href="https://gist.github.com/astarasikov/9547918">https://gist.github.com/astarasikov/9547918</a><br />
<br />
<b><i>A more advanced example</i></b><br />
The approach described above will not work when you want to replace a function that is used by some dynamically loaded binary since the library will have its own set of GOT and PLT tables.<br />
<br />
On X86_64, we can find the PLT table by examining the contents of the ELF file. The section we care about is called ".rela.plt".<br />
<br />
We need to example the ELF header (Ehdr), find the section header (Shdr) and find the "dynsym" and "rela.plt" tables. The dynsym table unsurprisingly contains the pointers to dynamically resolved functions and can be used to find the function by name. You can find the link to the code below. I've mmap'ed the library file to get a pointer to its contents and use it to read the data. Some examples floating around the internet are using the "malloc", "read()" and "memcpy" calls for the same purpose. To get the pointer to the library in the application's virtual memory space, you can call "dlopen" and dereference the pointer returned by the function. You will need this address to convert the PLT offset into the address. Technically you can get away without reading/mmap'ing the library from disk, but since some sections are not mapped for reading, you will need to use "mprotect" to access them without receiving SIGSEGV.<br />
<br />
<a href="https://github.com/astarasikov/sxge/blob/vaapi_recorder/apps/src/sxge/apps/demo1_cube/hook-elf.c#L126">https://github.com/astarasikov/sxge/blob/vaapi_recorder/apps/src/sxge/apps/demo1_cube/hook-elf.c#L126</a><br />
<br />
<b><i>Alternatives</i></b><br />
The most straightforward alternative for such runtime patching is using the "LD_PRELOAD" variable and specify a library with the implementation of the hooked routine. The linker will then redirect all calls to that function in all libraries into the preloaded library. However, the obvious limitation of this approach is that it may be hard to get it working if you preload multiple libraries overriding the same symbol. It breaks some tools like "apitrace" (which is a tool to trace and replay OpenGL calls).<br />
<br />
<b><i>Practical example with Mesa</i></b><br />
Many GPUs nowadays contain encoders which will encode video to H.264 or other popular formats. Intel GPUs (namely HD4000 and Iris Graphics series) have an encoder for H.264. Some solutions like NVIDIA Grid utilize hardware capabilities to stream games over the network or provide the remote desktop facilities with low CPU load.<br />
<br />
Unfortunately both proprietary and the open-source drivers for the Intel hardware lack the ability to export and OpenGL resource into the encoder library which is a desired option for implementing a screen recorder (actually the proprietary driver from the SDK implements the advanced encoding algorithm as a binary library but uses the pre-compiled versions of the vaapi and libdrm for managind resources).<br />
<br />
If we don't share a GL texture with the encoder, we cannot use the zero-copy path. The texture contents will need to be read with "glReadPixels()" and then converted to the NV12 or YUV420 format (though it can be done by a GLSL shader before reading the data) and re-uploaded to the encoder. This is not an acceptable solution since each frame will take more than 30ms and cause 80% CPU thus leaving no processing power for other parts of the application. On the other hand, using a zero-copy approach will allow us to have a smooth 60FPS performance - the OpenGL side will not be blocked by "glReadPixels()" and CPU load will never exceed 10 percent.<br />
<br />
To be more precise, resource sharing support is present if one uses the "EGL" windowing system and the one of the MESA extensions. Technically, all GPU/encoder memory is managed by the DRM framework in Linux kernel. There is an extension which allows to obtain a DRM handle (which is an uint32) from an OpenGL texture. It is used by Wayland in the Weston display server. An example of using VAAPI to encode a DRM handle can be found in Weston source code:<br />
<a href="http://cgit.freedesktop.org/wayland/weston/tree/src/vaapi-recorder.c">http://cgit.freedesktop.org/wayland/weston/tree/src/vaapi-recorder.c</a><br />
<br />
Now, the problem is to find a handle of an OpenGL texture. Under EGL that can be done by creating an EGLImage from the texture handle and subsequently calling eglExportDRMImageMESA.<br />
<br />
In my case the problem was that I didn't want to use EGL because it is quite difficult to port a lot of legacy GLX codebase to EGL. Besides, GLX support is more mature and stable with the NVIDIA binary driver. So the problem is to get a DRM buffer out of a GL texture in GLX.<br />
<br />
Unfortunately GLX does not provide an equivalent extension and implementing one for Mesa and X11 is rather complicated due to the complexity of the code. One option would be to link against Mesa and use its internal headers to reach for the needed resources as done by Beignet, the open-source OpenCL driver for Intel (<a href="http://cgit.freedesktop.org/beignet/tree/src/intel/intel_dri_resource_sharing.c">http://cgit.freedesktop.org/beignet/tree/src/intel/intel_dri_resource_sharing.c</a>) which is error-prone. Besides, setting up such a build environment is complicated.<br />
<br />
Luckily, as we've seen above, the memory is managed by the DRM which means every memory allocation is actually an IOCTL call to kernel. Which means we can hook the ioctl and steal the DRM handle and later use it as the input to VAAPI. We need to look for the DRM_I915_GEM_SET_TILING ioctl code.<br />
<br />
The obvious limitation is that you will need to store a global reference which will be written by the ioctl hook which makes the code thread-unsafe. Luckily most OpenGL apps are single-threaded and even when they're not, there are less than dozen threads which can access OpenGL resources so lock contention is not an issue and a pthread mutex can be used.<br />
<br />
Another limitation is that to avoid memory corruption or errors, we need to carefully track the resources. One solution would be to allocate OpenGL texture prior to using VAAPI and deallocate after the encoder is destroyed. The other one is to hook more DRM calls to get a pointer to the "drm_intel_bo" structure and increase its reference count.<br />
<br />
<a href="https://github.com/astarasikov/sxge/blob/vaapi_recorder/apps/src/sxge/apps/demo1_cube/demo1_cube.cc#L119">https://github.com/astarasikov/sxge/blob/vaapi_recorder/apps/src/sxge/apps/demo1_cube/demo1_cube.cc#L119</a>I hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com0tag:blogger.com,1999:blog-1416969656167659289.post-91988639997153508032014-09-07T21:57:00.000+04:002014-09-08T00:34:19.965+04:00GSoC results or how I nearly wasted summer<span style="font-family: Arial, Helvetica, sans-serif;">This year I got a chance to participate in the Google Summer of Code with the FreeBSD project.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">I have long wanted to learn about the FreeBSD kernel internals and port it to some ARM board but was getting sidetracked whenever I tried to do it. So it looked like a perfect chance to stop procrastinating.</span><br />
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif;">Feel free to scroll down to the "other stuff" paragraph if you don't feel like reading thre paragraphs of a typical whinig associated with debugging C code.</span><br />
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;"><i><span style="background-color: purple; color: yellow;">Initially</span></i> I wanted to bring up some real piece of hardware. My choice was the Nexus 5 phone which has Qualcomm SoC. However, some of the FreeBSD developers suggested that instead I port the kernel to Android Emulator. It sounded like a sane idea since Android Emulator is available for the major OSes and is easy to set up which will allow to expose more people to the project. Besides, unlike QEMU, it can emulate some peripherals specific to a mobile device such as sensors.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /><i><span style="background-color: purple; color: yellow;">What is Android Emulator?</span></i> In fact, it is an emulator based on QEMU which emulates a virtual board called "Goldfish" which includes a set of devices such as interrupt controller, timer, input devices. It is worth noting that Android Emulator is based on an ancient version of QEMU though starting with Android L one can obtain the binaries of the emulator from git which is based on a more recent build.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />I have started from implementing the <span style="background-color: purple;"><span style="color: yellow;">basics: the interrupt controller, the timer and the UART</span></span> driver. That has allowed me to see the boot console but then I got stuck for literally months with kernel crashing at various points in what seemed like a totally random fashion. I have spent nights single-stepping the kernel with my sight slowly turning from barely red eyes to emitting hard radiation. It was definitely a problem with caching but ultimately I gave up trying to fix it. Since I knew that FreeBSD kernel worked fine on ARM in a recent version of QEMU, it was clearly a bug in Android Emulator, even though it has not manifested itself in Linux. Since Android Emulator is going to eventually get updated to a new QEMU base and I was running out of time, I decided to just use a nightly build of emulator instead of the one coming with the Android SDK and that fixed this particular issue. </span><span style="font-family: Arial, Helvetica, sans-serif;">But hey! At least I've learnt about the FreeBSD virtual memory management and the UMA allocator the hard way.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />Next up was fighting the timer and random freezes while initializing the random number generator (sic!). That was easy - turns out I just forgot to call the function. Anyway it would make sense to move that call out of the board files and call it for every board that does not define the "NO_EVENTTIMERS" option in the kernel config.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">Fast forward to the start of August when there are only a couple days left and I still don't have a working block device driver to boot rootfs! Writing a MMC driver turned out to be very easy and I started celebrating when I got the SD card image from Raspberry PI booting in Android Emulator and showing the login prompt.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">It worked but with a <span style="background-color: purple; color: yellow;">huge kludge</span> in the OpenFirmware bindings. Somehow one of the functions which should've returned the driver bus name returned (seemingly) random crap so instead of a NULL pointer check I had to add a check that the address points to the kernel VM range. I have then tracked down the real source of the problem.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">I was basing my MMC driver on the LPC SoC driver. LPC driver itself is suspicious - for example, the place where DMA memory is allocated says "allocating Framebuffer memory" which may be an indicator that it was written in a hurry and possibly only barely tested. At the MMC bus attach routine, it was calling the "<span style="line-height: 16.799999237060547px; white-space: pre;">device_set_ivars" function and setting the ivars pointer to the mmc host structure. I have observed the similiar pattern in some other drivers. In the OpenFirmware code, the ivars structure was dereferenced as a completely other structure and a string was taken from a member of this structure.</span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 16.799999237060547px; white-space: pre;"><br /></span></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 16.799999237060547px; white-space: pre;"><span style="background-color: purple; color: yellow;">How the hell did it happen</span> in a world where people are using the recent Clang compiler and compile with "-Wall" and "-Werror"? Uhh, well, if you're dealing with some kind of OOP in plain C, casting to void pointer and back and other ill-typed wicked tricks are inevitable. Just look at that prototype and cry with me:</span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 16.799999237060547px; white-space: pre;"><br /></span></span>
<div style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: x-small;">>> device_set_ivars<span style="background-color: white;">(</span>device_t<span style="background-color: white;"> </span>dev<span style="background-color: white;">, </span>void<span style="background-color: white;"> </span>*ivar<span style="background-color: white;">);</span></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="background-color: white;"><br /></span></span>
<span style="font-family: Arial, Helvetica, sans-serif;">For now, I'm pushing the changes to the following branch. Please note that I'm rebasing against master and force-pushing in process if you're willing to test.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">https://github.com/astarasikov/freebsd/tree/android_goldfish_arm_master</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">
So, what has been achieved and what am I still going to do? Well, debugging MMU issues and OpenFirmware bug has delayed me substantially so I've not done everything I've planned. Still</span><br />
<ul>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Basic hardware works in Goldfish: Timer, IRQs, Ethernet, MMC, UART</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">It is using NEWCONS for Framebuffer Console (which is cool!)</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">I've verified that Android Emulator works in FreeBSD with linuxulator</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">I have fixed Android Emulator to compile natively on FreeBSD and it mostly works (at least, console output) except graphics (which I think should be an easy fix. there are a lot of places where "ifdef __linux__" is completely unjustified and should rather be enabled for all unices)</span></li>
</ul>
<span style="background-color: purple; color: yellow; font-family: Arial, Helvetica, sans-serif;">How to test and contribute?</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">You can use the linux build of Android Emulator using Linuxulator. Unfortunately, linuxulator only supports 32-bit binaries so we cannot run nightly binaries which have the MMU issues fixed.</span><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;">Therefore, I have fixed the emulator to compile natively on FreeBSD! I've pushed the branch named "l-preview-freebsd" to github. You will also need the gtest. I cloned the repo from CyanogenMod and pushed a tag named "l-preview-freebsd", which is actually "cm-11.0".</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: purple; color: yellow; font-family: Arial, Helvetica, sans-serif;">Compiling the emulator:</span><br />
<span style="font-size: x-small;">>> git clone https://github.com/astarasikov/qemu.git <br />>> https://github.com/astarasikov/android_external_gtest <br />>> cd cd android_external_gtest <br />>> git checkout cm-11.0 <br />>> cd ../qemu.git <br />>> git checkout l-preview-freebsd <br />>> bash android-build-freebsd.sh</span><br />
<span style="font-size: x-small;"><br /></span>
<div style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Please refer to <a href="https://github.com/astarasikov/freebsd/blob/android_goldfish_arm_master/sys/arm/goldfish/README">README file in git</a> for the detailed instructions on building kernel.</span></div>
<br />
<div>
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif;"><span style="background-color: white;">The only thing that really bothers me about low-level and embedded stuff is that it is extremely difficult to estimate how long it may take to implement a certain feature because every time you end up debugging obscure stuff and the "useful stuff you learn or do" / "time wasted" ratio is very small. On a bright side, while *BSD lag a bit behind Linux in terms of device drivers and performance optimizations, reading and debugging *BSD code is much easier than GNU projects like EGLIBC, GCC and GLib.</span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br style="background-color: white; color: #222222;" /></span>
<span style="background-color: purple; color: yellow; font-family: Arial, Helvetica, sans-serif;">Other stuff.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="background-color: white; color: #222222;">Besides GSoC at the start of summer I had a chance to work with Chris Wade (who is an amazing hacker, by the way). We started working on an ARM virtualization project and have spent nice days debugging caching issues and instruction decoding while trying to emulate a particular cortex A8 SoC on an A15 chip. Unfortunately working on GSoC, going to a daily job and doing this project simultaneously turned out to be surprisingly difficult and I had to quit at least one of the activities. Still I wish Chris good luck with this project and if you're interested in virtualization and iPhones, sign up for early demo at </span><a href="http://virtu.al/" style="background-color: white; color: #1155cc;" target="_blank">virtu.al</a></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br style="background-color: white; color: #222222;" /></span>
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif;">Meanwhile I'm planning to learn ARMv8 ISA. It's a pity there is no hackable hardware available for reasonable prices yet. QEMU works fine with VirtIO peripherals though. But I'm getting increasingly worried about devkits costing more and more essentially making it more difficult for a novice to become an embedded software engineer.</span></div>
I hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com0tag:blogger.com,1999:blog-1416969656167659289.post-36062275392431316382014-06-06T08:01:00.000+04:002014-06-06T08:01:18.624+04:00On Free Software and if proprietary software should be considered harmfulI communicate with a lot of people on the internet and they have various opinions on FOSS ranging from "only proprietary software written by a renowned company can deliver quality" to "if you're using binary blobs, you're a dick". Since these issues arise very often in discussions, I think I need to write it up so I can just shove the link next time.<br />
<br />
On the one hand, I am a strong proponent of free and open-source software and I have contributed to several projects related to porting Linux to embedded hardware, especially mobile phones (htc-linux.org, Replicant and I also consulted some of the FreeSmartPhone.org people). Here are the reasons I like free software (as well as free hardware and free knowledge):<br />
<br />
<ul>
<li>The most important for me is that you can learn a lot. I have mostly learnt C and subsequently other languages by hacking on the linux kernel and following interesting projects done by fellow developers</li>
<li>Practically speaking, it is just easier to maintain and develop software when you have the source code</li>
<li>When you want to share a piece of data or an app with someone, if you deal with closed software, you force them into buying a paid app which may compromise their security</li>
<li>You can freely distribute your contributions, your cool hacks and research results without being afraid of pursuit by some patent troll</li>
</ul>
But you know, some people are quite radical in their FOSS love. They insist that using anything non-free is a sin. While I highly respect them for their attitude, I have a different point of view and I want to comment on some common arguments against using or developing non-free software:<br />
<div>
<ul>
<li>"Oh, it may threaten your privacy, how can you run untrusted code"? My opinion here is that running untrusted firmwares or drivers for devices is not a big deal because unless you manufacture the SoC and all the peripherals yourself, you can not be sure of what code is running in your system. For example, most X86 CPUs have SMM mode with a proprietary hypervisor, most ARMs have TrustZone mode. If your peripheral does not require you to load the firmware, it just means that the firmware is stored in some non-volatile memory chip in hardware, and you don't have the chance to disable the device by providing it with a null or fake binary. On the other hand, if your device uses some bus like I2C or USB which does not have DMA capabilities or uses IOMMU to restrict DMA access, you are relatively safe even if it runs malicious code.</li>
<li>"You can examine open source software and find backdoors in it". Unfortunately this is a huge fallacy. First of all, some minor errors which can lead to huge vulnerabilities can go unnoticed for years. Recent discoveries in OpenSSL and GnuTLS came as a surprise to many of us. And then again, have you ever tried to develop a piece of legacy code with dozens of nested ifdefs when you have no clue which of them get enabled in the final build? In this case, analyzing disassembled binaries may even be easier.</li>
<li>"By developing or using non-free software you support it". In the long run it would be better for humanity to have all knowledge to be freely accessible. However, if we consider a single person, their priorities may differ. For example, until basic needs (which are food, housing, safety) are satisfied, one may resort to developing close-sourced software for money. I don't think it's bad. For me, the motivation is knowledge. In the end, even if you develop stuff under an NDA, you understand how it works and can find a way to implement a free analog. This is actually the same reason I think using proprietary software is not bad in itself. For example, how could you expect someone to write a good piece of free software - an OpenGL driver, an OS kernel, a CAD until they get deeply familiar with existing solutions and their limitations?</li>
</ul>
Regarding privacy, I am more concerned about the government and "security" agencies like NSA which have enough power to change laws and fake documents. Officials change, policy changes and even people who considered themselves safe and loyal patriots may be suddenly labeled traitors.</div>
<div>
<br /></div>
<div>
In the end, it boils down to individual people and communities. Even proprietary platforms like Palm, Windows Mobile or Apple iOS had huge communities of helpful people who were ready to develop software, reverse-engineer hardware and of course help novices. And there are quite some douchebags around free software. Ultimately, just find the people you feel comfortable around, it is all about trust.</div>
I hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com0tag:blogger.com,1999:blog-1416969656167659289.post-83352849656340830092014-06-06T07:56:00.003+04:002014-06-06T21:07:17.985+04:00Minor notes about recent stuffI've been quite busy with work and university recently so I did not have much time to work on my projects or write rants, so I decided to roll out a short post discussing several unrelated ideas.<br />
<br />
<i><b style="background-color: lime;">On deprecating Linux (not really)</b></i><br />
<span style="background-color: white;">Recently several Russian IT bloggers have been writing down their ideas about what's broken with Linux and how it should be improved. Basically, one guy started by saying we need to throw away POSIX and write everything from scratch and the other two are trying to find a compromise. You may fire up Google Translate and follow the discussion at </span><a href="http://lionet.livejournal.com/131533.html">http://lionet.livejournal.com/131533.html</a><br />
<br />
I personally think that these discussions are a bit biased because all these guys are writing from the point of view of an engineer working on distributed web systems. At the same time, there are domains like embedded software, computer graphics, high-performance computations which have other challenges. What's common is that people are realizing <span style="background-color: cyan;">the obvious idea</span>: generic solutions designed to work for everyone (like POSIX) limit the performance and flexibility of your system, while domain-specific solutions make your job easier but they may not fit well into what other people are doing.<br />
<br />
Generally both in high-performance networking and graphics people are trying to do the same: reduce the number of context switches (it is true that on modern X86 a context switch is relatively cheap and we may use a segmented memory model like in Windows CE 5.0 instead of the common user/kernel split, but the problem is not the CPU execution mode switch but that system calls like "flush", "ioctl" and library calls like "glFlush()" are used as a point of synchronization where a lot of additional work is done besides the call itself) and move all execution into userspace. Examples include asynchronous network servers using epoll and cooperative scheduling, Intel's networking stack in user land (Netmap, DPDK), modern Graphics APIs (Microsoft DirectX 12, AMD Mantle, Apple Metal). The <span style="background-color: cyan;">cost of maintaining abstractions - managing buffers and contexts in drivers</span> - has risen so high that neither hardware vendors want to write complex drivers nor they can deliver performance. So, we'll have to step back and learn to use hardware wisely once again.<br />
<br />
Actually I like the idea of using <span style="background-color: cyan;">minimalistic runtimes on top of hypervisors</span> like Erlang on Xen from the point of simplicity. Still, security and access control are open questions. For example, capability-based security as in L4, has always looked interesting, but whether cheap system calls and dramatically reduced "trusted code base" promises have been fulfilled is very arguable. Then again, despite the complexity of linux, its security is improving because of control groups which are heavily utilized by docker and systemd distros. Another issue is that lightweight specific solutions are rarely reusable. Well, from the economic point of view a cheap solution that's done overnight and does its job is just perfect, but generally the amount of NIH and engineers basically writing the same stuff - drivers, applications, libraries and servers in an absolutely identical way but for a dozen identical OSs is amazing and rather uncreative.<br />
<br />
Anyway, it looks like we're there again: <a href="http://dreamsongs.com/RiseOfWorseIsBetter.html">rise of Worse is Better</a><br />
<br />
<i><b style="background-color: lime;">Work (vaapi, nix).</b></i><br />
At work, among other things, I was asked to figure out how to use the Intel GPU for H.264 video encoding. Turns there are two alternatives: the open source VAAPI library and the proprietary Intel Media SDK. Actually, the latter still uses a modified version of VAAPI, but I feel that fitting it into our usual deployment routine is going to be hard, because even basic critical components of the driver stack, such as the kernel KMS module and libdrm.so are provided in the binary form.<br />
<br />
Actually VAAPI is very low-level. One thing that puzzled me initially is that it does not generate H.264 bitstream. You have to make it yourself and feed into the encoder via a special buffer type. Besides, you have to manually take the reconstructed picture and feed it as a reference for subsequent frames. There are several implementations using this API for encoding: gstreamer, ffmpeg, vlc. I have spent quite some time until I got it to encode a sample chunk of YUV data. Ultimately my code started looking identical to the "avcenc.c" sample from libva except that encoder data is stored in a "context" structure instead of global variables.<br />
<br />
My advice is that if you want to learn about video decoding APIs on linux and Android, take a look at Chromium source code (well, you may expect to find a solution for any relevant computer science or engineering problem given how much code it contains). And also take a look at GST, FFMPEG and VLC. Notice how each of them has its own way of managing buffers (and poor error handling btw).<br />
<ul>
<li><a href="http://src.chromium.org/svn/trunk/src/content/common/gpu/media">Chromium</a></li>
<li><a href="https://gitorious.org/vaapi/gstreamer-vaapi/source/e670e3600709602ddc6a9db651cc18e91dba4a09:gst/vaapi/gstvaapiencode_h264.c">GSTreamer</a></li>
<li><a href="https://gitorious.org/vaapi/gstreamer-vaapi/source/e670e3600709602ddc6a9db651cc18e91dba4a09:gst-libs/gst/vaapi/gstvaapiencoder_h264.c">GSTreamer (continued)</a></li>
<li><a href="https://github.com/BtbN/vlc-vaapi-enc/blob/master/vlc-h264-vaapi-enc.c">VLC (VideoLAN)</a></li>
<li><a href="https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/vaapi_h264.c">FFMpeg (libavcodec)</a></li>
</ul>
<div>
Another thing we're using at work is the Nix package manager. I have always wanted to try it but did not really do it until coming to work to this place. Nix is a fantastic concept (Even <a class="g-profile" href="https://plus.google.com/114072895780946523032" target="_blank">+Norman Feske</a> got inspired by it for their Genode OS). However, we've notices a slowdown when compiling software under Nix. Here are some of my observations:</div>
<div>
<ul>
<li>Compiling a simple C file with just "main" function takes 30ms in linux but >300ms in Nix chroot. Nix installs each library to a separate prefix and uses LDFLAGS and CFLAGS to direct the compiler to use them. Gcc then iterates over each of these directories trying to find each library which ends up being slow. <span style="background-color: cyan;">Anyone knows a workaround?</span></li>
<li>Running gcc under the "perf" profiler shows that it spends most of its time in multibyte string collation functions. I find it ridiculous that exporting "LC_ALL=C" magically makes the compilation time fall down to 100ms.</li>
</ul>
<div>
<b><i style="background-color: lime;">Hacking on FreeBSD</i></b></div>
</div>
<div>
As a part of my GSoC project I got the FreeBSD kernel booting on Android emulator and I've just have to write the virtual ethernet and MMC drivers. Unfortunately my university has distracted me a lot from this project but now that I have time I hope to finish it by midterm. And then I'll probably port FreeBSD to Qualcomm MSM8974 SoC. Yay, red Nexus 5 is an awesome phone!</div>
<div>
<br /></div>
<div>
<i><b style="background-color: lime;">My little project (hacking is magic)</b></i></div>
<div>
Long time ago I decided to give Windows Mobile emulation a shot and got the kernel to start booting in QEMU. Since Microsoft's Device Emulator was open-source and emulated a real Samsung S3C2410 SoC, it was easy. I still plan to get it fully working one day.</div>
<div>
<br /></div>
<div>
But QEMU is a boring target actually. What's more interesting is developing a bare-metal hypervisor for A15 and A7 CPUs. Given the progress made by Rob Clark on reverse-engineering Qualcomm Adreno GPU driver, I think it could be doable with reasonable effort to emulate the GPU and consequently enough hardware to run Windows Mobile and Windows RT. A very interesting thing is that ARMs trap almost all coprocessor registers for guest access (privilege levels 0 and 1) meaning you can fake any CPU ID or change memory access behavior by modifying caching and buffering settings.</div>
<div>
<br /></div>
<div>
What is really interesting is that there are a lot of Chinese phones which copy Nokia smartphones, iPhones, Android phones. Recent revisions of Mediatek MTK SoCs are Arm A7 meaning they support virtualization. Ultimately it means someone could make a high quality replica of any phone and virtualize any SoC without a trace which has interesting security implications.</div>
<div>
<br /></div>
<div>
<b><i style="background-color: lime;">Software sucks!</i></b></div>
<div>
The other day some <span style="background-color: yellow;">systemd</span> update came out which totally broke my debian bootup. Now, there's a race condition between the encrypted disk password entry and root password entry for "recovery". Then, latest kernel (3.15-rc3) OOPSes and crashes randomly on ACPI events which subsequently breaks networking and spawning new processes.</div>
<div>
<br /></div>
<div>
Ultimately after a day of struggle my rootfs broke and after and FSCK everything was gone. So now I'm deciding what distro I should try instead. Maybe ArchLinux? Either way, I have to build a lot of software from source and install to /usr/local or custom prefixes for development.</div>
<div>
<br /></div>
<div>
The easiest way would be to install into a VM in OS X. But then, I want to fix issues with drivers, especially GPU switching in the macbook. On the other hand, I spent ages fixing switchable graphics on an older Sony VAIO laptop and resorted to an ugly hack to force Intel GPU. Since GPU switching is not working properly in Linux, maybe I should write a graphics multiplexor driver for FreeBSD and ditch Linux altogether? FreeBSD looks much more attractive these days with clang and no systemd and no FreeDesktop.org and no Lennart Poettering.</div>
I hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com0tag:blogger.com,1999:blog-1416969656167659289.post-80791223734045603912014-03-18T02:53:00.001+04:002014-03-18T02:53:19.039+04:00A story about clang toolsI've always wanted to try writing a toy compiler, but have not made myself actually learn the theory of parsing (I plan to do it and post some notes into the blog soon though). However, recently I've been playing with Clang and LLVM. I've not yet used it for compiling, but I want to share my experience of using and extending Clang's error detection tools.<br />
<br />
LLVM is a specification of platform-independent bytecode and a set of tools aimed to make the development of JITed interpreters and portable native binaries easier. A significant portion of work for LLVM was done by Apple and nowadays it is widely used in industry. For example, NVIDIA uses it for compiling CUDA code, AMD uses it to generate shaders in its open-source driver. Clang is a parser and a compiler for a set of C-like languages, which includes C, C++ and Objective-C. This compiler has several properties that may be interesting for developers:<br />
<ul>
<li>It allows you to traverse the parsed AST and transform it. For example, add or remove the curly brackets around if-else conditionals to troll your colleagues.</li>
<li>It allows you to define custom annotations via the __attribute__ extension which again can be intercepted after the AST is generated but is not yet compiled.</li>
<li>It supports nearly all the features of all revisions of C and C++ and is compatible with the majority of GCC compiler options which allows to use it as a drop-in replacement. By the way, FreeBSD has switched to Clang, and on Apple OS X gcc is actually a symlink to clang!</li>
</ul>
So, why should you care? Imagine how many times you wanted to add some cool feature but realized macros were not enough. Or you wanted to enforce some code convention? With clang one could easily write a static code analyzer that would catch the notorious Apple "double fail" bug. Which makes me wonder why they did not use their own technology :)<br />
<br />
LLVM provides several frameworks for finding bugs at runtime. For example, AddressSanitizer and MemorySanitizer to catch access to uninitialized or unallocated memory.<br />
<br />
I was given the following interesting problem at work: build some solution that would allow to detect where the application is leaking memory. Sounds like a common problem, with no satisfying answer.<br />
<ul>
<li>Using Valgrind is prohibitively slow - a program running under it can be easily 50 times slower than without it.</li>
<li>Using named SLAB areas (like linux kernel does) is not an option. First of, in the worst case using SLAB means only half of the memory is available for the allocation. Secondly, such approach allows to know objects of what class are occupying the memory, but now where and why they were allocated</li>
<li>Using TCMalloc which hooks malloc/free calls also turned out to be slow enough to cause different behaviour in release and debugging environment, so some lightweight solution had to be designed.</li>
</ul>
<div>
Anyway, while thinking of a good way to do it, I found out that Clang 3.4 has something called LeakSanitizer (also lsan and liblsan) which is already ported to GCC 4.9. In short, it is a lightweight version of tcmalloc used in Google Perftools. It collects the information about memory allocations and prints leak locations when the application exits. It can use the LLVM symbolizer or GCC libbacktrace to print human-readable locations instead of addresses. However, it has some issues:</div>
<div>
<ul>
<li>It has an explicit check in the __lsan::DoLeakCheck() function which disallows it to be called twice. Therefore, we cannot use it to print leaks at runtime without shutting down the process</li>
<li>Leak detection cannot be turned off when it is not needed. Hooked malloc/memalign functions are always used, and the __lsan_enable/disable function pair only controls whether statistics should be ignored or not.</li>
</ul>
</div>
<div>
The first idea was to patch the PLT/GOT tables in ELF to dynamically choose between the functions from libc and lsan. It is a very dirty approach, though it will work. You can find a code example at https://gist.github.com/astarasikov/9547918.</div>
<div>
<br /></div>
<div>
However, patching GOT we only divert the functions for a single binary, and we'd have to patch the GOT for each loaded shared library which is, well, boring. So, I decided to patch liblsan instead. I had to patch it either way, to remove the dreaded limitation in DoLeakCheck. I figured it should be safe to do. Though there is a potential deadlock while parsing ELF header (as indicated by a comment in lsan source), you can work around it by disabling leak checking in global variables.</div>
<div>
<br /></div>
<div>
What I did was to set up a number of function pointers to the hooked functions, initialized with lsan wrappers (to avoid false positives for memory allocation during libc constructors) and add two functions, __lsan_enable_interceptors and __lsan_disable_interceptors to switch between libc and lsan implementations. This should allow to use leak detection for both our code and third-party loadable shared libraries. Since lsan does not have extra dependencies on clang/gcc it was enough to stick a new CMakeLists.txt and it can now be built standalone. So now one can load the library with LD_PRELOAD and query the new functions with "dlsym". If they're present - it is possible to selectively enable/disable leak detection, if not - the application is probably using vanilla lsan from clang.</div>
<div>
<br /></div>
<div>
There are some issues, though</div>
<div>
<ul>
<li>LSAN may have some considerable memory overhead. It looks like it doesn't make much sense to disable leak checking since the memory consumed by LSAN won't be reclaimed until the process exits. On the other hand, we can disable leak detection at application startup and only enable it when we need to trace a leak (for example, an app has been running continuously for a long time, and we don't want to stop it to relaunch in a debug configuration).</li>
<li>We need to ensure that calling a non-hooked free() on a hooked malloc() and vice-versa does not lead to memory corruption. This needs to be looked into, but it seems that both lsan and libc just print a warning in that case, and corruption does not happen (but a memory leak does, therefore it is impractical to repeatedly turn leak detection on and off)</li>
</ul>
</div>
<div>
We plan to release the patched library once we perform some evaluation and understand whether it is a viable approach.</div>
<br />
Some ideas worth looking into may be:<br />
<ul>
<li>Add a module to annotate and type-check inline assembly. Would be good for Linux kernel</li>
<li>Add a module to trace all pointer puns. For example, in Linux kernel and many other pieces of C code, casting to a void pointer and using the container_of macro is often used to emulate OOP. Now, using clang, one could possibly allow to check the types when some data is registered during initialization, casted to void and then used in some other function and casted back or even generate the intermediate code programmatically.</li>
<li>Automatically replace shared variables/function pointer calls with IPC messages. That is interesting if one would like to experiment with porting Linux code to other systems or turning Linux into a microkernel</li>
</ul>
I hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com0tag:blogger.com,1999:blog-1416969656167659289.post-12942818340898378942014-01-27T02:56:00.002+04:002014-01-27T02:56:48.902+04:00porting XNU to ARM (OMAP5 CPU)Two weeks ago I have taken some time to look at the XNU port to the ARM architecture done by the developer who goes by the handle "winocm". Today I've decided to summarize my experience.<br />
<br />
Here is a brief checklist if you want to start porting XNU to your board:<br />
<ul>
<li>Start from reading Wiki https://github.com/darwin-on-arm/wiki/wiki/Building-a-bootable-system</li>
<li>Clone the DeviceTrees repository: https://github.com/darwin-on-arm/DeviceTrees . Basically, you can use slightly modified DTS files from Linux, but due to the fact that DTC compiler is unfinished, you'll have to rename and comment out some entries. For example, macros in included C headers are not expanded, so you'll have to hardcode values for stuff like A15 VGIC IRQs</li>
<li>Get image3maker which is a tool to make images supported both by GenericBooter and Apple's iBoot bootloaders https://github.com/darwin-on-arm/image3maker</li>
<li>Use the DTC from https://github.com/winocm/dtc-AppleDeviceTree to compile the abovementioned DTS files</li>
<li>Take a look at init/main.c . You may need to add a hackish entry the way it's done for the "HD2" board to limit the amount of RAM available.</li>
<li>I have built all the tools on OS X, but then found out that it's easier to use the prebuilt linux chroot image available at: https://github.com/stqism/xnu-chroot-x86_64</li>
</ul>
<br />
The most undocumented step is actually using the image3maker tool and building bootable images. You have to put the to the "images" directory in the GenericBooter-next source. As for the ramdisk, you may find some on github or unpack the iphone firmware, but I simply created an empty file, which is OK since I've not got that far in booting.<br />
<br />
Building GenericBooter-next is straightforward, but you need to export the path to the toolchain, and possibly edit the Makefile to point to the correct CROSS_COMPILE prefix<br />
<br />
<span style="background-color: black; color: lime;">rm images/Mach.*</span><br />
<span style="background-color: black; color: lime;">../image3maker/image3maker -f ../xnu/BUILD/obj/DEBUG_ARM_OMAP5432_UEVM/mach_kernel -t krnl -o images/Mach.img3</span><br />
<span style="background-color: black; color: lime;">make clean</span><br />
<span style="background-color: black; color: lime;">make</span><br />
<div>
<br /></div>
For the ramdisk, you should use the "rdsk" type, and "dtre" for the Device Tree (apparently there's also xmdt for xml device tree, but I did not try that);<br />
<br />
Booting on the omap5 via fastboot (without u-boot):<br />
<span style="background-color: black; color: lime;">./usbboot -f &</span><br />
<span style="background-color: black; color: lime;">fastboot -c "-no-cache serial=0 -v -x cpus=1 maxmem=256 mem=256M console=ttyO2" -b 0x83ff8040 boot vmlinux.raw</span><br />
<br />
Some notes about the current organization of the XNU kernel and what can be improved:<br />
<ul>
<li>All makefiles containing board identifiers should be sorted alphabetically, just for convenience when adding newer boards.</li>
<li>Look into memory size limit. I had to limit the RAM to 256Mb in order to boot. If one enables the whole 2Gb available on the OMAP5432 uEVM board, the kernel fails to "steal pages" during the initialization (as can be seen in the screenshot).</li>
<li>I have encountered an issue </li>
</ul>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgY1S72oHOyYbWe-oBkdJVdf_6ptSbSDYwmuWOjil7CEHtGI2Y8f2fDI9l0YIdO7iEim4SK7-XInwaISRjyArFl9W_r8eKF_-ZJsonGZCo259zhQ7ZKkGPi3ajMjl3s5fwiw04EfHIS3RU/s1600/xnu_screenshot.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgY1S72oHOyYbWe-oBkdJVdf_6ptSbSDYwmuWOjil7CEHtGI2Y8f2fDI9l0YIdO7iEim4SK7-XInwaISRjyArFl9W_r8eKF_-ZJsonGZCo259zhQ7ZKkGPi3ajMjl3s5fwiw04EfHIS3RU/s1600/xnu_screenshot.jpg" height="424" width="640" /></a></div>
OMAP5 is an ARM Cortex-A15 core. Currently XNU port only has support for the A9 CPU, but if we're booting without SMP and L2 cache, the differences between these architectures are not a big problem. OMAP5 has a generic ARM GIC (Global Interrupt Controller), which is actually compatible to the GIC in the MSM8xxx CPUs, namely with APQ8060 in HP TouchPad, the support for which was added by winocm. UART is a generic 16550, compatible to the one in OMAP3 chip. Given all this, I have managed to get the kernel basically booting and printing some messages to the UART.<br />
<br />
Unfortunately, I have not managed to bring up the timer yet. The reason is that I was booting the kernel via USB directly after OMAP5 internal loader, skipping u-boot and hardware initialization. Somehow the internal eMMC chip in my board does not allow me to overwrite the u-boot partition (although I did manage to do it once when I received the board)<br />
<br />
I plan to look into it once again, now with hardware pre-initialized by u-boot, and write a detailed post. Looks like the TODO is the following:<br />
<br />
<ul>
<li>Mirror linux rootfs (relying on 3rd party github repos is dangerous)</li>
<li>Bringing up OMAP5 Timer</li>
<li>Building my own RAMDisk</li>
<li>Enabling eMMC support</li>
<li>Playing with IOKit</li>
</ul>
I hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com0tag:blogger.com,1999:blog-1416969656167659289.post-92191746819551485142013-12-27T03:43:00.000+04:002013-12-27T03:43:07.186+04:00I don't evenLook, to some extent I like Mac OS X. It's a UNIX, it has some software (though, very little compared to linux). I like the objective-c language, and developing for iOS is a huge buzz with high salaries. Oh, and it has DTrace. Other than that I don't really have a reason to like it.<br />
<br />
Some things about this OS are undocumented and badly broken. Take file system management for example. Tonight I looked at the free disk space and found out that some entity named "Backups" occupied 40GB. Turns out it's Time Machine's local snapshots. The proper way to get rid of them would be to disable automatic backups in Time Machine. One can also disable local snapshots from command line like:<br /><br />
<span style="background-color: black; color: lime;">tmutil disablelocal</span><br />
<span style="background-color: white;">And this is where I've effed up. This did not remove any space. So, I went ahead and removed the ".MobileBackups" and "Backups.backupdb" folders. NEVER EVER FUCKING DO IT. The thing is that Time Machine, according to some reports, creates hard links to directories (sic!), and now I just lost those 40 gigs - they ain't showing up in "du -s", but they show up as "Others" in the disk space info. Sooo. next time, use the "tmutil delete" to delete those directories.</span><br />
<span style="background-color: white;"><br /></span>
<span style="background-color: white;">Ok, I've re-enabled the snapshots with "tmutil enablellocal" and disabled them with the GUI. After that, I opened the Disk Utility and clicked "Verify Disk". It reported that the root FS was corrupted, I had to reboot to the recovery image and run the "Disk Repair". It's really confusing that OS X can perform live fsck (it just freezes all IO operations until fsck is done) but can't repair a live FS.</span><br />
<span style="background-color: white;"><br /></span>
<span style="background-color: white;">And a bonus picture for you. In the process I've unplugged my USB disk several times without unmounting and the FS got corrupted. This is what I got for trying to repair it.</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmquCX1U3x2o8Zape8xt2kzjOd4YS_pX1E2xhBrrFEpEPRC94kwEgRd1N2lWZNikid6OMNz-qNAMiNojY85GlU_pwY1D-rx-j4RDMKaMT81KzKLNiSakXeJR4wDrKJBLEXDH-jg_BqPPo/s1600/Screen+Shot+2013-12-27+at+3.31.00.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="118" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmquCX1U3x2o8Zape8xt2kzjOd4YS_pX1E2xhBrrFEpEPRC94kwEgRd1N2lWZNikid6OMNz-qNAMiNojY85GlU_pwY1D-rx-j4RDMKaMT81KzKLNiSakXeJR4wDrKJBLEXDH-jg_BqPPo/s320/Screen+Shot+2013-12-27+at+3.31.00.png" width="320" /></a></div>
<span style="background-color: white;"><br /></span>I hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com0tag:blogger.com,1999:blog-1416969656167659289.post-51845635543139576272013-11-26T01:04:00.000+04:002013-11-26T01:04:17.481+04:00KVM on ARM Cortex A15 (OMAP5432 UEVM)Hi! In this post I'll summarize the steps I needed to do in order to get KVM working on the OMAP5 ARM board using the virtualization extensions.<br />
<br />
<h3>
ARM A15 HYP mode.</h3>
In Cortex-A15, ARM have introduced a new operating mode called HYP (hypervisor). It has lower permissions than TruztZone. In fact, HYP splits the "insecure" world into two parts, one for hypervisor and the other one for the guests. By default on most boards the system boots into the insecure non-HYP mode. To enter the HYP mode, one needs to use platform-specific ways. For OMAP5 this involves making a call to the TrustZone which will restart the insecure mode cores.<br />
<br />
A good overview of how virtualization support for ARM was added to Linux is available at LWN.<br />
<a href="http://lwn.net/Articles/557132/">http://lwn.net/Articles/557132/</a><br />
<br />
<h3>
Ingo Molnar HYP patch</h3>
There was a patch for u-boot to enable entering HYP mode on OMAP5 by Ingo Molnar. Too bad, it was either written for an early revision of omap5 or poorly tested. It did not work for my board, so I had to learn about OMAP5 TrustZone SCM commands from various sources and put up a patch (which is integrated to my u-boot branch).<br />
If you're interested, you can take a look at the corresponding mailing list entry.<br />
<br />
<a href="http://u-boot.10912.n7.nabble.com/RFD-OMAP5-Working-HYP-mode-td163302.html">http://u-boot.10912.n7.nabble.com/RFD-OMAP5-Working-HYP-mode-td163302.html</a><br />
<h3>
Preparing u-boot SD card</h3>
Get the android images from TI or build them yourself. You can use the usbboot tool to boot images from the PC. Or even better, you can build u-boot (this is the preferred way) and then you won't need android images. But you may need the TI GLSDK for the x-loader (MLO). Creating an SD card with u-boot is the same as for omap3 and omap4, so I'll leave this out. There is some magic with creating a proper partition table, so I advise that you get some prebuilt image (like ubuntu for pandaboard) and then replace the files in the FAT partition.<br />
<br />
<a href="http://software-dl.ti.com/omap/omap5/omap5_public_sw/OMAP5432-EVM/5AJ_1_5_Release/index_FDS.html">http://software-dl.ti.com/omap/omap5/omap5_public_sw/OMAP5432-EVM/5AJ_1_5_Release/index_FDS.html</a><br />
<a href="http://www.omappedia.com/wiki/6AJ.1.1_Release_Notes">http://www.omappedia.com/wiki/6AJ.1.1_Release_Notes</a><br />
<br />
Please consult the OMAP5432 manual on how to set up the DIP switches to boot from SD card.<br />
<h3>
Source code</h3>
For u-boot:<br />
<a href="https://github.com/astarasikov/uboot-tegra/tree/omap5_hyp_test">https://github.com/astarasikov/uboot-tegra/tree/omap5_hyp_test</a><br />
<br />
For linux kernel:<br />
<a href="https://github.com/astarasikov/linux/tree/omap5_kvm_hacks">https://github.com/astarasikov/linux/tree/omap5_kvm_hacks</a><br />
<br />
Linux kernel is based on the TI omapzoom 3.8-y branch. I fixed a null pointer in the DWC USB3 driver and some issues with the 64-bit DMA bitmasks (I hacked the drivers to work with ARM LPAE, but this probably broke them for anything else. The upstream has not yet decided on how this should be handled).<br />
<br />
<h3>
Compiling stuff</h3>
First, let's build the u-boot<br />
<span style="background-color: black; color: lime;">#!/bin/bash</span><br />
<span style="background-color: black; color: lime;">export PATH=/home/alexander/handhelds/armv6/codesourcery/bin:$PATH</span><br />
<span style="background-color: black; color: lime;">export ARCH=arm</span><br />
<span style="background-color: black; color: lime;">export CROSS_COMPILE=arm-none-eabi-</span><br />
<span style="background-color: black; color: lime;">U_BOARD=omap5_uevm</span><br />
<span style="background-color: black; color: lime;">make clean</span><br />
<span style="background-color: black; color: lime;">make distclean</span><br />
<span style="background-color: black; color: lime;">make ${U_BOARD}_config</span><br />
<span style="background-color: black; color: lime;">make -j8</span><br />
<br />
you'll get the u-boot.bin and the u-boot.img (which can be put to the SD card). Besides, that will build the mkimage tool that we'll need later.<br />
<br />
Now, we need to create the boot script for u-boot that will load the kernel and the device tree file to RAM.<br />
<br />
<span style="background-color: black; color: lime;">i2c mw 0x48 0xd9 0x15</span><br />
<span style="background-color: black; color: lime;">i2c mw 0x48 0xd4 0x05</span><br />
<span style="background-color: black; color: lime;">setenv fdt_high 0xffffffff</span><br />
<span style="background-color: black; color: lime;">fdt addr 0x80F80000</span><br />
<span style="background-color: black; color: lime;">mmc rescan</span><br />
<span style="background-color: black; color: lime;">mmc part</span><br />
<span style="background-color: black; color: lime;">fatload mmc 0:1 0x80300000 uImage</span><br />
<span style="background-color: black; color: lime;">fatload mmc 0:1 ${fdtaddr} omap5-uevm.dtb</span><br />
<span style="background-color: black; color: lime;">setenv mmcargs setenv bootargs console=ttyO2,115200n8 root=/dev/sda1 rw rootdelay=5 earlyprintk nosmp</span><br />
<span style="background-color: black; color: lime;">printenv</span><br />
<span style="background-color: black; color: lime;">run mmcargs</span><br />
<span style="background-color: black; color: lime;">bootm 0x80300000 - ${fdtaddr} </span><br />
<br />
Now, compile it to the u-boot binary format:<br />
<span style="background-color: black; color: lime;">./tools/mkimage -A arm -T script -C none -n "omap5 boot.scr" -d boot.txt boot.scr</span><br />
<br />
<br />
<h3>
Building linux:</h3>
<span style="background-color: black; color: lime;">export PATH=/home/alexander/handhelds/armv6/linaro-2012q2/bin:$PATH</span><br />
<span style="background-color: black; color: lime;">export ARCH=arm</span><br />
<span style="background-color: black; color: lime;">export CROSS_COMPILE=/home/alexander/handhelds/armv6/linaro-2012q2/bin/arm-none-eabi-</span><br />
<span style="background-color: black; color: lime;">export OMAP_ROOT=/home/alexander/handhelds/omap</span><br />
<span style="background-color: black; color: lime;">export MAKE_OPTS="-j4 ARCH=$ARCH CROSS_COMPILE=$CROSS_COMPILE"</span><br />
<span style="background-color: black; color: lime;"><br /></span>
<span style="background-color: black; color: lime;">pushd .</span><br />
<span style="background-color: black; color: lime;">cd ${OMAP_ROOT}/kernel_omapzoom</span><br />
<span style="background-color: black; color: lime;">make $MAKE_OPTS omap5uevm_defconfig</span><br />
<span style="background-color: black; color: lime;">make $MAKE_OPTS zImage</span><br />
<span style="background-color: black; color: lime;">popd</span><br />
<span style="background-color: black; color: lime;"><br /></span>
<span style="background-color: white;">Now, we need to compile the DTS (device tree source code) using the dtc tool. If you choose to use the usbboot instead of u-boot, you can enable the config option in kernel and simply append the DTB blob to the end of zImage.</span><br />
<span style="background-color: white;">(Boot Options -> Use appended device tree blob to zImage)</span><br />
<br />
<span style="background-color: purple; font-size: 12px;"><span style="color: yellow;">./scripts/dtc/dtc arch/arm/boot/dts/omap5-uevm.dts -o omap5-uevm.dtb -O dtb</span></span><br />
<span style="background-color: purple; color: yellow;">cat kernel_omapzoom/arch/arm/boot/zImage omap5-uevm.dtb > kernel</span><br />
<span style="background-color: purple; color: yellow;">./usbboot -f; fastboot -c "console=ttyO2 console=tty0 rootwait root=/dev/sda1" -b 0x83000000 boot kernel</span><br />
<div>
<br /></div>
<h3>
<span style="font-size: 12px;">VirtualOpen</span></h3>
For userspace part, I've followed the manual from VirtualOpenSystems for versatile express. The only tricky part was building qemu for the ArchLinux ARM host, and the guest binaries are available for download.<br />
<a href="http://www.virtualopensystems.com/media/kvm-resources/kvm-arm-guide.pdf">http://www.virtualopensystems.com/media/kvm-resources/kvm-arm-guide.pdf</a><br />
<br />
<br />
P.S. please share your stories on how you're using or plan to use virtualization on ARMI hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com0tag:blogger.com,1999:blog-1416969656167659289.post-13058365139191947812013-11-25T02:43:00.000+04:002013-11-25T02:47:51.473+04:00An update on OSX Touchscreen driverAfter playing with the HID interface in OS X, I have found out there exists an API for simulating input events from user space, so I've implemented the touchscreen driver using it.<br />
<br />
One unexpected caveat was that you need to set up an increasing event number to be able to click the menu bar. Even worse, when launched from XCode, the app would hang if you clicked the XCode menu bar. If you click any other app or launch it outside XCode, everything's fine. Caused some pain while debugging.<br />
<br />
Now I still want to implement the driver as a kernel module. On the other hand, the userspace version is also fine. I want to add the support for multitouch gestures, though I'm not sure it is possible to provide multiple pointers to applications.<br />
<br />
<iframe width="560" height="315" src="//www.youtube.com/embed/ccMOPNel-2k" frameborder="0" allowfullscreen></iframe><br />
<br />
The code is at github https://github.com/astarasikov/osxhidtouch<br />
<br />
You can get the binary OSX touchscreen driver for the Dell S2340T screen from the link below. Basically, you can replace the HidTouch binary with your own, and if you put it to "HidTouch.app/Contents/MacOS/HidTouch", you will get an app bundle that can be installed to Applications and added to the auto-start.<br />
https://drive.google.com/file/d/0B7wcN-tOkdeRRmdYQmhJSWsta1U/edit?usp=sharingI hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com0tag:blogger.com,1999:blog-1416969656167659289.post-83666336337666036772013-11-22T03:55:00.003+04:002013-11-22T03:59:42.445+04:00Multitouch touchscreen support in OS XHi there!<br />
<br />
I happen to have a Dell S2340T multitouch monitor (quite an expensive toy btw) which has a touch controller from 3M. It works fine in Windows (which I don't have any plans to use), sort of works in linux (which is my primary work environment) and does not work at all in OS X (not a big deal but it would be cool to have it).<br />
<br />
So I set out on the search for a driver and have figured out the following:<br />
<br />
<ul>
<li>There is some old driver from TouchBase that sucks. I mean, it's using some stinky installer that pollutes the OS X root file system, it has the UI from 90s. More than that, it's not signed and does not support my touchscreen. Even adding the USB Vendor ID to its Info.plist did not fix a thing</li>
<li>There is some new trial driver from TouchBase that should work for most devices but is limited to 100 touches (a demo version). Well, I didn't want to grovel before them and sign up at their stupid site. And still, even if I patched the driver to remove the trial limitations, I would have to sign it with my certificate and there would be no legal way for me to distribute it on the internetz</li>
<li>There is a full version of the TouchBase driver that costs $100. Are they fucking crazy? On the other hand, I would do the same. See, home users don't care, multitouch displays are very rare, and people are used to pirating sofrware. But one could raise quite some money selling the driver to the workshops that build customized Apple computers (like ModBook) or car PCs.</li>
<li>There are some drivers for other touchscreens, but they're for the old single-touch devices</li>
</ul>
<br />
Doesn't look promising. Now, one may wonder "WTF ain't it working out of the box? It's a HID device, should work everywhere". Well, there are two problems:<br />
<br />
<ul>
<li>Multitouch devices have the new HID event type (called HID usages) for the Digitizer class which the OS X does not support in IOKit. Actually, since it uses a new HID class (different to single-touch monitors), it is not even classified as a touchscreen by the OS (rather as a multitouch trackpad)</li>
<li>The touchscreen gets recognized by the generic HID driver as a mouse. And here comes the problem - when a finger is released, the device reports the pointer to be at the origin (point <0, 0>) which causes the cursor to be stuck at the top left corner</li>
</ul>
<div>
I decided to learn about the HID system in OS X and write a driver. I started with the sample code from the HID documentation and added some event debugging. I found out that the device reports the events both as a digitizer and as a pointing device. So for now I prepared a demo app in Qt5 that can recognize multiple touches and draw points in different colors when the screen is touched. Have a look at it in action:<br />
<br />
<iframe width="560" height="315" src="//www.youtube.com/embed/pXpa6xL5J0w" frameborder="0" allowfullscreen></iframe></div>
<div>
<br /></div>
<div>
The source code is at https://github.com/astarasikov/osx-multitouch-demo .</div>
<br />
<br />
Well, looks like I should figure out how to turn this into a device driver. I will probably need to figure out how to enumerate HID devices based on their class and make a kext file (Kernel EXTension) for XNU (the kernel used in OS X) and then.. sell my driver for some $20, right?I hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com0tag:blogger.com,1999:blog-1416969656167659289.post-18821409325655877042013-11-12T23:33:00.000+04:002013-11-12T23:37:09.749+04:00Using a hybrid graphics laptop on linux<h3>
Introduction.</h3>
I have a laptop with so-called hybrid graphics. That is, it has two GPUs - one of them is part of the SoC (Intel HD3000 GPU), the other one is the "discrete" PCIe Radeon HD 6630M from AMD. Typicaly, older models of dual-GPU laptops (and new Apple Macbook Pro machines) have a multiplexer that switches the output from the GPU to display. As the majority of modern laptops, my features a "muxless" combination of the GPUs - that is, the more powerful GPU does not have any physical connection to the display panel. Instead, it can write to the framebuffer memory of the less powerful GPU (or it can perform only the computational part of OpenGL/DirectX and let the primary GPU handle 2D drawing and blitting).<br />
<br />
Linux support for the hybrid graphics if far from perfect. To be honest, it just sucks even now that such kind of laptop have been dominating the market for some 4-5 years already. With the recent advent of the "DRI-PRIME" interface the linux kernel and userspace now supports using the secondary GPU for offloading intensive graphic workloads. Right now to utilize the separate GPU, an application has to be launched with a special environment variable (namely, "DRI_PRIME=1") and the system is supposed to dynamically power-up the GPU when it is needed and turn it off to reduce power consumption and heating at other times. Unfortunately, the support for power management is still deeply broken. In this post I summarize my findings and scripts which allow to turn off the external GPU permanently to save power and increase the battery life of the laptop. I am not using the secondary GPU because OpenGL and X11 drivers for the Intel GPU are more stable, and the open-source Radeon driver/mesa does not yet support OpenCL (the computational API for GPUs).<br />
<br />
<h3>
vga_switcheroo</h3>
The kernel interface for controlling the power for the external GPU is called "vga_switcheroo" and it allows to power-down the GPU. Unfortunately, I have found out that my laptop (and most others) enable the GPU after a suspend-resume cycle. I think this behaviour is intentional, because ACPI calls should be used to control the power of the GPU. However, it confuses the vga_switcheroo which thinks the card is still powered down. Meanwhile, the card drains some 10-30 watts of power, effectively reducing the laptop battery life from 7 hours to 2 hours or so.<br />
<br />
My first stab at this problem was a patch that forced the vga_switcheroo to disable all the cards that were not held by the userspace at resume. That did solve the problem for a while, but was hackish and never made it into the mainline kernel. However, it is still useful for linux kernels up to 3.9.<br />
<a href="http://lkml.indiana.edu/hypermail/linux/kernel/1204.3/02530.html">http://lkml.indiana.edu/hypermail/linux/kernel/1204.3/02530.html</a><br />
<br />
<h3>
kernel 3.10+</h3>
As of linux version 3.10, several changes were made in regards to the hybrid graphics power management. First is the introduction of the dynamic power management (in the form of clock control code and parsing the predefined power states provided by the OEM BIOS) for the Radeon chips. Second one is the change to the vga_switcheroo which <span style="background-color: purple; color: yellow;">allowed it to power down the card when unused</span> without the user interaction. It does work to some extent, but the problem with the card being powered on after a sleep-resume cycle remains.<br />
<br />
The problem is that now, <span style="background-color: purple; color: yellow;">when I manually disable the card via the vga_switcheroo, the PCI device is gone - it is removed and never appears again. The same behaviour could be exhibited on pre-3.10 kernels if one did issue the "remove" command to the DRM node</span> (/sys/class/drm/card1/). Besides, my hack to the vga_switcheroo stopped working since these upgrades. Now, this did not make me happy and I set out to figure out the solution.<br />
<br />
<h3>
acpi_call</h3>
<div>
Linux powers off the radeon GPU using the PCIe bridge and GPU PCIe registers. A "portable" and "official" way to control the power of a GPU is using an ACPI call. ACPI is an interface for the Intel-X86 based computers (although now also implemented for the ARM CPUs in an attempt to provide the support for UEFI, Windows RT and an unified linux kernel binary capable of running on any ARM SoC) intended to provide abstraction of hardware enumeration and power management. It contains tables with lists of PCI and other PNP (plug-and-play) peripherals which can be used by the OS kernel instead of the unsafe "probing" mechanism. Moreover, it contains a specification for the interpreted bytecode language. Some methods are implemented by the OEM inside the BIOS ACPI tables to perform certain functions - query battery status, power up- and down the devices etc.</div>
<div>
<br /></div>
<div>
Folks have since long figured out to use ACPI calls in linux to control the power of the discrete GPU. Although it could interfere with the vga_switcheroo interface, in case we either completely disable the external GPU or power it off via vga_switcheroo first, we're safe to use it. Moreover, <span style="background-color: purple;"><span style="color: yellow;">I have found out that I can use an ACPI call to power on the device and make it visible to the system after it is removed as described in the previous paragraph</span></span>!</div>
<div>
<br /></div>
<div>
There exist a module for the linux kernel that allows to perform arbitrary ACPI calls from the userspace. Turns out that a direct ACPI call can even work around the new vga_switcheroo. There's a good guide on installing the acpi_call module into the DKMS subsystem so that it is automatically packaged and built any time you upgrade the linux kernel on your machine.</div>
<div>
<a href="http://garyservin.wordpress.com/2012/01/06/disabling-discrete-gpu-in-debian-gnulinux-wheezy/">http://garyservin.wordpress.com/2012/01/06/disabling-discrete-gpu-in-debian-gnulinux-wheezy/</a></div>
<br />
The module contains the turn_off_gpu.sh script in the examples folder which can be used to power down the GPU. I ran it and took a notice of the method used for my laptop - it was the <span style="background-color: purple; color: yellow;">"</span><span style="font-size: 12px;"><span style="background-color: purple; color: yellow;">\_SB.PCI0.PEG0.PEGP._OFF"</span><span style="background-color: white;"> (which means South Bridge -> PCI controller 0 -> Pci Express Graphics -> Pci Express Graphics Port).</span></span><br />
<span style="font-size: 12px;"><span style="background-color: white;"><br /></span></span>
<span style="font-size: 12px;"><span style="background-color: white;">Now, I did the following magical trick:</span></span><br />
<span style="background-color: black; color: lime;">echo "\_SB.PCI0.PEG0.PEGP._ON" > /proc/acpi/call</span><br />
And BANG! after being gone when turned off via the vga_switcheroo, the GPU was identified by linux and enabled again. Neato.<br />
<br />
<span style="background-color: purple; color: yellow;">Putting it all together.</span><br />
Now the real showstopper is that the card still drains power after resume. I figured I had to write a script which would power down the card. It <span style="background-color: purple; color: yellow;">turned out that sometimes if the laptop was woken up and then it immediately went to sleep due to some race condition, systemd did not execute the resume hook.</span><br />
<br />
<span style="background-color: purple; color: yellow;">Instead, I used the udev</span>. Udev is a daemon that listens for the events that the kernel sends when hardware state changes. Each time the laptop wakes up, the power supply and battery drivers are reinitialized. Besides, the primary GPU also sends wakeup events. Another two observations are that I'm not using the discrete GPU and it is safe to call the ACPI "OFF" method multiple times. So, it is not a problem if we call the script too many times. I decided to just call it on any event from the two aforementioned subsystems. Here is how it can be done (note that you need to have root permissions to edit the files in /etc. If you are a newcomer to linux, you can become root by issuing a "sudo su" command):<br />
<br />
Create the file "/etc/udev/rules.d/10-battery-hook.rules" with the following contents:<br />
<span style="background-color: black; color: lime;">ACTION=="change", SUBSYSTEM=="power_supply", RUN+="/etc/gpu_poweroff.sh"</span><br />
<span style="background-color: black; color: lime;">ACTION=="change", SUBSYSTEM=="drm", RUN+="/etc/gpu_poweroff.sh"</span><br />
<div>
<br /></div>
Now, create the /etc/gpu_poweroff.sh script with the contents mentioned below. You can uncomment the "echo" calls to debug the script and verify it is getting called. By the way, the power_profile part is not necessary but is an example of how to put a radeon GPU into a low-power state without disabling it.<br />
<br />
<h3>
gpu_poweroff.sh</h3>
<span style="background-color: #f1c232;">#!/bin/bash</span><br />
<span style="background-color: #f1c232;"><br /></span>
<span style="background-color: #f1c232;">#this is a script for lowering the power consumption</span><br />
<span style="background-color: #f1c232;">#of a radeon GPUs in laptops. comment out whatever portion</span><br />
<span style="background-color: #f1c232;">#of it you don't need</span><br />
<span style="background-color: #f1c232;"><br /></span>
<span style="background-color: #f1c232;">#echo "gpu script @`date`" >> /tmp/foo</span><br />
<span style="background-color: #f1c232;"><br /></span>
<span style="background-color: #f1c232;">if [ -e /sys/class/drm/card0 ] ;then</span><br />
<span style="background-color: #f1c232;"><span class="Apple-tab-span" style="white-space: pre;"> </span>for i in /sys/class/drm/card*; do</span><br />
<span style="background-color: #f1c232;"><span class="Apple-tab-span" style="white-space: pre;"> </span>if [ -e $i/device/power_method ]; then</span><br />
<span style="background-color: #f1c232;"><span class="Apple-tab-span" style="white-space: pre;"> </span>echo profile > $i/device/power_method</span><br />
<span style="background-color: #f1c232;"><span class="Apple-tab-span" style="white-space: pre;"> </span>fi</span><br />
<span style="background-color: #f1c232;"><span class="Apple-tab-span" style="white-space: pre;"> </span>if [ -e $i/device/power_profile ]; then</span><br />
<span style="background-color: #f1c232;"><span class="Apple-tab-span" style="white-space: pre;"> </span>echo low > $i/device/power_profile</span><br />
<span style="background-color: #f1c232;"><span class="Apple-tab-span" style="white-space: pre;"> </span>fi</span><br />
<span style="background-color: #f1c232;"><span class="Apple-tab-span" style="white-space: pre;"> </span>done</span><br />
<span style="background-color: #f1c232;">fi</span><br />
<span style="background-color: #f1c232;"><br /></span>
<span style="background-color: #f1c232;">if [ -d /sys/kernel/debug/vgaswitcheroo ]; then</span><br />
<span style="background-color: #f1c232;"><span class="Apple-tab-span" style="white-space: pre;"> </span>echo OFF > /sys/kernel/debug/vgaswitcheroo/switch </span><br />
<span style="background-color: #f1c232;">fi</span><br />
<span style="background-color: #f1c232;"><br /></span>
<span style="background-color: #f1c232;">acpi_methods="</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.P0P1.VGA._OFF</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.P0P2.VGA._OFF</span><br />
<span style="background-color: #f1c232;">\_SB_.PCI0.OVGA.ATPX</span><br />
<span style="background-color: #f1c232;">\_SB_.PCI0.OVGA.XTPX</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.P0P3.PEGP._OFF</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.P0P2.PEGP._OFF</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.P0P1.PEGP._OFF</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.MXR0.MXM0._OFF</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.PEG1.GFX0._OFF</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.PEG0.GFX0.DOFF</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.PEG1.GFX0.DOFF</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.PEG0.PEGP._OFF</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.XVR0.Z01I.DGOF</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.PEGR.GFX0._OFF</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.PEG.VID._OFF</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.PEG0.VID._OFF</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.P0P2.DGPU._OFF</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.P0P4.DGPU.DOFF</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.IXVE.IGPU.DGOF</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.RP00.VGA._PS3</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.RP00.VGA.P3MO</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.GFX0.DSM._T_0</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.LPC.EC.PUBS._OFF</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.P0P2.NVID._OFF</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.P0P2.VGA.PX02</span><br />
<span style="background-color: #f1c232;">\_SB_.PCI0.PEGP.DGFX._OFF</span><br />
<span style="background-color: #f1c232;">\_SB_.PCI0.VGA.PX02</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.PEG0.PEGP.SGOF</span><br />
<span style="background-color: #f1c232;">\_SB.PCI0.AGP.VGA.PX02</span><br />
<span style="background-color: #f1c232;">"</span><br />
<span style="background-color: #f1c232;"><br /></span>
<span style="background-color: #f1c232;"># turn off the dGPU via an ACPI call</span><br />
<span style="background-color: #f1c232;">if [ -e /proc/acpi/call ]; then</span><br />
<span style="background-color: #f1c232;"><span class="Apple-tab-span" style="white-space: pre;"> </span>for i in $acpi_methods; do</span><br />
<span style="background-color: #f1c232;"><span class="Apple-tab-span" style="white-space: pre;"> </span>echo $i > /proc/acpi/call</span><br />
<span style="background-color: #f1c232;"><span class="Apple-tab-span" style="white-space: pre;"> </span>done</span><br />
<span style="background-color: #f1c232;"><span class="Apple-tab-span" style="white-space: pre;"> </span>#echo "turned gpu off @`date`" >> /tmp/foo</span><br />
<span style="background-color: #f1c232;">fi</span><br />
<span style="background-color: #f1c232;"><br /></span>
<span style="background-color: #f1c232;">exit 0</span><br />
<br />
Make it executable by issuing a <span style="background-color: black;"><span style="color: lime;">"chmod +x /etc/gpu_poweroff.sh"</span></span> command.<br />
Also, take a look at the "/etc/rc.local" script. If it does not exist, simply create it with the following contents and make it executable:<br />
<span style="background-color: #f1c232;"><br /></span>
<span style="background-color: #f1c232;">#!/bin/sh -e</span><br />
<div>
<div>
<span style="background-color: #f1c232;">bash /etc/gpu_poweroff.sh</span></div>
</div>
<div>
<div>
<span style="background-color: #f1c232;">exit 0</span></div>
</div>
<div>
<br /></div>
If it does exist, insert the call to the "/etc/gpu_poweroff.sh" before the "exit" line.<br />
<h3>
Bonus</h3>
To silence the fans on the Sony Vaio laptop, you can add the following to your rc.local script (I advise against putting it to the gpu_poweroff script because this command has a noticeable delay):<br />
<br />
<span style="background-color: #f1c232;">if [ -e /sys/devices/platform/sony-laptop/thermal_control ]; then</span><br />
<span style="background-color: #f1c232;"><span class="Apple-tab-span" style="white-space: pre;"> </span>echo silent > /sys/devices/platform/sony-laptop/thermal_control</span><br />
<span style="background-color: #f1c232;">fi</span>I hate softwarehttp://www.blogger.com/profile/10541307407645458323noreply@blogger.com0