Doing it wrong: application preferences

One thing that's been always bothering me is the lack of a unified way to keep persistent settings.

When you're installing a shiny new piece of software, you never know which surprises it brings. The location and syntax of the configuration files varies between applications and environments. Let's take a look at the most commonly used persistent storage formats and analyze their strong and weak points. Note that I only want to discuss the issue only in the aspect of desktop usage, where usability and maintainability should not be sacrificed over performance as we do in the embedded world. Also, note that I'm mainly focusing on the formats used in the desktop *NIX distributions because that's what I'm mostly using.

One key difference between many formats is whether they allow text data with arbitrary formatting and spacing or have a fixed binary layout. The former is obviously human-readable (and sometimes, editable), the latter can in most cases be easier parsed by the machine.

Plain text
Pro:

• Human-readable

Contra:

• No unified syntax (everyone invents their own)
• No way to contol the correctness of the file contents (as opposed to xml schema for which there exists a multitude of tools and libraries)

XML
+

• Can be edited by a human (to some extent)
• Strict structure definition is a part of a standard (user can define custom types and limits for variable values)
• Supported by many tools

-

• Hard to read by a human
• Inefficient to parse (though, better than a plain text without schema)

GConf is an XML-based configuration system for the GNOME and GTK applications. Basically, it inherits all the pluses and minuses of the XML.

JSON
+

• Easy to read and edit by a human
• Easy API in most libraries (we can think of it either as of a tree of objects or as hashmap of hashmaps of hashmaps...)

-

• Lack of built-in schema checking (i.e., it has a formal syntax and we can check that a JSON is valid, but we cannot verify that the data structures are initialized with correct values)

Now, let's discuss binary formats.

Custom formats

Windows Registry
+

• It's fast.

-

• Most system-critical data is kept inside a single file (well, actually around 6 files in %windir%\System32\config). A single failure means the system is unable to boot.
• All applications store their data inside a single database. If an application is running with 'Administrator' (something like root, but for retards), it can easily modify or read the data of other applications. This is a huge security issue.
• Application activity is not logged (at least, by default). It means mischievous software leaves trails even after it is uninstalled which is especially true of shareware and other crapware.
• Application and system registry are not isolated. You cannot just reinstall the system without keeping all preferences (like, keep the home dir but replace the rootfs in *NIX)
• There are only several basic data types (DWORD, Text, Binary) and the user cannot introduce their own

DConf is a replacement for GConf from the GNOME project. The only real difference is using a binary format instead of XML. While it is arguably faster, it is completely incomprehensible to a human-being and when things go bad, I have no idea what to do.

BSON is a binary version of JSON.
+

• It is faster
• It consumes less space

-

• It introduces custom types (like UUID) which means that any JSON can be converted to BSON, but not the other way round.

MessagePack is a binary serialization format that is aiming to be compatible with BSON while being optimized for network usage
+

• It offers high compression ratio reducing network traffic
• Comes with the IDL/serialization code for most mainstream languages (C++, ruby, python)
• Offers type checking when used with statically typed languages
• Comes with an RPC implementation which can be useful for network applications

-

• Performance can theoretically be a bit lower than BSON due to bit-packing 

There of course exists a multitude of other config formats, but the most used on a linux destkop are plain text and GConf. Therefore, I'm lazy to describe every other thing under the sun.

Now, here is what I think application developers should do

• Use glib/qt4/java built-in APIs for keeping preferences. Don't reinvent the wheel
• Don't break the config format with time (ideally, you don't know anything about the config format if you use the existing API)
• Make it easy for the users to backup the configuration data

Ideally, I would like that all apps (at least, all the free and open-source apps on linux) would use the same API for keeping preferences so that the system administrator can choose various backends, be it xml, json, sql or what not without modifying the applications.

Why I adore Java NOT.

Ok, here are some of my thoughts about Java as both a language and a development platform.

TL;DR: when all you have is a hammer, everything looks like a nail. Don't try shoving java up every task. Even better, don't use it at all.

Let me warn you: I am fully aware that Java was created by enterprise consultants to allow building enterprise bullshit quickly and to make sure you don't have to make efforts to support your crap once written. I fully recognize the importance of Java language as such, but any claims to use Java as a general-purpose language make me cry.

The language and the JVM:
Ok, first of all, Java language is very dull and unpleasant to program in. And it sucks hard as an OO language.

1. It has both objectified and non-objectified POD. That is, it has both int and Integer. And this leads to several interesting consequences. You can't just call a toString() method on an integer. like "1.toString()". Or, well, here's a better example. Try it out yourself in java or BeanShell


$ cat Foo.java 

class Foo {

static Integer a = 9999;

static Integer b = 9999;

static Integer c = 9;

static Integer d = 9;

public static void main(String args[]) {

System.out.println(a == b);

System.out.println(c == d);

}

}

$ java Foo

false

true

Do you find it logical? Well, if you've been programming in java for a while, you may be braindamaged enough to think it's OK, but it is not. In fact, the behaviour of a complex system must be predictable without reading through all the dark corners of documentation. (for those who are still living in the bliss of ignorance: the behaviour observed is because JVM keeps cached copies of Integers in the range [-127, 128])

Ok, you may argue that it's a nice idea performance-wise, but it is not. Proper VMs and languages (CLR/C#, ocaml) have found better ways around this problem. For example, CLR allows to directly work with the raw untyped memory and all numeric types actually consume just the amount of bits needed to keep precision while all 'object' methods are implemented as syntactic sugar - i.e., compiler voodoo)

2. Generic types just suck. The reason is that in order to keep compatibility with the ancient crap, generics were introduced as a syntactic sugar. Once you compile your 'generic' code, type information is erased. Therefore, you lose type safety when using reflection - you can't determine the real type of a generic type variable if the concrete type is a class inheriting from multiple interfaces. Moreover, type erasure is the same precise reason why you can't create an array of generic types.

3. The language has no lambda closures. Really, creating tons of one-function interfaces like Runnable is just a pain. Even more pain is declaring an interface just to pass a single closure.

4. No on-stack memory allocation. This is a problem for applications using little amounts of memory, because all memory, even for locals, has to be heap-allocated. This raises several performance problems (of course, it ends up being placed in stack or registers, but there's no explicit way to state that).

5. There's no proper way to interact with native code and untyped data. For example, the already mentioned .Net has the p/invoke interface which allows to conveniently call into native libraries without modifying them, without writing a C stub calling into the VM, without additional marshalling overhead. And you even have the same flexibility when specifying structure layout as you would in C.

[StructLayout(LayoutKind.Explicit, Size=8)]
public struct Foo
{
    [FieldOffset(0)]
    public byte A;
    [FieldOffset(1)]
    public int B;
    [FieldOffset(5)]
    public short C;
    [FieldOffset(7)]
    public byte D;
}
[DllImport("Foo.dll", EntryPoint="TransmitToHell")]
public static extern int SendToLimbo(struct Foo foo);

6. No pointer arithmetic. C# does have that. While one may argue that it is unsafe, in a managed language running in an environment with virtual memory, it is possible to check boundary crossing and prevent typical C buffer overflow and stack smashing vulnerabilities while still delivering flexibility.

7. No type inference. Seriously, what the fuck is that?
CoolFoo coolFoo = new CoolFoo();
why couldn't I just write
let coolFoo = new CoolFoo();?
The compiler can clearly deduce the variable type here. And proper languages (ML, Haskell) can actually type-check the whole program without explicitely specifying types in most cases due to Hindley-Milner type inference.

8. Badly programmed standard library violating OO principles. Javatards like to claim their language is fully OO, but violate the principle multiple times. The most obvious cases:

• Empty interfaces (like, Cloneable). Like, they ain't doing anything. Well, they should've not been doing anything, but then reflection voodoo comes in.
• Unmutable mutable data. You all know that the final modifier makes a memory location immutable. But not all people know that using reflection you can obtain write access to it. Even standard streams (System.out, System.in) can be reassigned using System.setOut, System.setIn calls. R U MAD?

The aforementioned factors make programming in java a miserable exercise in futility when it comes to integrating with native code or writing algorithms that have to crunch large arrays of numbers (DSP). One interesting example is the Android platform where most high-level Java code is just a set around native libraries written mainly in C and DSP assembly extensions (NEON, SSE, AVX). The lack of built-in implementations of many DSP routines (convolution, FFT) and the buggy implementation of the RenderScript library make writing high-performance video processing applications an extremely daunting task.

Enterprise crap:

• Counter-intuitive poorly documented frameworks. Indeed, do you know anyone who could figure out Spring or Hibernate without digging through tutorial blog posts?
• XML, XML everywhere. Completely human-unfriendly and resource-consuming. There are better formats out there - JSON which is kind of human-readable and BSON/MsgPack when speed matters.
• AbstractFactoryFactoryImpl and so on. Design patterns are a real cargo cult of java pseudoengineering

So, to sum up. If you want to set up yet another enterprise-level megaproject and drain all the money of a local enterprise, go ahead and use Java. But please don't make me have any business with java ever again.

x86 vs arm. lies, blatant lies and standards

I wanted to publish a rant about x86 vs ARM architecture long ago but had no time to really do so. Therefore, I'm just publishing a slightly edited draft written back in spring.

In short: x86 sucks.

The last two decades of personal computing were dominated by the x86 architecture, while embedded hardware, like cell phones, media players and microwave ovens traditionally used simpler and less power-hungry designs, namely, ARM, MIPS and SH3 architectures.

Ever since first 'smart' devices (PDAs/phones where you could install your own custom applications) have appeared, it has been a dream of many people to reduce the gap in both computing power and abilities of these devices and desktop computers. There have been a lot of community projects to run full-blown linux/BSD distros on handheld devices, but it was not until recently that the major players like M$ and Apple turned their attention to the non-x86 architectures. 2012 is going to see the end of the world appearance of a lot of ARM-based laptops, since Windows 8 is going to be ported to this architecture.

Migration to (or, rather, adoption of) a new architecture has caused a lot of FUD and confusion. Let's just consider some common myths about x86 vs non-x86 debate and analyze the strong and weak points of both camps.

Myth 1: x86 has a huge software base and you cannot replace it.
Reality: most commercial vendors (including M$, Adobe etc) are porting or have already ported their software to the new architecture. Open-source software, because of the efforst of major distributions like Debian, is already ported to virtually anything that can compute.

Of course, there are tons of legacy software, for which even source code may be lost. But that's just because someone was stupid enough to make their software closed-source from the start. Even though this appeals to managers, from an engineering point of view closed source code is the worst sin because it reduces interoperability with other pieces of software, causes a lot of problems when API/ABI breaks (at major updates) and of course you cannot trust the binaries when you don't see the code.

One approach to the problem of legacy software is what the industry has been doing for ages - virtualization. Hardware emulation, to be more precise. While it reduces performance by multiple orders of magnitude, it may be a savior in some extreme cases. For example, when you need your ancient business app once in two months, but want to enjoy your cool and light ARM laptop the rest of the time.

Myth 2: x86 is standardized. Even without vendor support, a generic set of drivers works anywhere. ARM, where is your ACPI?
Reality: The reality is that even though standards exist, they are rarely implemented properly. Even on x86 things like ACPI, hybrid graphics, function keys are often broken and non-standard which means chances are your laptop will not work until you install the customized set of drivers from the OEM.  Or will work but drain battery like crazy.

Myth 3: ARM will never be as fast as X86. While this is true that modern ARM SoCs are typically lag almost a decade behind X86 in terms of processing power, it is possible to build a high-performance RISC SoC. The problem is that increasing performance by the means of reducing die size will raise the power exponentially. Emulating an X86 CISC on a RISC core increases the complexity of the CPU and therefore reduces performance per watt. The reality is that ARM CPUs are mainly utilized in portable hardware, e.g., laptops, smartphones, tablets, where power saving is preferred over performance.

The real problem with X86 is that it is a collective term used to refer to a whole bunch of hardware dating back to calculators from the prehistoric area to the newest 'ivy bridge' systems. One notable problem of this architecture is the presence of three incompatible execution modes: 16-bit real mode, 32-bit protected mode, 64-bit long mode. Legacy software tends to be a mix of these and involves context switches which increase latency and potentially reduce the level of security.

While X86 is ridden by a multitude of remnants of the past, it still has advantages over ARM/MIPS typically used in embedded hardware:

• Harware virtualziation support (hypervisor). ARM, for example, is yet to release the Cortex A15 core and armv8 instruction set adding virtualization extensions. While it is possible to use the TrustZone hypervisor to virtualize hardware, it should be noticed that there are absolutely no standards describing what APIs must a TZ OS provide, and TrustZone is designed to protect the user from tampering with their hardware and not to protect the user's data from misbehaving execution environment.
• Standardized BIOS/EFI support for common hardware: VESA framebuffer, SATA/EHCI/XHCI controllers, i8042 keyboard etc. While these interfaces typically do not allow to use the advanced computational and power-saving capabilities of the hardware, they allow to get a custom kernel running on a new board lacking documentation from the hardware vendor.

So, the question is: Is migration to a new architecture needed?
The answer is: yes and no.

What is needed is adapting software to the new computing paradigm. 80-s and 90-s were dominated by the impression that CPU frequency can be increased indefinitely and RAM is expensive. Recently, however, RAM has become very affordable and CPUs are hitting the barrier of frequency due to the technology used and powersaving constraints (well, there are some alternatives like using optical interconnects or returning back to using germanium for semiconductors, but we'll eventually hit the bottleneck once again in a couple of years). Modern computing involves assymetric multiprocessor systems with heterogenous architectures and using vector processors (SSE, NEON, CUDA) for exploiting data-level parallelism.