arighi's blog

install busybox from source on Samsung GT-I9100

2011-08-30T11:52:00.000+02:00

If you want a complete set of unix tools in your phone, here are the steps to cross-compile and install busybox from source (there's no need to root the phone and/or install everything using the busybox installer from the market).

Download the latest ARM gnueabi toolchain from codesourcery.

Get the busybox source code from the git repository:
$ git clone git://busybox.net/busybox.git

Copy my busybox config file (or use your own config if you prefer, this can be just a starting point):

$ cd busybox
$ wget -O .config http://www.develer.com/~arighi/android/busybox/config

[optional] Change the busybox config if you want by running:
$ make menuconfig

Cross-compile busybox:
$ make oldconfig && make

At the end of the build the busybox binary should be available as a statically linked ELF for ARM:
$ file busybox
busybox: ELF 32-bit LSB executable, ARM, version 1 (SYSV), statically linked, for GNU/Linux 2.6.16, stripped

Upload the busybox binary to the device (you don't need root access to your phone):
$ adb push busybox /data/local/tmp/

Now busybox it's ready to be used:
$ adb shell /data/local/tmp/busybox CMD

Example:
$ adb shell /data/local/tmp/busybox lsusb
Bus 001 Device 001: ID 1d6b:0002
Bus 001 Device 002: ID 1519:0020

=== NOTE: all the following steps are optional ===

If you have enabled the adb root shell access to your phone (i.e., by rooting the phone or by installing my custom kernel), you can also install busybox in the /system partition and have all the commands available in the $PATH.

Remount the /system partition in read-write mode on your phone:
$ adb shell "mount -oremount,rw /dev/block/mmcblk0p9 /system

Upload the busybox binary to the /system partition:
$ adb push busybox /system/xbin/busybox

Install busybox by using busybox itself:
$ adb shell "chmod 755 /system/xbin/busybox"
$ adb shell "/system/xbin/busybox --install -s /system/xbin"

Remount /system in read-only:
$ adb shell "mount -oremount,ro /dev/block/mmcblk0p9 /system"

Now all the busybox applets are in your $PATH:

$ adb shell lsusb
Bus 001 Device 001: ID 1d6b:0002
Bus 001 Device 002: ID 1519:0020

HOWTO: custom kernel on Samsung Galaxy S II I9100

2011-08-30T00:33:00.001+02:00

Recently I've replaced my Bravo HTC Desire with a new Android phone: a Samsung Galaxy S II I-9100. I couldn't resist too much with the stock kernel, so finally I've found some spare time to cook a custom kernel starting from the original Samsung kernel source code GT-I9100_OpenSource_Update2.

In this post I report (for me to remember later and for those who are interested) all the steps that I did to setup the build environment, cross compile the custom kernel and flash it into the phone.

DISCLAIMER: I take no responsibility for anything that may go wrong by you following these instructions. Proceed at your own risk!

=== Requirements ===

- A Samsung Galaxy S II (not necessarily rooted! you'll get a root shell when you'll flash the new kernel)
- The latest android SDK
- The arm-none-eabi cross-compile toolchain (you can get it from the CodeSourcery website)

=== HOWTO ===

Download and install the arm toolchain from the CodeSourcery website: be sure that arm-none-eabi-gcc is in your $PATH.

Get the autobuild script:
$ git clone git://github.com/arighi/gt-i9100.git

Run the script:
$ ./build-kernel.sh

The script downloads the "-arighi" kernel source code, a initramfs template and builds a new kernel ready to be flashed into the device.

At the end of the autobuild process the file kernel-gt-i9100-arighi.tar can be used to flash the new kernel to the phone using Odin (search on the web or in the xda-developers forum, there are tons of howtos/tutorials for this).

=== Results ===

The score with Quadrant benchmark is not bad at all, I got always > 4000, but remember that we're cheating during the IO test, due to the fsync-disable patch.

Anyway overall result looks good enough.

== Additional notes ===

- All the custom *.ko files are included into the initramfs to avoid potential errors/problems with the original kernel modules (so it is always possible to flash back the original kernel later, all the old kernel modules are still there, untouched).

- After you've flashed the -arighi kernel the first time you will also have root access to your device. The initramfs template enables adb root shell (ro.secure == 0), so an adb shell will immediately drop you to a root shell. This means you can also re-flash your device from Linux directly using the flash-kernel.sh script.

- For the complete list of all the patches applied to this kernel have a look at the git log here.

- IMPORTANT: the fsync-disable patch (enabled in the kernel by default) can increase performance and battery life, but it is dangerous, because it might eat your data!! It makes the software no longer crash safe, so if you start to randomly kill your apps you may lose some data

[UPDATE: the fsync-disable patch is no more enabled by default in the kernel, to enable it just set CONFIG_FILE_SYNC_DISABLE=y in the kernel .config)]

Android: automated per-uid task group

2011-01-20T00:09:00.005+01:00

Android is a privilege-separated operating system, in which each application runs with a distinct system identity: the Linux user ID (uid).

With this patch (sched-automated-per-uid-task-groups.patch) the kernel automatically creates a distinct task group for each uid (when a process calls set_user()) and places all the tasks that belong to a single uid into the same task group. In this way each application can get a fair amount of the CPU bandwidth (guaranteed by the CFS scheduler), independently by the number of task/threads spawned.

The patch is against the CyanogenMod's 6.1.1 kernel (2.6.35.10) and I tested it successfully on my HTC Desire (Bravo).

I've used the following testcase:

- run 4 cpu hogs in background as user app_35 (com.android.email in my case):

# su - app_35

$ for i in `seq 4`; do yes >/dev/null & done

- run the Quadrant benchmark in parallel and measure the result with and without the patch applied

Without the patch (output of top):

PID PPID USER STAT VSZ %MEM CPU %CPU COMMAND

6533 123 10070 R 202m 48.6 0 20.0 com.aurorasoftworks.quadrant.ui.st

6506 1 10035 R 1128 0.2 0 20.0 yes

6507 1 10035 R 1128 0.2 0 20.0 yes

6508 1 10035 R 1128 0.2 0 20.0 yes

6509 1 10035 R 1128 0.2 0 20.0 yes

Benchmark result: 676

uid 10035 (cpu hog) : 60.0 % cpu quota

uid 10070 (benchmark): 20.0 % cpu quota

With automated per-uid task group (output of top):

PID PPID USER STAT VSZ %MEM CPU %CPU COMMAND

6784 123 10070 S 209m 51.4 0 50.0 com.aurorasoftworks.quadrant.ui.st

6852 1 10035 R 1128 0.2 0 12.5 yes

6853 1 10035 R 1128 0.2 0 12.5 yes

6854 1 10035 R 1128 0.2 0 12.5 yes

6855 1 10035 R 1128 0.2 0 12.5 yes

Benchmark result: 816

uid 10035 (cpu hog) : 50.0 % cpu quota

uid 10070 (benchmark): 50.0 % cpu quota

Total speedup: ~1.2 (the benchmark is about 20% faster in this case), because in the last case the benchmark gets ~50% of the CPU time and in the other case it gets only ~20%, despite the fact that there are 2 "pair" applications that should be correctly considered as equal from the user's perspective.

This patch is based on the patch "sched: automated per tty task groups" by Mike Galbraith.

HOWTO: install a custom kernel on HTC Desire

2011-01-10T23:28:00.001+01:00

=== Disclaimer ===

I take no responsibility for anything that may go wrong by you following these instructions. Proceed at your own risk!

I tested this howto with a Bravo HTC Desire, rooted with Unrevoked 3.22.

=== Requirements ===

- A rooted HTC Desire (Bravo)

- The android SDK + NDK (to get the cross-compile toolchain):

- The latest HTC Desire kernel (choose "HTC Desire - Froyo MR - 2.6.32 kernel source code")

- The koush's AnyKernel template (to generate the update.zip at the end of the build process)

=== HOWTO ===

- Prepare the cross-compiler environment (replace /opt/android with the path where you have installed the Andorid NDK):

$ export PATH=/opt/android/android-ndk-r5/toolchains/arm-eabi-4.4.0/prebuilt/linux-x86/bin:$PATH

At this point arm-eabi-gcc, as well as other binutils and compiler binaries, should be in your $PATH.

- Untar the kernel

- save the kernel config (if you want to restore the original kernel config):
$ adb pull /proc/config.gz

- [optional] Apply the following patches to the kernel:
   - 0001-sync-disable-fsync-fdatasync-sync_file_range-syscall
   - 0002-writeback-change-default-dirty-memory-settings
   - 0003-sched-replace-CFS-with-the-BFS-scheduler

- [optional] Take my kernel configuration:
$ wget -O bravo-2.6.32-gd96f2c0/.config http://www.develer.com/~arighi/android/linux/config

Or use the previously saved config.gz either.

- Build the kernel:
$ cd bravo-2.6.32-gd96f2c0/
$ make ARCH=arm CROSS_COMPILE=arm-eabi- oldconfig
$ make ARCH=arm CROSS_COMPILE=arm-eabi-

Now you should find the fresh new kernel, ready to be flashed on your HTC Desire, in arch/arm/boot/zImage.

- Apply the following patch to the koush's AnyKernel source (to fix a syntax error when trying to flash update.zip from ClockworkMod):
   - 0001-updater-script-specify-the-mount-options-for-the-sys

- Replace the zImage in the AnyKernel template with your zImage:
$ cp bravo-2.6.32-gd96f2c0/arch/arm/boot/zImage AnyKernel/kernel/zImage

- Go back to the template directory (you will see three subdirectories: META-INF, kernel & system) and generate the update.zip:
$ zip -r ../update.zip *

- Connect your phone via a USB cable (be sure to turn on USB debugging on your phone: Settings -> Applications -> Development -> USB debugging)

- Push update.zip and the wireless module to the SD card of your phone:
$ adb push update.zip /sdcard/update.zip
$ adb push bravo-2.6.32-gd96f2c0/drivers/net/wireless/bcm4329_204/bcm4329.ko /sdcard/bcm4329.ko

- Reboot your phone in ClockworkMod recovery (power-on while holding Volume down key and select RECOVERY)

- In ClockworkMod select "apply sdcard:update.zip" (confirm: choose Yes)

- Reboot your phone via "reboot system now"

At this point your custom new kernel should boot.

=== Fix the bcm4329 wireless module loading without S-OFF ===

The bcm4329.ko module can't be properly overwritten in the /system partition without S-OFF the device and so give access in read-write to the /system partition. However, we can enforce the usage of our new module binding any other writable directory to /system/lib/modules (i.e., /data/local).

Prerequisites:
- the latest ARM toolchain downloadable from the CodeSourcery site

Download and install the ARM toolchain and be sure that arm-none-linux-gnueabi-gcc is in your $PATH.

- get the latest version of busybox from git (or download a recent stable version if you prefer):
$ git clone git://busybox.net/busybox.git

Howto:
- use my busybox config file:
$ cd busybox
$ wget -O .config http://www.develer.com/~arighi/android/busybox/config

- cross-compile busybox:
$ make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- oldconfig
$ make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi-

- copy the busybox binary into the /data/local directory on your phone:
$ adb push busybox /data/local/busybox

- bind the directory /system/lib/modules with /data/local:
$ adb shell
$ su
# cat /sdcard/bcm4329.ko > /data/local/bcm4329.ko
# /data/local/busybox mount --bind /data/local /system/lib/modules

After this trick go on your phone check Settings -> Wireless & networks -> Wi-Fi. The wireless connection should start normally.

=== Results ===

Here is my score with this kernel using the Quadrant benchmark: 1370!

Linux kernel hacking: file notification system with kernel tracepoints

2009-09-11T18:53:00.000+02:00

An example of how to use kernel tracepoints to create a simple real-time backup / file change notification system. The article (only in italian, sorry) is available at stacktrace.it.

mutt + gmail notifier

2009-06-26T14:30:00.005+02:00

I really enjoy the power of mutt, and I've to say that I've not too far from reaching the email Nirvana with it :). OK, it's not the email client for everybody, it's just for the people that prefer the keyboard to the mouse and love the command line interfaces.

There's only one missing feature in mutt: a nice way to notifiy new emails. The problem with mutt is that I need to periodically switch to the mutt shell to check for new emails. And I don't even like the crappy notification balloons that cover the useful part of the desktop (e.g., thunderbird). A small tray icon could be a solution (and I did it this way for a while, patching mutt), but with the icon I don't see immediately the message that I receive.

This led me to notice a large unused area in the gnome panel at the top (recently I moved from Fluxbox to Gnome, yeah! :) now I've a ultra-very-fast SSD I can also use a fancy desktop environment). So, why not to use the top panel to notify the subject of the last email I received in my mailbox? ta-da! the solution: a small python gnome applet that periodically fetches the last unread email from a generic IMAPS folder in gmail and prints the subject to the panel.

Here's the code: gmail-check, very minimalist and designed just for my particular desktop environment, so it may not work in some cases...

New SSD disk

2009-06-08T13:58:00.002+02:00

I just got one new SSD disk MTRON MOBI 3000 for my Dell Latitude D430 notebook. It's very small, only 32GB, but it definitely ROCKS!!! I can boot in about 12 seconds, without any deep tuning of the kernel and boot services, but the _responsiveness_ is the most relevant thing, apart the read/write 100MB/s throughput (that is not so important for a desktop system). The impressive part is the ~5500 iops (IO operations per second) obtained using a workload of 4KB random reads/writes!

Linux kernel hacking: process containers

2009-05-14T23:39:00.003+02:00

A basic overview of the Linux cgroups. My article is available at stacktrace.it (in italian).

battery life in bash prompt

2009-05-02T16:00:00.004+02:00

I've just reconfigured my .bashrc to execute this bash script that allows to show the perentage of battery life at the beginning of the command prompt. Geeze, really nice! :) At this point I can turn off the guidance-power-manager applet and enjoy a faster boot.


#!/bin/bash
GRAY="1;30"
CYAN="0;36"
LIGHT_CYAN="1;36"
LIGHT_BLUE="1;34"
YELLOW="1;33"
WHITE="0;1"
NO_COLOR="0"
LIGHT_RED="1;31"
LIGHT_GREEN="1;32"
BROWN="0;33"

function battery_info()
{
    BATT_INFO=$(acpi -b | awk -F', ' '{print $2}')
    AC_INFO=$(acpi -aB | awk -F': ' '{print $2}')

    if [ $AC_INFO = "off-line" ]; then
        BATT_PERC=${BATT_INFO:0:${#BATT_INFO}-1}

        if [ $BATT_PERC -ge 75 ]; then
            COLOR=$LIGHT_GREEN
        elif [ $BATT_PERC -le 25 ]; then
            COLOR=$LIGHT_RED
        else
            COLOR=$YELLOW
        fi
    else
        COLOR=$NO_COLOR
    fi
}

PROMPT_COMMAND=battery_info
PS1="\[\033[\$(echo -n \$COLOR)m\]\$(echo -n \$BATT_INFO)\
\[\033[${NO_COLOR}m\] \u@\h:\[\033[${WHITE}m\]\w\[\033[${NO_COLOR}m\]\$ "

iozone: buffer overflow in ubuntu jaunty

2009-04-28T23:24:00.003+02:00

In the latest Ubuntu Jaunty iozone immediately crashes with a nice *** buffer overflow detected *** message, that means it is practically unusable. Fortunately the cause of the bug is very simple: a wrong length used to copy a string by gethostname(). I posted a fix here.

cgroup: io-throttle controller (v13)

2009-04-16T17:17:00.002+02:00

A new version of my IO controller for Linux cgroups.

LWN.net coverage at http://lwn.net/Articles/328484/.

Performance improvement of parallel applications with user-space spinlocks

2009-03-10T16:22:00.020+01:00

Mutex locks provided by the operating system are expensive to create and acquire and a high contention can dramatically reduce or even eliminate the advantage of parallelism.

GCC provides some atomic operations, directly mapped to the atomic instructions actually provided by the underlying hardware. These are the same instructions that are used to implement the traditional locking primitives (mutexes), but they can be used to implement user-space locking primitives and save a significant amount of overhead.

However, this is not always the best solution: user-space spinlocks have their own drawbacks and overheads. Spinlocks are good when another CPU has the lock and it is likely to release it as soon as possible. In general, where there are more threads than physical CPUs, a spinlock simply wastes the CPU, until the OS decides to preempt it. Moreover, with spinlocks, the CPUs spins (really? :)) forever trying to acquire the lock. This leads to more power consumption and more heat to be dissipated, so this could be a really _bad_ solution for many embedded / ultra-portable devices (this is the same reason because asynchronous interrupt-driven handlers are better than polling for embedded devices). [ Actually, there are also some memory barrier and out-of-order execution troubles, but we will not discuss about this topic now... maybe I'll report some details in another post... ].

Anyway, in the following example we will see a typical problem where the usage of user-space spinlocks can bring evident benefits for performance (however, there is still the power consumption issue, but we don't care about it in this case).

Setting the USER_SPINLOCK option in the Makefile it is possible to choose between (0) the traditional pthread_mutex primitives and (1) custom user-space spinlock implementation.

The problem is the classical dining philosopher problem.


=== Makefile ===

N_CPUS=$(shell getconf _NPROCESSORS_ONLN)
CACHELINE_SIZE=$(shell getconf LEVEL1_DCACHE_LINESIZE)

USER_SPINLOCK=1

TARGET=userspace-spinlock

all:
    gcc -g -O3 -lpthread -o$(TARGET) \
        -DN_CPUS=$(N_CPUS) \
        -DCACHELINE_SIZE=$(CACHELINE_SIZE) \
        -DUSER_SPINLOCK=$(USER_SPINLOCK) \
        $(TARGET).c

clean:
    rm -f $(TARGET)

=== userspace-spinlock.c ===

#define _GNU_SOURCE

#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <sched.h>
#include <errno.h>
#include <time.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/syscall.h>

static pthread_t threads[N_CPUS];
static int input[N_CPUS];

#if USER_SPINLOCK
static int shared[N_CPUS * CACHELINE_SIZE];

static inline void lock(int *l)
{
    while (__sync_lock_test_and_set(l, 1));
}

static inline void unlock(int *l)
{
    *l = 0;

}
#else /* USER_SPINLOCK */
static pthread_mutex_t shared[N_CPUS * CACHELINE_SIZE];

static inline void lock(pthread_mutex_t *l)
{
    pthread_mutex_lock(l);
}

static inline void unlock(pthread_mutex_t *l)
{
    pthread_mutex_unlock(l);
}
#endif /* USER_SPINLOCK */

/*
 * From GETTID(2):
 *
 *   Glibc does not provide a wrapper for this system call; call it using
 *   syscall(2).
 *
 */
static inline pid_t gettid(void)
{
    return syscall(SYS_gettid);
}

static void *thread(void *arg)
{
    int n = *(int *)arg;
    int first, second;
    cpu_set_t cmask;
    pid_t pid = gettid();
    int i = 1E7;

    /* Set CPU affinity */
    CPU_ZERO(&cmask);
    CPU_SET(n % N_CPUS, &cmask);
    if (sched_setaffinity(pid, sizeof(cmask), &cmask) < 0) {
        fprintf(stderr,
            "could not set cpu affinity to core %d.", n % N_CPUS);
        exit(1);
    }
    /*
     * Acquire two locks and avoid deadlock; see also the dining
     * philosopher problem.
     */
    if (n % 2) {
        first = (n + 1) % N_CPUS;
        second = n;
    } else {
        first = n;
        second = (n + 1) % N_CPUS;
    }
    while (i) {
        lock(&shared[first * CACHELINE_SIZE]);
        lock(&shared[second * CACHELINE_SIZE]);
        i--;
        unlock(&shared[second * CACHELINE_SIZE]);
        unlock(&shared[first * CACHELINE_SIZE]);

    }
    return NULL;
}

int main(int argc, char **argv)
{
    struct sched_param sched;
    int i;

    fprintf(stdout, "running on %d cpus\n", N_CPUS);
    sched.sched_priority = sched_get_priority_max(SCHED_RR);
    if (sched_setscheduler(0, SCHED_RR, &sched) == -1) {
        perror("error setting SCHED_RR");
        return -1;
    }
    for (i = 0; i < N_CPUS; i++) {
        input[i] = i;
        if (pthread_create(&threads[i], NULL, thread, &input[i]) < 0) {
            perror("pthread_create failed");
            exit(1);
        }
    }
    for (i = 0; i < N_CPUS; i++)
        pthread_join(threads[i], NULL);

    return 0;
}

And here some results of a run in my laptop with Intel(R) Core(TM)2 CPU U7600 @ 1.20GHz:


=== USER_SPINLOCK=0 (use pthread_mutex) ===
$ /usr/bin/time -v sudo ./userspace-spinlock 
running on 2 cpus
        Command being timed: "sudo ./userspace-spinlock"
        User time (seconds): 17.01
        System time (seconds): 17.09
        Percent of CPU this job got: 190%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:17.91
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 0
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 658
        Voluntary context switches: 8426
        Involuntary context switches: 31
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

=== USER_SPINLOCK=1 (use user-space spinlocks) ===
$ /usr/bin/time -v sudo ./userspace-spinlock 
running on 2 cpus
        Command being timed: "sudo ./userspace-spinlock"
        User time (seconds): 13.12
        System time (seconds): 0.04
        Percent of CPU this job got: 191%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:06.89
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 0
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 657
        Voluntary context switches: 3
        Involuntary context switches: 17
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

The elapsed time with traditional pthread_mutex locking is 17.91 sec; with user-space spinlocks the execution needs only 6.89 sec! This means a speed-up of ~2.6!

As said above a disadvantage is the spinning of the CPUs that leads to a greater power consumption. This behaviour can be seen also looking at the voluntary context switches: 8426 in the pthread_mutex case and only 3 with user-space spinlocks (note: if you look at the code you can see that we're running real-time threads, this is the reason of this really low number of context switches). This also means that the whole system is really more reactive if the applications use pthread_mutex primitives, but if cases when reactiveness is not our goal (or we care about the reactiveness of few specific applications) we know that we can achieve a good speed-up of our parallel applications using user-space spinlocks.

potential framebuffer deadlock

2009-01-21T09:58:00.005+01:00

In the latest kernel (2.6.29-rc2) there's a potential deadlock condition in the frame buffer between fb_ioctl() and fb_mmap().

The cause is that fb_mmap() is called under mm->mmap_sem (A) held, that also acquires fb_info->lock (B); fb_ioctl() takes fb_info->lock (B) and does copy_from/to_user() that might acquire mm->mmap_sem (A) if a page fault occurs.

So we've a classic deadlock condition: a process holds the lock A and attempts to obtain the lock B, but B is already held by a second process that attemps to lock A.

A possible fix is to prevent the deadlock condition, that means "push down" the mutex fb_info->lock into the fb_ioctl() implementation and avoid the occurrence of the page fault with fb_info->lock held (A). But this also requires to define two basic primitives i.e. lock/unlock_fb_info() and use them opportunely inside *all* the framebuffer drivers. For now I've tried to fix at least the main fb_ioctl() function (with the common ioctl ops) that is shared by all the drivers.

systemimager-light: opening a new branch?

2009-01-21T09:33:00.002+01:00

I wonder if it's worth to open a new SystemImager branch without shipping the standard BOEL kernel anymore. Providing the packages for all the supported architectures requires a *huge* amount of time and it's not possible for me to build everything... :( (also because I don't have all the required architectures). So, probably the best way to proceed is to just remove BOEL and set UYOK the default. UYOK allows to create a boot&install package (kernel+initrd.img) for SystemImager using the kernel shipped in any distribution (preferably the kernel shipped with the distribution we want to install) and an initrd_template (shipped with the SystemImager packages). And with the UYOK-only version the time to build the packages is 10 times faster!

I've just opened the new branch locally in my PC using a git repository and pushed the initial release to download.systemimager.org (to get it: git-clone git://download.systemimager.org/local/git/systemimager-light), but the server is down again! :( grrr.... it seems we need a newer and more powerful server...

cache line bouncing

2008-12-15T21:50:00.013+01:00

Usually we don't realize how expensive is cacheline bouncing in parallel systems. Following is a simple example to evaluate the bouncing cost.

A multi-threaded application uses shared data in some of its thread:

struct shared_data_struct {
 unsigned int data1;
 unsigned int data2;
};

Suppose data1 is used only by thread1 and data2 is used by thread2. A natural way to optimize it is to pack data together in order to reduce the size of the application, thus maximizing the amount of memory that fits into the cache.

Unfortunately, this inevitably leads to poor performance: if both threads write to their assigned memory location the cache line must be always in exclusive state in the L1 data cache (L1D) of each core/processors and this generates a big cache coherency overhead.

For example, in the Intel Core 2 processor the cacheline size is 64 bytes (this can be retrieved using the command `getconf LEVEL1_DCACHE_LINESIZE` from the shell), so in the example above both data1 and data2 share the same L1D cacheline, though apparently they're using different independent memory locations.

We can measure the cost of the cacheline bounces using oprofile and a simple example. In cache-parallel.c (see the code below) we have the same `struct shared_data_struct` with an optional pad, depending on DISTINCT_CACHE_LINES symbol.

Let's see what happens without the pad, commenting the #define DISTINCT_CACHE_LINES:

Configure oprofile to account L1D cache misses:
$ sudo opcontrol --setup --event=L1D_PEND_MISS:500

Start oprofile:
$ sudo opcontrol -s
Using 2.6+ OProfile kernel interface.
Reading module info.
Using log file /var/lib/oprofile/samples/oprofiled.log
Daemon started.
Profiler running.

Run cache-parallel _without_ the pad in shared_data_struct:
$ time ./cache-parallel
...
real    0m29.274s
user    0m47.540s
sys     0m0.121s

Stop oprofile:
$ sudo opcontrol -h
Stopping profiling.
Killing daemon.

And see the results:
$ opannotate --source ./cache-parallel | grep data[12]++
1752  0.7326 :                sd->data1++;
237090 99.1361 :                sd->data2++;
^^^^^^
|
A lot of misses here!

If we add the pad (defining DISTINCT_CACHE_LINES):

$ time ./cache-parallel-with-pad
...
real    0m11.330s
user    0m20.686s
sys     0m0.037s

$ opannotate --source ./cache-parallel-with-pad | grep data[12]++
49 43.7500 :                sd->data1++;
24 21.4286 :                sd->data2++;
^^^^^^
|
Cache misses are dramatically reduced now!

A speedup of 29.274 / 11.330 = 2.583, in other words the cacheline bouncing effect produced, in this case, a slowdown of ~260%!!!

Following the source code of the cache-parallel example:

/*
* cache-parallel.c
*
* build with:
* gcc -DCACHELINE_SIZE=$(getconf LEVEL1_DCACHE_LINESIZE) -lpthread -Wall \
* -g -ocache-parallel cache-parallel.c
*/

#define _GNU_SOURCE

#include <stdio.h>
#include <stdlib.h>
#include <sched.h>
#include <unistd.h>
#include <pthread.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/syscall.h>

#define unlikely(expr) __builtin_expect(!!(expr), 0)
#define likely(expr) __builtin_expect(!!(expr), 1)

#define __cacheline_aligned__ __attribute__((__aligned__(CACHELINE_SIZE)))

#define LOOPS_MAX 2000000000
#define STACK_SIZE 4096

/*
* From GETTID(2):
*
*   Glibc does not provide a wrapper for this system call; call it using
*   syscall(2).
*
*/
static inline pid_t gettid(void)
{
return syscall(SYS_gettid);
}

/* XXX: comment this to see the effect of the cache line bouncing */
#define DISTINCT_CACHE_LINES
struct shared_data_struct {
unsigned int data1;
#ifdef DISTINCT_CACHE_LINES
unsigned char pad[CACHELINE_SIZE - sizeof(unsigned int)];
#endif
unsigned int data2;
};

static struct shared_data_struct shared_data __cacheline_aligned__;

static void dump_schedstats(pid_t pid, pid_t tid)
{
char buffer[256];
char filename[64];
FILE *f;

snprintf(filename, sizeof(filename),
"/proc/%d/task/%d/status", pid, tid);
f = fopen(filename, "r");
if (unlikely(f == NULL)) {
perror("could not read scheduler statistics");
exit(1);
}
while (fgets(buffer, sizeof(buffer), f))
fprintf(stdout, "[%d:%d] %s", pid, tid, buffer);
fclose(f);
}

static void *inc_first(void *arg)
{
struct shared_data_struct *sd = (struct shared_data_struct *)arg;
pid_t pid = getpid(), tid = gettid();
cpu_set_t cmask;
register long i;

/* set affinity */
CPU_ZERO(&cmask);
CPU_SET(0, &cmask);
if (unlikely(sched_setaffinity(pid, sizeof(cmask), &cmask) < 0)) {
perror("could not set cpu affinity for the child.");
exit(1);
}
/* periodically increment first member of shared struct */
for (i = 0; i < LOOPS_MAX; i++)
sd->data1++;
dump_schedstats(pid, tid);

return NULL;
}

static void *inc_second(void *arg)
{
struct shared_data_struct *sd = (struct shared_data_struct *)arg;
pid_t pid = getpid(), tid = gettid();
cpu_set_t cmask;
register long i;

/* set affinity */
CPU_ZERO(&cmask);
CPU_SET(1, &cmask);
if (unlikely(sched_setaffinity(0, sizeof(cmask), &cmask) < 0)) {
perror("could not set cpu affinity for current process.");
exit(1);
}
/* periodically increment second member of shared struct */
for (i = 0; i < LOOPS_MAX; i++)
sd->data2++;
dump_schedstats(pid, tid);

return NULL;
}

int main(int argc, char **argv)
{
void *child_stack;
pthread_t child_thr;

/* allocate memory for other process to execute in */
if (unlikely((child_stack = malloc(STACK_SIZE)) == NULL)) {
perror("cannot allocate stack for child");
exit(1);
}

/* create the child */
if (unlikely(pthread_create(&child_thr, NULL,
&inc_second, &shared_data) < 0)) {
perror("pthread_create failed");
exit(1);
}
inc_first((void *)&shared_data);
pthread_join(child_thr, NULL);

return 0;
}

SystemImager @ LinuxDay 2008 in Ferrara

2008-10-27T14:45:00.002+01:00

I presented a quick overview of SystemImager, how it works and typical use cases, at the LinuxDay 2008 in Ferrara. The slides are available here. There are also some nice pictures of the event.

After my talk Andrea Arcangeli presented an interesting talk about recent core kernel features (expecially mmu_notifier), that makes KVM to reliable swap guest mapped pages (without a mmu_notifier pages mapped in secondary MMU are pinned and cannot be swapped), perform balooning and save host memory with KSM (map different virtual address to common pages shared between different VMs).

a new website

2008-10-15T22:16:00.002+02:00

I've finished to write my new homepage hosted at dii.unisi.it... a plain and essential website (that is always the best choice IMHO) fully created with vim. It has been a long time since I wrote a website from scratch, and vim is still my preferred HTML editor! :)

fine-grained dirty_ratio and dirty_background_ratio

2008-10-08T10:35:00.004+02:00

A process that writes something to a file generates dirty pages in the page cache. Dirty pages must be kept in sync with their backing store (the file defined on the block device).

In the Linux kernel the frequency to writeback dirty pages is controlled by two parameters: vm.dirty_ratio and vm.dirty_background_ratio. Both are expressed a percentage of dirtyable memory, that is the free memory + reclaimable memory (active and inactive pages in the LRU list).

The first parameter controls when a process will itself start writing out dirty data, the second controls when the kernel thread [pdflush] must be woken up and it will start writing out dirty data globally on behalf of the processes (dirty_background_ratio is always less than dirty_ratio; if dirty_background_ratio >= dirty_ratio the kernel automatically set it to dirty_ratio / 2).

Unfortunately, both percentages are int and the kernel doesn't even allow to set them below 5%. This means that in large memory machine those limits are too coarse. On a machine that has 1GB of dirtyable memory the kernel will start to writeback dirty pages in chunks of 50MB (!!!) minimum (with dirty_ratio = 5).

Even if it could be fine for batch or server machines, this behaviour could be unpleasant for desktop or latency-sensitive environments, when the large writeback can be perceived as a lack of responsiveness in the whole system.

IMHO we really need an interface to define fine-grained limits (to writeback small amount of data, often) and the best solution for this without breaking the compatibility with the old interface seems to introduce a new interface to define pcm (milli-percent) values.

At least this would resolve the problem for today machines... until 1TB memory servers will become popular...

SystemImager @ CINECA

2008-10-07T10:11:00.003+02:00

An article I wrote (only in italian.. sorry) for CINECA news about SystemImager and the advantages of the BitTorrent transport, documenting the installation of the whole BCX cluster (1290 nodes).

cgroup I/O bandwidth controller results

2008-10-04T12:15:00.011+02:00

Some experimental results of the tests I ran on my box with the latest version of my cgroup-io-throttle patchset against the -mm kernel (2.6.27-rc5-mm1).

The goal of this test is to demonstrate the effectiveness of applying a throttling controller to enhance the IO performance predictability in a shared system.

The graphic highlights the bursty behaviour of the IO rate that we have with a plain CFQ IO scheduling (red line), and the smoother and contained behaviour using cgroup-io-throttle. We can also see the differences between the leaky bucket (green line) and token bucket (blue line) policies: first one is even more smooth and gives a better guarantee to respect the IO limit (hard limit), the second one allows a small irregularity degree (soft limit), but is better in terms of efficiency, expecially at high IO rates.

See my previous post to have an overview of the advantages this controler could provide.

cgroup: block device i/o bandwidth controller

2008-07-13T11:46:00.002+02:00

I've posted to the LKML ([1], [2], [3], [4]) the latest version of my patch to provide I/O bandwidth controlling for Linux cgroups. The objective of this patch is to improve I/O performance predictability of different cgroups sharing the same block devices.

Some advantages of providing this feature:

Allow I/O traffic shaping for block device shared among different cgroups
I/O performance predictability allows to better satisfy timing requirements for real-time applications
Limiting rules do not depend of the particular I/O scheduler (anticipatory, deadline, CFQ, noop) and/or the type of the underlying block devices
The bandwidth limitations are guaranteed both for synchronous and asynchronous operations, even the I/O passing through the page cache or buffers and not only direct I/O
It is possible to implement a simple user-space application to dynamically adjust the I/O workload of different process containers at run-time, according to the particular users' requirements and applications' performance constraints
It is even possible to implement event-based performance throttling mechanisms; for example the same user-space application could actively throttle the I/O bandwidth to reduce power consumption when the battery of a mobile device is running low (power throttling) or when the temperature of a hardware component is too high (thermal throttling)

task i/o accounting in Linux: provide distinct tgid/tid i/o statistics

2008-05-28T16:51:00.007+02:00

Here a new patch sent to the LKML. This patch (coverage at LWN.net) allows to properly account i/o statistics of threaded applications (tasks spawning threads by clone() syscall + CLONE_THREAD flag).

Without this fix a generic application can bypass of the accounting of the actual i/o statistics creating a bunch of threads doing the real i/o work.

The patch permits to account aggregate i/o statistics in /proc/PID/io, so it allows to easily find the top i/o consumer simply looking at this file, even when PID spawns the i/o worker threads. Moreover, a new file /proc/PID/task/TID/io (one for each TID) is created in procfs to account single thread i/o statistics, allowing in this way, to find which thread(s) of the top i/o consumer is eating the whole i/o bandwidth.

Recover data from hexdump output in the console

2008-05-08T23:16:00.004+02:00

It happens sooner or later to delete some blocks of a disk by a wrong "dd" executed as root... :-) Ok, if you've been so lucky to "hexdump -v -C" those blocks to the console before you can just copy&paste the dump-ed ouput and send it to the standard input of this one-liner perl script:

map { map { print pack('H[2]', $_) } split(/\s+/, $1)
if ((/^\w+\s+((\w{2}\s+){1,16})\s+|.*$/i) && ($1)) }

The corresponding binary data can be taken from the standard output. That's it! :-)

rsync and sparse files

2008-03-23T19:32:00.007+01:00

Unfortunately there's not a syscall in Linux to check if a file is sparse or not. There're some nice ideas to extend the lseek() syscall and implement such feature in ZFS (SEEK_HOLE and SEEK_DATA for sparse files), but there's nothing ready for production filesystems yet.

The common approach for user space applications is to implement a heuristic to check if a file can be treated as sparse (and save disk space) or not (and just write bytes to disk).

rsync, for example, checks every chunk of 1024 bytes before writing data to a generic destination file. If a chunk starts or ends with 0-s, these 0-s are just skipped by lseek().

Unfortunately, in some filesystems, typically optimized for large sequential I/O throughputs (like IBM GPFS, IBM SAN FS, or distributed filesystems in general), a lot of lseek()s operations can strongly impact on performances.

In this cases it can be very helpful to enlarge the block size used to handle sparse files.

For example, using a sparse write size of 32KB, I've been able to increase the transfer rate of an order of magnitude copying the output files of scientific applications from GPFS to GPFS or GPFS to SAN FS.

Read this thread on rsync mailing list.

And here is the patch to add --sparse-block=SIZE option to rsync, allowing to tune this parameter at run-time.

Xen vs KVM/QEMU for testing SystemImager installs

2008-03-22T13:18:00.006+01:00

Xen is great, but for SystemImager testing I've found many advantages using KVM/QEMU.

First of all, I don't need to run all my applications in a dom0 (or domU), that means I don't have any virtualization overhead for various openoffice instances, thunderbird, firefox, kopete, lastfm, kernel builds, SystemImager builds, and all the common activities I do in my laptop. With KVM/QEMU only the guest OSes are affected by virtualization overhead (more overhead with KVM/QEMU for the I/O anyway, respect to Xen guests). In fact, from this point of view KVM/QEMU is better suited for desktop machines.

Second, to test a standard SystemImager install I always need to use the unmodified BOEL or UYOK kernel, that means to run Xen fully-virtualized guests (HVM mode) and falls back to the QEMU hardware emulated model for I/O operations, aka "QEMU device manager" (qemu-dm), a patched version of the original qemu device emulation. So, in principle, there's not a big difference respect to run guest OSes with KVM/QEMU in terms of performance.

Third, I really like the command line syntax of kvm/qemu. In principle the xm syntax of Xen is better, because it supports both a configuration file and/or command line parameters, but I always prefer the qemu syntax, probably it's because I'm more familiar with it.

And what about VMWare? Well.. I've not been able to create x86_64 guests with VMPlayer, and I've not even found anything in the web that pointed me in the right direction. I liked the NAT-only network configuration of VMPlayer, but I think I can survive also with the little more complex bridged network setup for QEMU/KVM or Xen.

Here is my network config to start a KVM/QEMU virtual machine in Ubuntu 7.10:


/etc/dbus-1/event.d/25NetworkManager stop

modprobe tun
modprobe kvm-intel

brctl addbr br0
ifconfig br0 up
ifconfig eth0 up
brctl addif br0 eth0

tunctl -b -u 1000 -t qtap0
brctl addif br0 qtap0
ifconfig qtap0 up 0.0.0.0 promisc

chmod a+rw /dev/net/tun
chmod a+rw /dev/kvm

kvm -cdrom /home/righiandr/systemimager.iso \
        -hda /home/righiandr/xen/domains/Debian4/disk.img -boot d \
        -m 512 -net nic,model=rtl8139 -net tap,ifname=qtap0,script=no -smp 1

NOTE: systemimager.iso is made by si_mkautoinstallcd.

Automating Xen VM deployment with SystemImager

2008-03-22T12:46:00.005+01:00

Now, that I own an Intel VT capable PC, I'm able to test any kind of automatic Xen VM deployment with SystemImager directly from my laptop, even HVM Xen VMs.

Next step will be to test the deployment of a Xen VM with a Windows raw-disk image, cloned by SystemImager from a VMWare (vmplayer) machine. With this we're finally able to cover all the kind of migrations: virtual-to-virtual (that means migrations between equal or even different virtualization technologies), physical-to-virtual, virtual-to-physical, and physical-to-physical (obviously last one was the main task of SystemImager). And not only with Linux! Anyway, at the moment, this still needs some manual tweaks (read the howtos above) and I would really like to spend some efforts to implement a SystemImager GUI to fully automatize all these steps.

picasa upload & download

2008-03-16T15:37:00.004+01:00

After uploaded a lot of photos to picasa web albums via this small script I wrote (picasaupload), I realized that there's not a "magic key " to download the pictures back again to my pc. So, here is another script that works in the other way: picasadownload.

As suggested by the name this script allows to download images from a Google Picasa web album, passed as command line argument.

For example to download the pictures I made at Celsa, a wonderful castle near Siena:

picasadownload http://picasaweb.google.com/righi.andrea/Celsa

A new notebook...

2008-03-13T14:18:00.002+01:00

After carrying around my notebook with all my books and notes I finally considered to buy an ultraportable model: a Dell Latitude D430.

My specific model is configured as follows:

Intel U7600 Ultra Low Voltage (ULV) Core 2 Duo
2GB DDR2 533MHz SDRAM (1 x 1GB + 1GB integrated)
12.1" WXGA Display (1280 x 800)
80GB Toshiba 5200RPM 1.8'' HDD (I'll move to a SSD soon ;-))
MediaBase 8x DVD+/-RW Drive - W Euro
Intel 4965AGN wireless card
FreeDOS inside! Woo-hoo!!! (H.Simpson accent) no Microcrap included! :-) just reinstalled with Ubuntu 7.10.

And now I can finally build x86_64 SystemImager packages at home (or better everywhere) and play with Xen and a processor with Intel VT extensions!

SystemImager cluster map

2008-02-16T22:23:00.003+01:00

Finally we officially opened the SystemImager cluster map a Google maps integrated tool to keep track who's actually using SystemImager around the world. It's awesome isn't it? :-)

Linux: I/O throttling (again)

2008-01-17T21:59:00.000+01:00

I've improved my per-task I/O throttling patch to support per-uid/gid I/O throttling. As reported in the patch description:

Allow to limit the I/O bandwidth for specific uid(s) or gid(s) imposing
additional delays on those processes that exceed the limits defined in a
configfs tree.

A typical use of this patch could be on a shared Linux system under heavy load condition due to I/O-intensive applications. In this scenario it's possible to assign a different amount of available I/O bandwidth for each group or user (read also fair sharing, I/O shaping, etc.): for example 5MB/s to group A (students), 20MB/s to group B (professors), unlimited MB/s for user C (sysadmin), etc.

But a vastely more interesting approach would be to implement a control group (cgroup) based I/O throttling... and I've just started to work on this! ;-)

Linux: per-task I/O throttling

2008-01-13T17:08:00.000+01:00

I've posted a patch on the LKML that allows to limit the I/O bandwidth per-task via /proc filesystem. Writing a value > 0 in /proc/PID/io_throttle allows to set the upper bound limit of the I/O bandwidth (in 512-bytes sector per second) usable by the process PID.

The patch itself it's not really useful, the same result can be obtained by ionice and a good I/O scheduler (like CFQ), but my patch it's a very simple proof-of-concept that it's possible to implement a kind of UID/GID (or even process-container) based policy of I/O bandwidth shaping (like network bandwidth shaping).

Anyway, just right now I'm running my new 2.6.24-rc7-io-throttle kernel and using the following script to throttle the I/O consumption of the backup, that now can run in backgrund with a very small impact in my other applications. ;-)

WARNING: obviously this script requires my kernel patch to work...

$ cat ~/bin/iothrottle
#!/bin/sh

[ $# -lt 2 ] && echo "usage: $0 RATE CMD" && exit 1

rate=$1
shift
$* &
trap "kill -9 $!" SIGINT SIGTERM
[ -e /proc/$!/io_throttle ] && echo $rate >/proc/$!/io_throttle
wait %1

PyGFS: implementing a distributed filesystem in python

2008-01-05T15:19:00.000+01:00

In this post I try to explain how to implement a secure and robust distributed filesystem in user-space with python.

The advantages of user-space are many: no kernel modification, no OS crashes due to buggy code, debugging is easy, etc. Moreover, for the development point of view, in user-space it's possible to exploit all the nice features provided by the user-space libraries! It means that with few lines of code we can provide a lot of interesting features.

So, let's see some potential requirements for our filesystem:

the filesystem must support a complete set of standard POSIX APIs,
as a distributed filesystem it must provide data accessibility to remote hosts,
it must be reliable to hardware or network failures,
it must be secure (it must provide authentication, authorization and encryption mechanisms to provide secure access over insecure networks).

Even if the requirements seem to fit on a long-term project, it's possible to satisfy all of them with few lines of code. Let's see how.

The user-space accessibility is provided by FUSE, that allows to implement a full POSIX filesystem without any kernel changes (it provides all the required kernel APIs to register a filesystem without any kernel-space code). FUSE also allows to provide a secure method for non privileged users to mount their own filesystem.

A distributed filesystem also need a mechanism for communications (how to send data to the remote hosts). An interesting project that could help us for this is Pyro. Pyro allows to skip the development for a new networking communication protocol, since it provides an elegant and easy-to-use object oriented form of RPC. It also optionally supports x509 certificate encryption, that perfectly covers our security requirement.

At this point the real filesystem implementation is quite easy, we can use a simple client-server approach like NFS.

The client wraps all the POSIX syscalls in the filesystem defined by the FUSE interface and calls the equivalent OS routines on the remote server (using Pyro RPC); the server executes the OS procedues over the back-end filesystem and pass to the client the same result returned by the OS syscall (executed on the server filesystem).

Moreover, to provide reliability feautures it's possible to exploit the robust exception handling statements in python. In this way we can detect all the communication failures and call an opportune event handler to re-issue the operations when the server become reachable again. We can also increase the reliability using a client-side and a server-side file handles; in this way each file handle at the client-side can mapped to a different file handle at the server side. If the server goes down the mapping between the two file handles is simply re-initialized and this allows to transparently continue the operations on the clients as the server was never stopped.

So, I tried to implement a real example of this filesystem and I've called it PyGFS (it should be something like: python grid file system... in perspective I'd like to improve it with multiple servers to mirror or unify more filesystems in different hosts, just like a real grid-filesystem...). The source code is available to all who are interested on it... If you even have ideas to add new features let me know... ;-)

SystemImager @ LinuxDay 2007 in Bologna

2008-01-05T15:03:00.000+01:00

The slides of my talk "SystemImager and BitTorrent: a peer-to-peer approach to large scale OS deployment" presented at LinuxDay 2007 in Bologna are available here (the site is in italian bug the slides are still in english).

They're pretty the same slides on SystemImager website, with few small changes.

How to sync Google Calendar with a local iCal file

2007-10-02T21:30:00.000+02:00

I've written a simply python script (googlecalendarsync) to synchronize Google Calendar (only the default calendar) with a local iCal file. The advantage of this script, respect to the Provider for Google Calendar solution is that it's possible to work offline in the calendar and it allows to use any calendar client that supports the iCal format (so not only Lightning or Sunbird).

The synchronization is bi-directional: it means that it's possible to create/update/delete events directly from the google calendar interface or directly in your calendar client. The last modified events are taken if the same event is modified in both two places.

After any change you can synchronize the calendars running the script (without any argument).

I've opened a new project in the google project hosting; the tool is available here: http://googlecalendarsync.googlecode.com/

libnosync: are *sync really necessary?

2007-05-16T20:24:00.000+02:00

I was asking myself why user applications should care about the synchronization of their buffers. I suppose it's a task dedicated to the operating system, that actually knows what is better for the system. Looking at the manpage of FSYNC(2) we can see that:


NAME
       fsync,  fdatasync  -  synchronize  a file's complete in-core state with
       that on disk

[snip]

DESCRIPTION
       fsync copies all in-core parts of a file to disk, and waits  until  the
       device  reports  that all parts are on stable storage.  It also updates
       metadata stat information. It does  not  necessarily  ensure  that  the
       entry  in the directory containing the file has also reached disk.  For
       that an explicit fsync on the file descriptor of the directory is  also
       needed.

[snip]

OK, but... why? I wrote a simple glibc wrapper (see below) in order to have "fake" fsync() and fdatasync() - not for the simple sync(), so you can continue to run the famous `sync; sync; sync`, if you're paranoid enough ;-) - and I was impressed by the heavy use of them by the user applications... and the speed-up if you disable them.

In fact, if you have a journaled filesystem (hey! otherwise I think you should really consider to move to a journaled filesystem!) all the flushes of metadata causes a lot of writes in the journal (for example in ext3 a single fsync() causes the write of *everything*) and it's a lot of I/O for your PC. And this is a disadvantage also in term of power consumption.

So, where is the trick?! After some thoughts I realized that the main reson should be to *be* really sure that the internal metadata of the applications (like a DBMS for example), built on-top-of the filesystem, have been correctly written to the backing store area. Everything that implements its own concept of "journal" should use the *sync() functions. Otherwise if a crash occurs just in the middle of an "important" write, well... at the resume the metadata of your filesystem will be ok, but the metadata of the application (mapped into the filesystem data) could result corrupted. So, in order to have a robust desktop it's surely better to have those syscalls enabled.

OK, but is this really important for *all* your applications??? for example I don't think it's important for amarok... for example try to run a simple `strace -qfe trace=fdatasync,fsync amarok`. In my system I can see 36 syscalls of *sync!!! and this is too much... BTW I've nothing against amarok, it's a great application & my favourite music player :-)

Following the *sync() lib wrapper. Use this (always without any warranty) if you want to run your non-critical application faster. [IDEA] It would be interesting to run your apps with the wrapper and execute a `sync; sync; sync` just before the screensaver... :-)


/*
 *  libnosync
 *
 *  Copyright (C) 2007 Andrea Righi 
 *
 *  This program is free software; you can redistribute it and/or modify
 *  it under the terms of the GNU General Public License as published by
 *  the Free Software Foundation; either version 2 of the License, or
 *  (at your option) any later version.
 *
 *  This program is distributed in the hope that it will be useful,
 *  but WITHOUT ANY WARRANTY; without even the implied warranty of
 *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 *  GNU General Public License for more details.
 *
 *  You should have received a copy of the GNU General Public License
 *  along with this program; if not, write to the Free Software
 *  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 *
 * Compile:
 *     gcc -fPIC -Wall -O2 -g -shared -W1,-soname,libnosync.so.0 \
 *     -o libnosync.so.0.1 -lc -ldl
 *
 * Use:
 *     export LD_PRELOAD=`pwd`/libnosync.so.0.1
 *
 * Remove:
 *     unset LD_PRELOAD
 */

#define _GNU_SOURCE
#define __USE_GNU

#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 

#ifdef DEBUG
#define DPRINTF(format, args...) fprintf(stderr, "debug: " format, ##args)
#else
#define DPRINTF(format, args...)
#endif

int fdatasync(int) __attribute__ ((weak, alias("wrap_fsync")));
int fsync(int) __attribute__ ((weak, alias("wrap_fsync")));

int wrap_fsync(int fd)
{
        DPRINTF("called fsync/fdatasync on fd = %d\n", fd);
        return 0;
}

Thunderbird + Google Calendar

2007-05-13T16:45:00.000+02:00

I realized that I really need a calendar integrated in my email client (Thunderbird), and unfortunately the *great* vim + some shell script in cron are not enough... :-)

It's a couple of weeks that I'm using lightning with a cool extension called Provider for Google Calendar, that allows a bi-directional access (r/w) to google calendar directly from Thunderbird. I've also enabled the SMS notification (free) and honestly I have to admin that it's simply great! Now I can read my events, tasks, TODOs, etc. everywhere using a web browser or by my email client and receive alarms and notifications in my phone.

LinuxTag 2007 in Berlin

2007-05-12T10:50:00.000+02:00

I'll be at Linuxtag 2007 in Berlin. On thursday (31/05/2007) I'll present a paper about the integration of the BitTorrent protocol in SystemImager, to quickly deploy operating systems in large installations, like HPC clusters, big render farms or complex grid-computing environments.

Linux VM: per-user overcommit policy

2007-05-11T19:17:00.000+02:00

I wrote a simple patch that allows to define per-UID virtual memory overcommit handling.

Configuration is stored in a hash list in kernel space reachable through /proc/overcommit_uid (surely there're better ways to do it, i.e. via configfs).

Since most of the time we've readers, the concurrent read/write accesses of the hash list are synchronized using the RCU (Read Copy Update) mutual exclusion.

Hash elements are defined using a triple:

uid:overcommit_memory:overcommit_ratio

The overcommit_* values have the same semantic of their respective sysctl variables. If a user is not present in the hash, the default system policy will be used (defined by /proc/sys/vm/overcommit_memory and /proc/sys/vm/overcommit_ratio).

Example:

- admin can allocate full memory + swap:

root@host # echo 0:2:100 > /proc/overcommit_uid

- processes belonging to sshd (uid=100) and ntp (uid=102) users can be quite critical, so use the same policy of the admin:

root@host # echo 100:2:100 > /proc/overcommit_uid
root@host # echo 102:2:100 > /proc/overcommit_uid

- Others can allocate up to the swap + 60% of the available RAM:

root@host # echo 2 > /proc/sys/vm/overcommit_memory && echo 60 > /proc/sys/vm/overcommit_ratio

The result in the example above is that the memory is never overcommitted (due to the value 2 in overcommit_memory) and the 40% of the RAM is used as spare memory, reserved for root processes and critical services only. Normal users can use only the 60% of the RAM. So, in conclusions, non-privileged users never hog the machine.

You can play with per-user overcommit parameters to implement your own VM allocation rules.

This is only a very simple approach to user resource management. If you want a more flexible, complete and powerful approach look at the containers work, a very interesting project actively developed in Linux.

ZERO_PAGE or not ZERO_PAGE...

2007-04-16T19:16:00.000+02:00

Interesting discussion on the LKML about the opportuninty to remove the ZERO_PAGE for anonymous mappings (http://lkml.org/lkml/2007/4/3/432).

ZERO_PAGE is a single physical page that is always filled by 0 and it's used for zero-mapped memory areas.

It is used for example to initialize the anonymous pages of a task (not file-backed memory that exists only during the life of the task). When a program performs a malloc() the buffer returned by the function should be filled by zero. If the program tries to read from that buffer, the kernel, instead of allocating new physical free pages without any reasonable purpose, maps all the virtual accessed memory to the ZERO_PAGE.

Anyway, in general, an application that reads from a just allocated empty buffer is a quite stupid application :-) (except when you have to work with sparse matrices!) and the ZERO_PAGE handling has a cost in every COW faults.

The following patch removes the handling of the ZERO_PAGE for anonymous memory mappings and it simply allocates new physical pages in the case that a program wants to read empty buffers. Depending on your applications you should see a small improvement in terms of performance, but a bigger memory consumption if you runs that kind of applications mentioned above.

side note: I'm using it in my notebook and it works fine! :-)


--- linux-2.6.20.4/mm/memory.c.orig    2007-04-06 00:23:52.000000000 +0200
+++ linux-2.6.20.4/mm/memory.c    2007-04-06 00:25:48.000000000 +0200
@@ -1569,16 +1569,11 @@

  if (unlikely(anon_vma_prepare(vma)))
      goto oom;
-    if (old_page == ZERO_PAGE(address)) {
-        new_page = alloc_zeroed_user_highpage(vma, address);
-        if (!new_page)
-            goto oom;
-    } else {
-        new_page = alloc_page_vma(GFP_HIGHUSER, vma, address);
-        if (!new_page)
-            goto oom;
-        cow_user_page(new_page, old_page, address, vma);
-    }
+
+    new_page = alloc_page_vma(GFP_HIGHUSER, vma, address);
+    if (!new_page)
+        goto oom;
+    cow_user_page(new_page, old_page, address, vma);

  /*
   * Re-check the pte - we dropped the lock
@@ -2088,38 +2083,24 @@
  spinlock_t *ptl;
  pte_t entry;

-    if (write_access) {
-        /* Allocate our own private page. */
-        pte_unmap(page_table);
+    /* Allocate our own private page. */
+    pte_unmap(page_table);

-        if (unlikely(anon_vma_prepare(vma)))
-            goto oom;
-        page = alloc_zeroed_user_highpage(vma, address);
-        if (!page)
-            goto oom;
+    if (unlikely(anon_vma_prepare(vma)))
+        goto oom;
+    page = alloc_zeroed_user_highpage(vma, address);
+    if (!page)
+        goto oom;

-        entry = mk_pte(page, vma->vm_page_prot);
-        entry = maybe_mkwrite(pte_mkdirty(entry), vma);
+    entry = mk_pte(page, vma->vm_page_prot);
+    entry = maybe_mkwrite(pte_mkdirty(entry), vma);

-        page_table = pte_offset_map_lock(mm, pmd, address, &ptl);
-        if (!pte_none(*page_table))
-            goto release;
-        inc_mm_counter(mm, anon_rss);
-        lru_cache_add_active(page);
-        page_add_new_anon_rmap(page, vma, address);
-    } else {
-        /* Map the ZERO_PAGE - vm_page_prot is readonly */
-        page = ZERO_PAGE(address);
-        page_cache_get(page);
-        entry = mk_pte(page, vma->vm_page_prot);
-
-        ptl = pte_lockptr(mm, pmd);
-        spin_lock(ptl);
-        if (!pte_none(*page_table))
-            goto release;
-        inc_mm_counter(mm, file_rss);
-        page_add_file_rmap(page);
-    }
+    page_table = pte_offset_map_lock(mm, pmd, address, &ptl);
+    if (unlikely(!pte_none(*page_table)))
+        goto release;
+    inc_mm_counter(mm, anon_rss);
+    lru_cache_add_active(page);
+    page_add_new_anon_rmap(page, vma, address);

  set_pte_at(mm, address, page_table, entry);

A fast way to get users' disk usage on ext2/ext3 filesystems

2007-04-14T17:00:00.000+02:00

The simplest way to get the per user disk usage is to mount the filesystem with the quota accounting (see "man 8 mount"). This is the most reliale way to be sure that the quota limits will be respected, since the accounting and the checks are performed synchronously by the filesystem itself. Unfortunately this adds a little overhead in the whole filesystem.

Another approach is to periodically check the disk usage with a script (typically with a cron job). The script should sum the size of each file and directory in the filesystem grouping them by the respective owners.

This program (e2fsusage) uses the second approach, but it doesn't read directly the files and directories, it analyze the filesystem metadata, performing a sequential scan of all the inodes.

In this way it bypass the process for the translation of the file/dir names into the respective inodes and it strongly reduces the total time to scan the entire filesystem. Moreover, since it evaluates the real allocated blocks of the filesystem using i_blocks, instead of i_size (see the struct ext2_inode in /usr/include/ext2fs/ext2_fs.h) it is able to detect the true size
occupied by each user (read it as: it is able to correctly handle sparse files).

How to bypass the buffer cache in Linux

2007-04-04T21:27:00.001+02:00

Linux has 2 kind of caches: the page cache and the buffer cache. The role of the page cache is to speed-up the access of the files on disks, in a similar way the buffer cache contains buffers of pages read from or being written to block devices. Both of them are memory areas managed in different ways (one more optimized for file objects and the other more block device oriented).

From /proc/meminfo is possible to monitor the memory allocated for both caches (Buffers is the buffer cache, Cached is the page cache), for example:

# cat /proc/meminfo
...
Buffers:         15116 kB
Cached:          67912 kB
...

To perform an I/O benchmark on block devices (like /dev/sda, /dev/sdb, etc.) we usually use a simple `dd`, that loads data from device into memory (in read tests) or write from memory to device (in write tests). But in this cases data are accessed only once! There are no more reads or writes on their buffers. In these cases the buffer cache is only an overhead and it should be meaningful to bypass it.

A way is to open the files using the flag O_DIRECT. This flag allows to bypass the caching mechanisms and exploit directly the DMA from/to the block device and the userspace source/destination buffers.

Obviously there's not in the kernel a global flag to say: "ok just disable buffer cache" and it's not even possibile to disable the buffer cache for a single process.

In the case that you can (and you want) to patch and recompile your application you could explicitly set the flag O_DIRECT in every open()s, but it wouldn't be so handy... ;-)

Another solution is to write a simple glibc wrapper that intercepts all the open() and set the O_DIRECT flag.

Following an example:

libdirectio.c

#define _GNU_SOURCE
#define __USE_GNU

#include <stdio.h>
#include <stdarg.h>
#include <string.h>
#include <fcntl.h>
#include <dlfcn.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>

#define DEBUG

#ifdef DEBUG
#define DPRINTF(format, args...) fprintf(stderr, "debug: " format, ##args)
#else
#define DPRINTF(format, args...)
#endif

int open(const char *, int, ...) __attribute__ ((weak, alias("wrap_open")));
int __open(const char *, int, ...) __attribute__ ((weak, alias("wrap_open")));
int open64(const char *, int, ...) __attribute__ ((weak, alias("wrap_open64")));
int __open64(const char *, int, ...) __attribute__ ((weak, alias("wrap_open64")));

static int (*orig_open)(const char *, int, ...) = NULL;
static int (*orig_open64)(const char *, int, ...) = NULL;

static int __do_wrap_open(const char *name, int flags, mode_t mode,
int (*func_open)(const char *, int, ...))
{
    if (strncmp("/dev/null", name, sizeof("/dev/null"))) {
        DPRINTF("setting flags O_DIRECT on %s\n", name);
        flags |= O_DIRECT;
    }
    if (!strncmp("/dev/", name, sizeof("/dev/") - 1) ||
            !strncmp("/proc/", name, sizeof("/proc/") - 1))
        return fd;
    return func_open(name, flags, mode);
}

int wrap_open(const char *name, int flags, ...)
{
    va_list args;
    mode_t mode;

    va_start(args, flags);
    mode = va_arg(args, mode_t);
    va_end(args);

    DPRINTF("calling libc open(%s, 0x%x, 0x%x)\n", name, flags, mode);

    return __do_wrap_open(name, flags, mode, orig_open);
}

int wrap_open64(const char *name, int flags, ...)
{
    va_list args;
    mode_t mode;

    va_start(args, flags);
    mode = va_arg(args, mode_t);
    va_end(args);

    DPRINTF("calling libc open64(%s, 0x%x, 0x%x)\n", name, flags, mode);

    return __do_wrap_open(name, flags, mode, orig_open64);
}

void _init(void)
{
    orig_open = dlsym(RTLD_NEXT, "open");
    if (!orig_open) {
        fprintf(stderr, "error: missing symbol open!\n");
        exit(1);
    }
    orig_open64 = dlsym(RTLD_NEXT, "open64");
    if (!orig_open64) {
        fprintf(stderr, "error: missing symbol open64!\n");
        exit(1);
    }
}

Makefile

VERSION=0.1

TARGET=libdirectio.so.$(VERSION)
OBJS=libdirectio.o
CC=gcc
CFLAGS= -fPIC -Wall -O2 -g
SHAREDFLAGS= -nostartfiles -shared -W1,-soname,libdirectio.so.0

all: $(TARGET)

%.o: %.c
$(CC) -I. $(CFLAGS) -c $< -o $@

$(TARGET): $(OBJS)
$(CC) $(SHAREDFLAGS) $(OBJS) -o $(TARGET) -lc -ldl

clean:
rm -f $(OBJS) $(TARGET)

To compile the library simply run `make`.You can pre-load it using the LD_PRELOAD environment variable in this way:

# export LD_PRELOAD=$FULL_PATH_OF_YOUR_LIBRARY/libdirectio.so.0.1

Then you can run your brand-new direct I/O benchmark (typically `dd`) for block devices.To unload the library and restore the standard access simply run:

# unload LD_PRELOAD

disk I/O per-process accounting

2007-03-25T21:45:00.000+02:00

A common problem in Linux is how to find the most I/O intensive process when there is an intense disk activity of the system. In some cases you may want to kill the crazy process that caused this condition.

A lot of tools in Linux are able to deliver generic stats for your system: top, sar, dstat, iostat, vmstat, ... but unfortunately none of them is capable to show the particular disk activity done by each process.

The following kernel patch enables the userspace tools to access per-process I/O statistics (WARNING: I tested it only with 2.6.18.3 vanilla!!!):


--- include/linux/sched.h.orig 2007-03-25 21:42:50.000000000 +0200
+++ include/linux/sched.h 2007-03-25 21:42:56.000000000 +0200
@@ -990,6 +990,12 @@
  struct rcu_head rcu;
 
  /*
+  * disk I/O accounting informations
+  */
+ unsigned long long acct_disk_read;
+ unsigned long long acct_disk_write;
+
+ /*
   * cache last used pipe for splice
   */
  struct pipe_inode_info *splice_pipe;
--- block/ll_rw_blk.c.orig 2007-03-25 18:05:51.000000000 +0200
+++ block/ll_rw_blk.c 2007-03-25 18:12:51.000000000 +0200
@@ -2586,6 +2586,12 @@
   disk_round_stats(rq->rq_disk);
   rq->rq_disk->in_flight++;
  }
+
+ if (rw == READ) {
+  current->acct_disk_read += nr_sectors;
+ } else {
+  current->acct_disk_write += nr_sectors;
+ }
 }
 
 /*
--- fs/proc/array.c.orig 2007-03-25 18:13:07.000000000 +0200
+++ fs/proc/array.c 2007-03-25 18:15:00.000000000 +0200
@@ -412,7 +412,7 @@
 
  res = sprintf(buffer,"%d (%s) %c %d %d %d %d %d %lu %lu \
 %lu %lu %lu %lu %lu %ld %ld %ld %ld %d 0 %llu %lu %ld %lu %lu %lu %lu %lu \
-%lu %lu %lu %lu %lu %lu %lu %lu %d %d %lu %lu %llu\n",
+%lu %lu %lu %lu %lu %lu %lu %lu %d %d %lu %lu %llu %llu %llu\n",
   task->pid,
   tcomm,
   state,
@@ -457,7 +457,9 @@
   task_cpu(task),
   task->rt_priority,
   task->policy,
-  (unsigned long long)delayacct_blkio_ticks(task));
+  (unsigned long long)delayacct_blkio_ticks(task),
+  task->acct_disk_read,
+  task->acct_disk_write);
  if(mm)
   mmput(mm);
  return res;

The patch adds at the end of the process status array (see /usr/src/linux/fs/proc/array.c) two entries:

the I/O read activity of the process
the I/O write activity of the process

You can access to them via the proc filesystem, the process array is in /proc/[pid]/stat (see `man 5 proc`).

For example the following command shows the "top 10" list of the most I/O intensive processes of my system:


$ cat /proc/[0-9]*/stat | awk '{print $2 ":" $43 + $44}' | sort -rn -t : -k 2 | head
(pdflush):275240
(reiserfs/0):179064
(thunderbird-bin):74376
(cupsd):18904
(firefox-bin):15640
(Xorg):13632
(netstat):13512
(gaim):9096
(kswapd0):6032
(syslog-ng):4568

As expected at the first place there's the pdflush (the worker_thread that writes back filesystem data), followed by the reiserfs/0 worker_thread... but obviously you can't kill them! they're kernel thread... so in my case the most active I/O intensive userspace process is thunderbird! ;-)

You can also write your custom top-like userspace tools to monitor the I/O rate of each process, or a program to see if your processes are doing more reads or writes, etc...

2007-03-07T22:00:00.000+01:00

A quite old, but very nice howto about basic kernel bug hunting techniques:

kernel debugging (part 1)
kernel debugging (part 2)
kernel debugging (part 3)

2007-03-06T10:34:00.000+01:00

Excellent explanation about SVN merging. SVN merge is a good way to merge code changes between different directories within a repository and this is done often when it's necessary to apply the same fixes in different branches.

weekend at Marmoraia

2007-03-05T19:21:00.000+01:00

Yesterday visit to the beautiful church of Marmoraia, a romanic church of XI century in the hills of the Montagnola Senese, with a wonderful walk in the nearby woods of chestnut trees...

multi-threaded "cat-like" command for web pages...

2007-03-03T15:33:00.000+01:00

Following a very useful script to dump HTML of one or more web links, given as arguments, to standard output. If more than one link is passed it spawns a thread for each link (synchronizing the dump on stdout in mutual exclusion). The threaded approach reduce the average waiting time for the reply of the connections and it's strongly improve performances when we need to download a lot of pages at the same time (i.e. I usually use this script to dump and grep the linux kernel changelogs directly from web...).

BTW: python is great! ;-)


#!/usr/bin/env python

import sys, urllib, urllib
from threading import Thread, Lock

class webThread(Thread):
def __init__(self, url):
  self.url = url
  Thread.__init__(self)

def run(self):
  remotefile = urllib.urlopen(self.url)
  data = remotefile.read()
  remotefile.close()

  stdout_mutex.acquire()
  print "=== %s ===" % self.url
  print data
  stdout_mutex.release()

if __name__ == '__main__':
if len(sys.argv) <>" % sys.argv[0]
  sys.exit(1)
else:
  threads = []
  stdout_mutex = Lock()
  sys.stdout.flush()
  for url in sys.argv[1:]:
      t = webThread(url)
      t.start()
      threads.append(t)
  for t in threads:
      t.join()

interactive support for si_psh

2007-03-01T19:59:00.000+01:00

Today I improved my favorite distributed shell si_psh (that is part of SystemImager) adding the interactive support.

I need it for the BCX cluster when I have to run multiple commands on the same subset of nodes. Without the interactive support I used to edit the command line of the previous command in the shell, but with long commands this is not practice enough...

I discovered also that ssh (maybe via glibc, dunno...) knows when it has the stdin opened on a terminal or not and if it doesn't find a valid terminal it's not possible to catch the stderr inside a perl wrapper script! The problem is that when si_psh runs interactively stdin is not opened in the terminal, but it's opened inside the perl script, to get the user commands (like a typical shell). And so I wasn't able to get the stderr of the spawned ssh sessions...

It's possible to workaround spawning the ssh process via exec. For example in this case @out contains both stdout and stderr:

my @out = `exec 2>&1 ssh ...`;

And this doesn't work (@out doesn't contain the stderr):

my @out = `ssh ... 2>&1`;

memory leak bug with IA32 emulation on x86_64

2007-02-28T20:12:00.000+01:00

It seems that the kernel of some recent distributions (like RHEL4 or SLES9 for example) are affected by a memory leak bug in the committed memory: the virtual memory allocated by the userland applications and requested by *malloc(). This occurs only with 64-bit processors (like x86_64, in my case) when you run IA32 applications. If you start to run a lot of IA32 applications you can see the value of Committed_AS in /proc/meminfo to grow forever... it occurs only in the kernels of some distributions, not with recents vanilla.

But... is it a critical bug? it depends... virtual memory is not physical memory, applications can always request a memory region, but if they don't use it the physical memory is never allocated. The point is: should the kernel give virtual memory to the processes also if they are requesting more than the physical memory? In case of yes the system is overcommitting the memory.

Linux supports 3 overcommit handling policies (see /usr/src/linux/Documentation/vm/overcommit-accounting):

"guess" policy
always overcommit
never overcommit

By default Linux uses the "guess" policy: the kernel uses a heuristic to decide if a memory request can be committed or not. This heuristic does not depend on the value of the committed memory, but it depends essentially on the free physical memory. Also with always overcommit the counter of the committed memory is not important (except for accounting informations). But with never overcommit policy the value of the committed memory defines the result of the memory requests, (because the memory can't be overcommitted) so in this case it is functionally important. For more implementation details see __vm_enough_memory() in /usr/src/linux/mm/mmap.c.

The memory leak bug occurs because during exec(), setup_arg_pages() calls vm_enough_memory() for a vma without the VM_ACCOUNT flag set. When the process exits, exit_mmap() only calls vm_unacct_memory() if the vma has the VM_ACCOUNT flag set... hey! but so we're really leaking memory here...

The fix in this case is very simple:

--- include/asm-x86_64/page.h.orig 2007-02-27 17:31:05.000000000 +0100
+++ include/asm-x86_64/page.h 2007-02-27 17:24:26.000000000 +0100
@@ -134,7 +134,7 @@
#define __VM_DATA_DEFAULT_FLAGS (VM_READ | VM_WRITE | VM_EXEC | \
VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC)
#define __VM_STACK_FLAGS (VM_GROWSDOWN | VM_READ | VM_WRITE | VM_EXEC | \
- VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC)
+ VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC | VM_ACCOUNT)

#define VM_DATA_DEFAULT_FLAGS \
(test_thread_flag(TIF_IA32) ? vm_data_default_flags32 : \

Unfortunately there's another problem... in arch/x86_64/ia32/ia32_binfmt.c, security_vm_enough_memory() tend to forget to vm_unacct_memory() when a
failure occurs (this problem is more rare, but it can occur). For this problem the patch is the following:

--- arch/x86_64/ia32/ia32_binfmt.c.orig 2007-02-27 17:26:47.000000000 +0100
+++ arch/x86_64/ia32/ia32_binfmt.c 2007-02-27 17:27:01.000000000 +0100
@@ -347,11 +347,6 @@
if (!mpnt)
return -ENOMEM;

- if (security_vm_enough_memory((IA32_STACK_TOP - (PAGE_MASK & (unsigned long) bprm->p))>>PAGE_SHIFT)) {
- kmem_cache_free(vm_area_cachep, mpnt);
- return -ENOMEM;
- }
-
memset(mpnt, 0, sizeof(*mpnt));

down_write(&mm->mmap_sem);
--- fs/exec.c.orig 2007-02-27 17:27:39.000000000 +0100
+++ fs/exec.c 2007-02-27 17:28:08.000000000 +0100
@@ -413,11 +413,6 @@
if (!mpnt)
return -ENOMEM;

- if (security_vm_enough_memory(arg_size >> PAGE_SHIFT)) {
- kmem_cache_free(vm_area_cachep, mpnt);
- return -ENOMEM;
- }
-
memset(mpnt, 0, sizeof(*mpnt));

down_write(&mm->mmap_sem);
--- mm/mmap.c.orig 2007-02-27 17:27:50.000000000 +0100
+++ mm/mmap.c 2007-02-27 17:28:58.000000000 +0100
@@ -2024,6 +2024,9 @@
__vma = find_vma_prepare(mm,vma->vm_start,&prev,&amp;amp;rb_link,&rb_parent);
if (__vma && __vma->vm_start <>vm_end)
return -ENOMEM;
+ if ((vma->vm_flags & VM_ACCOUNT) &&
+ security_vm_enough_memory(vma_pages(vma)))
+ return -ENOMEM;
vma_link(mm, vma, prev, rb_link, rb_parent);
return 0;
}

If I apply the 2 patches above I can resolve my problems. For critical servers I use the "never overcommit" policy, because I hate when the OOM-killer always decide to terminate the most important application... :-) If the memory is never overcommitted OOM-killer is disabled and the applications can quit in a more graceful way; it's better to get a NULL from a *malloc()than get a SIGKILL from the kernel... :-)

first post on my blog...

2007-02-27T21:13:00.000+01:00

Today I decided to open this blog, just for test and fun. I'm not aware of any blog, I always played with forums, mailing lists, etc. but never with blogs, so now it's time to start. :-)

Following some useful links about me:

my personal website: Andrea Righi's forum
my favourite project: SystemImager
my favourite kernel: linux
the place where I work: CINECA