Playing around with linux kernel

No distribution

I have wondered if you could run vanilla Linux kernel all on its own without any kind of distribution. I knew about Linux from scratch, which is basically a tutorial on how to create your own Linux system. I haven’t worked my way through that book, but it’s a great way of getting to know the internals of a Linux system. I didn’t use that book because I prefer to tinker around on my own. It was still very useful resource to look things up sometimes.
When I tried to find information about running the bare kernel, the most fequent reaction was “why on earth would you want to do that”. The information seemed to suggest that the kernel could probably boot but would then complain about not being able to run the init process. That sounded good enough for me to go and try.

Why?

First of all it’s a great way of learning about the Linux kernel. I ended up spending about 3 days and learned a ton of stuff that I would probably never learn from just the user side. Also I’m studying robotics in university and being familiar with the Linux kernel will definitely be useful at some point.
Further down the road I would like to build something from scratch on real hardware. For example running custom kernel on some old android phone. I doubt it’s doable to get all of the phone hardware running because a lot of the firmware is closed source. Getting the CPU and some generic devices working shouldn’t be a problem. Modern phones have so much hardware in them, imagine buying a cheap used smartphone and using it for a robotics prototype. It would be so much cheaper than a dev kit and sensor modules. I don’t know how realistic it is, but that’s my ultimate goal with this.

How far did I get?

The most natural start seemed to be a x86 virtual machine. I got the kernel working on both VirtualBox and QEMU virtual machines. The kernel refused to boot without a bootloader, so I had to use Syslinux (ISOLINUX). Then I added BusyBox to have a shell and some basic commands like mount/cat/vi/echo and so on. With BusyBox I could even configure a network interface and connect to the internet. Obviously I didn’t have a browser, but I could download HTML with telnet. Eventually I wrote the thing on a USB stick and even got my laptop to boot the kernel and recognize my WIFI card. Doesn’t seem like much but it was a lot further than I expected to get.

Documentation of the process

First we need the kernel. You can download kernels from kernel.org

# first set up a new directory for everything
mkdir kernelbuild
cd kernelbuild
# download and pipe to tar
curl -L https://git.kernel.org/torvalds/t/linux-4.19-rc7.tar.gz | tar xvz
cd linux-4.19-rc7
# set arch to x86
arch=x86
# you can replace with x86_64_defconfig, I needed 32bit for virtualbox (no virtualization stupport on laptop)
make i386_defconfig
# enable all the features and drivers you need
make menuconfig
# if you're mssing some dependencies you're gonna have to install them here (https://www.kernel.org/doc/html/v4.15/process/changes.html)
make

Once you’ve done all this you should have a kernel image kernelbuild/linux-4.19-rc7/arch/x86/boot/bzImage
I’m not sure if it’s possible to convince the kernel to boot on it’s own. If I tried the kernel just printed “Use a bootloader”. Thus I used Syslinux.

cd ..
curl -L https://mirrors.edge.kernel.org/pub/linux/utils/boot/syslinux/Testing/6.04/syslinux-6.04-pre1.tar.gz | tar xvz
cd syslinux-6.03
make clean
make
cd ..
# let's start constructing a .iso image
mkdir iso
cd iso
# see https://www.syslinux.org/wiki/index.php?title=ISOLINUX
mkdir isolinux kernel images
cp ../syslinux-6.04-pre1/bios/core/isolinux.bin isolinux/
cp ../syslinux-6.04-pre1/bios/com32/elflink/ldlinux/ldlinux.c32 isolinux/
cp ../syslinux-6.04-pre1/bios/memdisk/memdisk images/
cp ../linux-4.19-rc7/arch/x86/boot/bzImage kernel/
# edit these to match below
touch isolinux/isolinux.cfg isolinux/boot.txt
cd ..

isolinux.cfg

display boot.txt
prompt 1
default 1
edd off

label 1
    kernel /kernel/bzImage
    append initrd=/images/init.igz edd=off console=ttyS0 console=tty0

boot.txt

  
 1 - Boot your awesome kernel

Just to see where we’re at we can create the iso and boot it.

# first add comment character infront of "append initrd=..." line in isolinux.cfg, otherwise bootloader won't let us boot
# don't forget to remove it later
mkisofs -o system.iso -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table ./iso
# apt install qemu (on debian/ubuntu) if you don't have QEMU
# this is 32bit emulator
qemu-system-i386 -cdrom ./system.iso

You should see that the kernel tries to boot up, but couldn’t mount rootfs. Last line in kernel log:

[   3.673814] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---

Basically Linux needs a filesystem to work with. Normally this would be the main HDD/SSD partition on your computer. That’s not the whole story though. To mount the real rootfs modern Linux systems also use a temporary filesystem that is stored in RAM. That temporary filesystem should contain everything needed to get the main one up. Including utilities like mount, firmware, kernel modules etc.
I won’t go that far in this article. I’ll just use the temporary filesystem as rootfs for now.
Long story short: the missing link is initramfs. We already specified where it’s gonna be located with initrd=/images/init.igz
The general idea is the same as with the iso: we create a directory and then turn that directory into an image file.

mkdir initramfs
cd initramfs
mkdir bin dev etc lib proc sbin sys usr var
# we only make the absole minimum amount of directories and files (and probably quite a bit less) to boot
touch init
# needs to be executed (doesn't matter here, beceause it's empty)
# without the empty init program the kernel doesn't recognize the filesystem
chmod 755 init
cd dev
# now we need some essential device files
# for that we use mknod [path] [c (char) / b (block) / p (pipe/fifo)] [major] [minor]
# more info https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/devices.txt
mknod null c 1 3
mknod zero c 1 5
mknod tty0 c 4 0
mknod tty1 c 4 1
mknod tty2 c 4 2
mknod tty3 c 4 3
mknod tty4 c 4 4
mknod ttyS0 c 4 64
mknod ttyS1 c 4 65
mknod ttyS2 c 4 66
mknod ttyS3 c 4 67
cd ../etc
echo "root:x:0:0:root:/root:/bin/sh" > passwd
touch group
cd ../..

This should be enough to hit the next snag. Now let’s turn this into a valid initramfs image. Initramfs is usally stored in a gzipped cpio archive. First we create the cpio archive and then we gzip it. Let’s create a bash script of the whole process so we don’t have to type this multiple times.

touch make_iso.sh
# now use vi, nano, gedit or whatever

make_iso.sh

# create cpio archive
cd initramfs
find . | cpio -H newc -o > ../initramfs.cpio
cd ..
# gzip it
cat initramfs.cpio | gzip > initramfs.igz
# copy to iso
cp ./initramfs.igz ./iso/images/init.igz
# remake the iso 
mkisofs -o system.iso -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table ./iso

Let’s try again.

source make_iso.sh
qemu-system-i386 -cdrom ./system.iso

This time:

[    4.681241] ---[ end Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance. ]---

This is where we’re gonna need BusyBox. Initially I built BusyBox with gcc-i686-linux-gnu cross compiler and I had some trouble with programs that use libc DNS lookup. The problem seemed to be that glibc needs more of the networking ifrastructure to be there, like nsswitch. I found out that by using a uClibc toolcain I could have ping working with just BusyBox. Also the BusyBox binary was 2x smaller with uClibc. That’s why I’m gonna use an uClibc toolchain to compile BusyBox.
I found the easiest way to do that was by using Buildroot. In fact Buildroot could do everything we have done so far for us. But where’s the fun in that?

curl -L https://buildroot.org/downloads/buildroot-2018.02.6.tar.bz2 | tar xvj
cd buildroot-2018.02.6/
# make sure the arch is right
# uClibc should be the default libc
# you can disable as much of the other stuff as possible (besides toolchain) to improve compile time
make menuconfig
make
cd ..
# add the built binaries to $PATH
# your diretory might be different
# the variable only stays in your current shell
PATH=~/kernelbuild/buildroot-2018.02.6/output/host/bin:$PATH

Now get BusyBox

curl -L https://busybox.net/downloads/busybox-1.29.3.tar.bz2 | tar xvj
cd busybox-1.29.3
# set Settings->Build static binary (no shared libs)
# set Settings->Cross compiler prefix to i686-buildroot-linux-uclibc-
# set Settings->Destination path for 'make install' to /home/youruserhere/kernelbuild/initramfs (or wherever your initramfs directory is, using ~ doesn't work)
make menuconfig
make
make install
cd ..

This should populate your initramfs with BusyBox binary and create links to it. Now we need to use BusyBox init as init process. For that we simply create a symlink from /init to /bin/busybox. Also we need to create inittab, which tells BusyBox what to do on init.

cd initramfs
rm init
# create symbolic link
ln -s ./bin/busybox init
touch /etc/inittab
# now edit it
cd ..

inittab

::askfirst:/bin/sh
::ctrlaltdel:/sbin/reboot
::shutdown:/sbin/swapoff -a
::shutdown:/bin/umount -a -r
::restart:/sbin/init

# optional serial port console
#::respawn:/sbin/getty -L ttyS0 9600 vt320

Now let’s try again.

source make_iso.sh
qemu-system-i386 -cdrom ./system.iso

This is what you should see:

[    3.694335] Write protecting the kernel read-only data: 2656k
[    3.695322] Run /init as init process
[    4.088901] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3
Please press Enter to activate this console.
/ # _

You’re in a root shell to your virtual machine. You could set a password with passwd if you wanted. The filesystem is stored in RAM so nothing persists yet though.

I’ll leave the pieces of how I got ethernet working for documentation purposes:

qemu-system-i386 -cdrom ./system.iso -netdev user,id=mynet0,net=192.168.76.0/24,dhcpstart=192.168.76.9 -device e1000,netdev=mynet0
# drivers in kernel menuconfig : Device Drivers > Network device support > Ethernet driver support > Intel(R) PRO/1000 Gigabit Ethernet support
# requires PCI to be enabled

Necessary initramfs files

touch etc/network/interfaces
mkdir var/run
mkdir etc/network/if-up.d etc/network/if-down.d etc/network/if-post-down.d etc/network/if-pre-up.d
touch /etc/resolv.conf

etc/network/interfaces file:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

etc/network/resolv.conf

# Google DNS
nameserver 8.8.8.8
nameserver 8.8.4.4

Bringing eth0 up

mount -t sysfs sys /sys
mount -t proc proc /proc
ifup -a
# normally you would have a script in usr/share/udhcpc/default.script to do this automatically
ifconfig eth0 192.168.76.9 netmask 255.255.255.0 up
route add default gw 192.168.76.2
# should work
ping -c 3 tammearu.eu

If you want to make this work on real hardware you either burn the iso to CD or DVD. To make usb stick work you need to use isohybrid on the iso first and then use dd or something else to get it on the usb stick (just copying the files won’t work).

Final words

I’d like to emphasize that this was just for learning purposes. I doubt doing everything by hand is the most efficient approach in real life. There are tons of tools out there for doing all this, but I think it’s useful to know how it works under the hood.