TIP Speeding up portage with tmpfs

From Gentoo Linux Wiki

[edit] Introduction

Compiling from source takes a long time. Tweaking around let me compile xorg-x11 (-minimal!) within about 15 minutes on 1800+ AMD CPU

[edit] How is that possible?

When emerging a package, Portage creates many temporary files in ${PORTAGE_TMPDIR} (default: /var/tmp) and transfers a lot of temporary data to/from the filesystem. With a tmpfs mounted on ${PORTAGE_TMPDIR}, the temporary data is kept in RAM, which is much faster than the hard disk. This speeds up the compilation, avoids hard disk fragmentation and avoids hard disk wearing.

[edit] Requirements

One should have at least 512 MB RAM to emerge a Desktop Environment like KDE or GNOME. Some especially troublesome packages such as Firefox require even more memory.

Kernel support for tmpfs is also required.

File systems -> Pseudo filesystems -> Virtual memory file system support (former shm fs)

[edit] tmpfs size requirements

A 300 MB tmpfs is not enough to compile:

• GCC

450 MB is not enough to compile:

• Xorg-x11

500 MB is enough for:

• GCC (without any USE flags enabled)
• Wine

Compiling www-client/mozilla-firefox-3.0 with no USE flags enabled and an empty LINGUAS variable took 662 MiB.

850 MB is not enough to compile:

• Open Office

850 MB is enough for:

• Xorg-x11
• qt-4.1.2

Compiling sys-devel/gcc-4.3.1-r1 with USE flags fortran gcj gtk mudflap openmp enabled took 1606 MiB

Greater than 1.5 mounted GB is needed for wxGTK. You may also need to adjust the number of inodes.

It is possible to compile KDE in 350 MB if you compile X11 and QT seperately.

openoffice needs a lot of memory, and at least with 3.4G mounted, 2G physical, its 10 min slower than without tmpfs.

With big applications there is also problem with number of inodes. One inode = one file and source code have big number of small files. For big applications you must increase number of inodes.

[edit] Set-up

As root, run this:

mount -t tmpfs tmpfs /var/tmp/portage

mount -t tmpfs tmpfs -o size=850M,nr_inodes=1M /var/tmp/portage

Check if it's mounted with

df -h | grep tmpfs

You can also mount tmpfs at boot by adding this to /etc/fstab :

none                    /var/tmp/portage   tmpfs  size=1000M,nr_inodes=1M         0 0

You can use bigger size, if you have big swap. On my machine I have 5 GB of swap, one 1 GB tmpfs mounted in /var/tmp/portage and 3 GB in /tmp. Tmpfs swapping is very fast.

[edit] Testing

Well, emerge something!

# emerge xorg-x11

Use genlop to check whether it's faster or not.

tmpfs:???
USE="=???"
emerge genlop -n
genlop -t xorg-x11
* x11-base/xorg-x11

Wed Apr  6 17:33:07 2005 >>> x11-base/xorg-x11-6.8.2-r1
  merge time: 7 minutes and 9 seconds.

tmpfs: 1024M
USE="-3dfx -3dnow +bitmap-fonts -cjk -debug -dlloader -dmx -doc -font-server -insecure-drivers \
     +ipv6 -minimal -mmx +nls -nocxx +opengl +pam -sdk -sse -static +truetype-fonts \
     +type1-fonts (-uclibc) -xprint +xv"
emerge genlop -n
genlop -t xorg-x11
* x11-base/xorg-x11

Fri Feb 10 10:45:02 2006 >>> x11-base/xorg-x11-6.8.2-r6
       merge time: 35 minutes and 19 seconds.

[edit] Script

A small script that mounts tmpfs for you. Go to any binary-dir on your $PATH, like /usr/local/bin, and create temerge with your $EDITOR.

#!/bin/bash
MEMSIZE=850M
mounted=false
 
. /etc/init.d/functions.sh
 
mounttmpfs() {
     mount -t tmpfs tmpfs -o size=$MEMSIZE /var/tmp/portage
     mounted="true"
}

compile() {
     einfo "emerging ${*}"
          emerge ${*}
}

unmount() {
     ebegin "unmounting tmpfs"
          umount -f /var/tmp/portage
     eend $?
}

ebegin "Mounting $MEMSIZE of memory to /var/tmp/portage"
if [ -z "$(mount | grep /var/tmp/portage)" ]
then
     mounttmpfs
else
     eerror "tmpfs already mounted!"
     exit 0
fi
eend $?

compile ${*}
 
if [ -n "$mounted" ]
then
     unmount
fi

chmod u+x temerge and use temerge instead of emerge for emerging. Use emerge for fetching/pretending.