PS2Linux memorycard main-memory maximisation tutorial (4m-cloop tutorial)

This tutorial depends on the no-hdd tutorial available at

http://playstation2-linux.com/download/nohdd/ps2nohdd.html

This tutorial does _not_ tell you how to exchange libc with a smaller library. However you can still apply the tutorial at

http://playstation2-linux.com/download/nohdd/uclibnohdd.htm

The problem:

As you probably have noticed, there there are a few disadvantages with using ramdisk-images. The main problem is, that the ramdisk uses up a lot of main memory, because it is being uncompressed into the main memory. Secondly the ramdisk-size is fixed, which leaves you with wasting some main memory if you do not recreate it every time. Thirdly the free space of the ramdisk might be filled with trash, because you deleted some files in it, which now compresses badly and wastes some space on your memorycard.

If you do not yet see the problem with ramdisks, read the following:
Let's assume you are a c++ programmer and use the libc and the stdc++ library from PS2linux to run your c++ programs. These two libraries combined with some startup-utilities will probably sum up to about 16MB. In compressed state this image uses up approximately 4MB on your memory-card. This means that if your kernel is 1.5MB big, you'll have 8MB-4MB-1.5MB = 2.5MB left on your Memorycard. And since the this image is uncompressed into main memory and the kernel is also loaded into main memory, you are left with 32MB-16MB-1,5MB = 14,5MB of main memory for your program.

Now consider that your were writing a bigger game which uses 16MB gamedata. That means that the image will be 16MB + 9MB (environment+libc) = 25MB big. The compressed image of this will be about 6.25MB. 6.25MB + 1.5MB (kernel) = 7.75MB of your memorycard. Now at startup the image gets uncompressed into ram: 32MB-25MB-1.5MB = 5.5MB of main-memory for working. And 5.5MB main memory are really not much if you use textures.

The solution:

Fortunately there is a kernel-module called "cloop", which allows mounting a compressed iso-image. This means that we can simply mount an image on the memcard, instead of uncompressing it into ram. However this kernel-module is not in our ps2linux-distribution, because it was created for Knoppix, a cd-only linux-distribution.

You can either download the precompiled binaries for it (and skip the next section), or compile from source.

Compiling cloop from source on PS2Linux:

Download the tarball from

http://developer.linuxtag.net/knoppix/sources/cloop_0.68-5.tar.gz

tar xfvz

Now we need to "backport" from 2.2.x to 2.2.1, because one function name has changed.
Note: Only do this step if you are using the original kernel of the kit. Skip it if you use the xrhino or the bb-navigator kernel for your memory-card.
Edit compressed_loop.c and replace all 3 occurences of "filp_close" with "close_fp".
Backporting done.

Compilation is a bit trickier, because Knoppix and cloop are designed for i386. Since mips are different, you have to do two runs of compilation: one for the the module and one for the executables. You cannot use the object-files of one run for the other.
Move into the directory zlib-1.1.4 and delete line 61 to 71 from "Makefile".
This prevents zlib from compiling its binaries, which we do not need anyway.

Then we need to set up the main-Makefile for the first run: the module.
Edit "Makefile" in cloop-0.68 and replace line 5 with the following (write all into one line):

CFLAGS:=-Wall -Wstrict-prototypes -O2 -I. -G 0 -mno-abicalls -fno-pic -fomit-frame-pointer -mcpu=5900 -mips1 -pipe -mlong-calls -DMODULE

all: zcode cloop.o

Now to the second run: the executables.
Edit the "Makefile" in cloop-0.68 again and alter line 5 so that it reads:

CFLAGS:=-Wall -Wstrict-prototypes -O2 -I. -G 0 -fomit-frame-pointer -mcpu=5900 -mips1 -mlong-calls

all: zcode create_compressed_fs compressloop

The new file hierarchy on the memorycard:

Unfortunately we cannot totally get rid of the initrd. We need "mount" and "insmod" on it and therefore also "libc" and "libld", because we need to insert a few modules before we can mount the cloop-romimage onto the initrd.
When the future memorycard will have booted, it will look like the following:

• The initrd is still the root-filesystem. /dev/ and such will still be on the initrd.
• The memorycard will be mounted at /mc00/ (however you will never directly use that mount)
• The cloop-romimage will be mounted at /usr/

This means that you can create bin, sbin and lib directories on the romimage, and the kernel will automatically find the files in them, because he sees them as /usr/bin, /usr/lib and /usr/sbin.

Also all your game-files should be in subdirectories of the romimage.

The new initrd:

As I mentioned before, we cannot work with the romimage directly after booting, because we do _not_ have access to it: Our romimage will be stored directly on the memorycard. This means we need to mount the memorycard after booting. And in order to mount the memorycard, we need to load the memorycard-kernel-modules first.

To keep it short: we need the following files on our _initrd-image_. Please make your "/root/initd/" look _exactly_ like that.

(Since the files in the initrd-image will only consume 9MB and will not be touched anymore, you should create a new ext2-initrd-image with 9MB, as described in the no-hdd tutorial.) (You can move some of the items over with "cp -R -d <source> <dest>".)

• bin/
• "mount" - For mounting the memorycard and the romimage
• "ash" - For executing mount and insmod in a script
• "sh" - softlink to "ash"
• sbin/
• "init" - For starting up correctly.
• "insmod" - For loading the memorycard-modules and the cloop-module
• lib/
• "ld-2.2.2-so" - needed by ALL executables
• "ld.so.1" - softlink to "ld-2.2.2-so"
• "libc-2.2.2-so" - needed by ALL executables
• "libc.so.6" - softlink to "libc-2.2.2-so"
• modules/ (create the directory)
• "cloop.o" - for loading the romimage
• "isofs.o" - for loading the romimage
• "ps2mc.o" - for mounting the memcard
• "ps2mcfs.o" - for mounting the memcard
• etc/
• "inittab" - the file for controlling starup (put the following 3 lines into it)

  id:1:initdefault:
  si::sysinit:/startscript
  sh:1:respawn:/bin/sh
  

• dev/
• "MAKEDEV"
• all the devicefiles as they were before
• "cloop" - the devicefile for cloop. Create it with "mknod cloop b 240 0"
• "ps2mc00" - the devicefile for the memcard. Create it with "mknod ps2mc00 b 245 0"
• mc00/ (empty directory as mountpoint for memcard)
• usr/ (empty directory as mountpoint for romimage)
• proc/ (empty directory as mountpoint for proc)

Then we need a script, which does all the mounting-stuff for us.
Create the file "/root/initrd/startscript" and fill it with the following:

#!/bin/sh
#load the modules for memcard access
insmod ./modules/ps2mc.o
insmod ./modules/ps2mcfs.o

#mount the memorycard readonly in order to access the image
mount -t ps2mcfs /dev/ps2mc00 /mc00 -o ro

#load the modues for romimage access
insmod ./modules/isofs.o
insmod ./modules/cloop.o file=/mc00/romimage

#mount the romimage to /usr
mount -o ro /dev/cloop /usr

#--invoke the startscript in romimage--#
cd /usr
./startscript

#--end of file--

Now we are done with the initrd.
Compress the image and move it to the memorycard as shown in the no-hdd tutorial.

Creating the romimage:

Starting with the romimage is simple: just create a directory and put everything you need into it.

mkdir /root/romimage

• /bin
• "ls"
• "rm"
• /lib
• "libm-2.2.2.so"
• "libm.so.6" - softlink to "libm-2.2.2.so"
• "libpthread-0.9.so"
• "libpthread.so.0" - softlink to "libpthread-0.9.so"
• "librt-2.2.2.so"
• "librt.so.1" - softlink to "librt-2.2.2.so"
• "libtermcap.so.2.0.8"
• "libtermcap.so.2" - softlink to libtermcap.so.2.0.8"
• "libstdc++-libc6.2-2.so.3"
• /modules (directory for other kernel-modules)

#!/bin/sh

#--end of file--

Compressing the romimage:

Basically everything _should_ work with the following command (currentdir == /root/):

mkisofs -r romimage | create_compressed_fs - 65536 > /mnt/mc00/romimage

What to do now? Well, we need to put something to the end of the image that does not compress very well and never gets read - a few kilobytes from /dev/random for example.

Unfortunately it is quite hard to predict where mkisofs will put the random file _inside_ the image, so we will simply put it _outside_ of the image by appending the random file to the iso-image via commandline-pipeing.

So first we need to create a random file which is not smaller than the blocksize (65536 bytes in our case).
The problem is that the throughput of /dev/random is less than 1 byte/sec on PS2Linux. - We would be generating random numbers for a day or two. This is why we take the pseudo-random device /dev/urandom instead. (These random numbers are weak for encryption, however the compression algorithm won't notice the difference.) You can create the file with the following command:

dd if=/dev/urandom of=64k_random ibs=1 obs=65536 count=65536

After you have the random file in the directory /root/, you can create the compressed image by using the following commands (currentdir == /root/):

mkisofs -r romimage > tmpimg
cat tmpimg 64k_random | create_compressed_fs - 65536 > /mnt/mc00/romimage

That's it! Have fun!

Remember the example with the 16MB of gamedata? The amount of memory used on the memcard stayed the same, but instead of 5.5MB, you now have 32MB-9MB-(16MB/4)-1.5MB = 17.5MB of main memory available for working. (And if you want to have even more space available on the memorycard and in main memory, then you should read the tutorial on replacing libc with ulibc.)

Mr.M