Ext4 defragmentation with e4defrag

[ Saturday, 31 May 2008, optimizationkit ]


Lately I have described ok_defrag — a simple tool for defragmenting Linux file systems. Today, I want to describe a specialized program designed to defrag the new Ext4 file system.

(For all who would like to say that Linux file systems don’t require defragmenting: Please redirect your great ideas to the Ext4 developers. They are the authors of this program — I do not have anything in common with it. As it will turn out, Ext4 is indeed quite different than other Linux file systems. Thanks in advance for your understanding :) )

To begin playing with e4defrag we must start compile of new kernel with patches needed to carry out the defragmentation. Below you will find a short description of that process, and if you would not like to use all those experimental patches for Ext4, then you should be interested in these seven only:

ext4-online-defrag-exchange-blocks-between-two-inodes.patch
ext4-online-defrag-relocate-file-data.patch
ext4-online-defrag-alloc-contiguous-blks.patch
ext4-online-defrag-ext4-defrag.patch
ext4-online-defrag-for-relevant-files.patch
ext4-online-defrag-check-for-freespace-fragmentation.patch
ext4-online-defrag-move-victim-files.patch

In first step we start from creation of directory, grabbing the kernel source, and downloading and unpacking needed patches.

mkdir linux-2.6-ext4
cd linux-2.6-ext4/
ketchup -G 2.6-git
wget http://www.kernel.org/pub/linux/kernel/people/tytso/\\
ext4-patches/2.6.26-rc2-git5-ext4-1/broken-out.tar.bz2
tar xjvf broken-out.tar.bz2
mv broken-out patches

Then we apply the patches on the sources.

quilt push -a

Then we start a process of configuring/building our new system kernel.

cp /our/beloved/.config .
cp /our/beloved/script/for/building/system/kernel/build.sh .

(on my PC it look like this:)

#!/bin/sh 

VER=`ketchup -m` 

make
sudo make modules_install
sudo cp arch/x86/boot/bzImage /boot/vmlinuz-$VER
sudo cp System.map /boot/System.map-$VER
sudo /sbin/new-kernel-pkg --mkinitrd --depmod --install $VER
)
make oldconfig
./build.sh

Meanwhile, when our new kernel compiles, we may open file: linux-2.6-ext4/patches/ext4-online-defrag-command.patch in our text editor of choice and we delete all lines from 1 to 18, and then from 1932 to 1935. Then, we copy our source code to other location.

cp linux-2.6-ext4/patches/ext4-online-defrag-command.patch e4defrag.c

And compile…

gcc -Wall e4defrag.c -o e4defrag

It is worth to grab some tools to handle the Ext4 file systems (the very latest version from the code repository).

git-clone git://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git
cd e2fsprogs
./configure
make
make install

After running our system with brand new kernel for the first time, we can finally start to test e4defrag. First we create and mount a test file system.

dd if=/dev/zero of=fs.img bs=256M count=1
/sbin/mkfs.ext3 fs.img

Now we must allow the use of experimental code on that partition.

/sbin/debugfs -w fs.img
debugfs 1.41-WIP (27-Apr-2008)
debugfs:  set_super_value s_flags 4
debugfs:  quit

Now our new file system is ready to be mounted

mount -t ext4dev -o loop,extents fs.img /mnt/loop0/

How to cause fragmentation at Ext4? – I have not yet tested any good way for fast fragmenting Ext4. This file system contrary to other Linux file systems is exceptionally resistant to fragmentation.

 /sbin/fsck.ext3 -nfv fs.img
e2fsck 1.41-WIP (27-Apr-2008)
First  course: checking i-nodes, blocks and sizes
Second course: checking directory structure
Third  course: checking communications directories
Fourth course: checking counters appeals
Fifth  course: checking the summary information about groups

     368 inodes used (0.56%)
       1 non-contiguous inode (0.3%)
         number i-nodes with blocks ind/dind/tind: 136/129/129
  202885 blocks used (77.39%)
       0 bad blocks
       0 large files 

     351 regular files
       8 directories
       0 character device files
       0 block device files
       0 fifos
       0 links
       0 symbolic links (0 fast symbolic links)
       0 sockets
--------
     359 files

It seems that Ext4 is a tough row to hoe, but its developers still deliver a defragmentation tool for it. Take a peep!

With use e4defrag we can defrag single files, for example:

e4defrag /patch/to/file

or all directories:

e4defrag -r /patch/to/directory/

also, there is a way to defrag a partition:

e4defrag /dev/sda1
e4defrag file_system_image.img

E4defrag contrary to other solutions – Con Kolivas defrag or ok_defrag is a true professional program designed to defrag Ext file system — it does not use a simple tricks with moving files, but it checks where on a drive the free blocks are located and then moves the data into those places.

A user, who will be capable to fragment Ext4, surely will be satisfied with the results of e4defrag. I am fully satisfied with the Ext4, for now I have no need to run e4d or even ok_defrag :) .

Subscribe to RSS feed for this article!

29 Comments

fold this thread Aaron  Sunday, 1 June 2008 o godz. 8:49 am #  Add karma Subtract karma  +1

As mentioned in my previous comment in your previous thread, fsck is only reporting inode fragmentation, *NOT* block fragmentation. Also, e4defrag is designed specifically for this use. Taking a look at one of the key features of ext4- extents, you will have an exceptionally difficult time fragmenting the inodes on this filesystem.

(Comments wont nest below this level)
 
fold this thread Giordano  Sunday, 1 June 2008 o godz. 6:48 pm #  Add karma Subtract karma  +2

In fact, if you have a look at the paper linked in the wikipedia ext4 article you can read these lines at the end:

To reduce system administrator’s effort, it is nec-
essary to automate defragmentation. This can be
realized by the following procedure.
1. The kernel notifies the occurrence of frag-
ments to user space.
2. The user space daemon catches the notifica-
tion and executes the defragmentation com-
mand.

This could really be a cool feature to have ^_^.
The effort required to the user to maintain an efficient system under the fs aspect would be basically cut down to 0 :)

(Comments wont nest below this level)
fold this thread ok  Sunday, 1 June 2008 o godz. 7:23 pm #  Add karma Subtract karma  --5

I will give you good advice – _never_ talk about Linux file systems fragmentation (unless you want to be burned at the stake).

 
 
fold this thread Giordano  Sunday, 1 June 2008 o godz. 6:49 pm #  Add karma Subtract karma  +0

There’s a spurious “in fact” in the beginning of my previous comment :$

(Comments wont nest below this level)
 
fold this thread Ben  Tuesday, 3 June 2008 o godz. 1:17 pm #  Add karma Subtract karma  +6

Nice article; thanks.
Regardless of whether the file systems fragment, at least you talk about it, and from what I can see everywhere, ext4 really is very good with not having much fragmentation.

(Comments wont nest below this level)
 
fold this thread scorp123  Tuesday, 3 June 2008 o godz. 10:05 pm #  Add karma Subtract karma  --17

LOL, more unqualified nonsense ….

(Comments wont nest below this level)
fold this thread A Real Professional  Sunday, 17 January 2010 o godz. 4:09 am #  Add karma Subtract karma  --1

So how many programs have YOU written “mr artsie administrator” scorp123/crap123 ? Let me guess, you never gotten past LOGO and MS-BASIC. You have no engineering background, and the according to your previous posts, the best you’ve been able to achieve in your career is to be a server adminstrator for TEN YEARS! In the high-tech if anyone stays in the same job for more than 5 years let alone 10 years, they are usually the most incompetent career-limited dead-enders i.e. career-limited for life. So I guess unqualified nonsense must be your forte.

 
fold this thread Frank  Wednesday, 10 March 2010 o godz. 2:03 pm #  Add karma Subtract karma  +1

you are my hero

 
 
fold this thread EdwardOCallaghan  Tuesday, 10 June 2008 o godz. 2:56 pm #  Add karma Subtract karma  --7

Errrm ..

ZFS .. OpenSolaris, OSX and FreeBSD Kernels have it..
Kernel land ZFS in Linux, get with the times Linus..

Give a crack;
http://www.opensolaris.com

Regards.

(Comments wont nest below this level)
 
fold this thread PolishJoke  Wednesday, 2 July 2008 o godz. 3:46 am #  Add karma Subtract karma  --12

You people are so very very high. Don’t even know where to start. RAID arrays die twice over before linux filesystems fragment. Desktops? User space? WTF are you people smoking.

(Comments wont nest below this level)
 
fold this thread Paul Evans  Wednesday, 17 December 2008 o godz. 10:31 pm #  Add karma Subtract karma  +0

Great article, guys.

The availability of ext4 and online disk fragmentation tools should alleviate this problem on open source UNIX implementations. Read my survey at

http://sharevm.wordpress.com/2008/12/16/435/

The Veritas file system supports online defragmentation on all platforms. ufs on Solaris also supports it.

(Comments wont nest below this level)
 
fold this thread UNF  Wednesday, 13 May 2009 o godz. 8:33 pm #  Add karma Subtract karma  --2

are there drivers so ext4 partitions can be r/w from HFS+ and NTFS ??

I want to dual boot OSes but keep my data on ext4.

Also, is there any need to defrag/align ext4 partitions on SSD ??

TYIA for any intel.

(Comments wont nest below this level)
fold this thread kiloalphaindia  Tuesday, 13 October 2009 o godz. 7:14 pm #  Add karma Subtract karma  +0

for NTFS you might want to use ntfs-3g – it is a fuse (filesystem in userspace) and thus can be used w/o any change to your kernel
- assuming your kernel supports fuse.

for SSD fragmentation is not an issue since the slowdown caused by fragmentation is a result of the seeks (hard disk head movement). An SSD does not do any seeks (no head :) ).

 
 
fold this thread Nick  Thursday, 24 September 2009 o godz. 12:43 pm #  Add karma Subtract karma  +1

Great article. Nice to see there are some people out there that can take an objective view.

Aparently, everyone else in the world has very large hard drives and/or very small files. As such, they do not see the same issues I do.

My files however, are very large (8G to 16G VM images) and my hard drive size is (comparatively) very small (160G) I occasionally experience a performance degradation that goes away after I employ an “IO cost reduction–don’t say the D word in mixed company.” strategy.

(Comments wont nest below this level)
 
fold this thread Luc  Thursday, 29 October 2009 o godz. 2:32 pm #  Add karma Subtract karma  +1

@kiloalphaindia

/quote

for SSD fragmentation is not an issue since the slowdown caused by fragmentation is a result of the seeks (hard disk head movement). An SSD does not do any seeks (no head ).

/end-quote

YES! finally someone who understands how a disk drive works! And why fragmentation is indeed an issue in ANY operating system!

Just as an extra note: it is not only the seek time for the heads, but also the rotational latency to consider as well… (for those who do not understand: the time it takes for a requested sector to appear under the read-heads, once the heads are on the right track)

This is not so much an issue if the read is sequential (i.e. a single contiguous stream) but as soon as jumps to various tracks (=fragmentation) are required this definitely becomes an additional factor!

The total seek time (access time) for a requested sector is:
software overhead to issue commands (very minute)
+ track seek time (considerable) (=accellerate + travel + decellerate + stabilize time)
+ rotational latency (=wait for sector to pass-by) (considerable, depends on drive rotation speed)
+ cache handling time on the drive (very small)
+ transport data time (very small).

(hence why SSD does not have the problem)

I can tell you from 30 years experience: optimizing definitely matters. Especially on high-load server systems. If you just read a text-file once in a while, then fragmentation wont matter indeed…

And I am not affraid to use the word “defrag”… Anyone who starts a flame war over this simply does not understand (or does not WANT to, even worse) how a hard disk works….

gr

(Comments wont nest below this level)
 
fold this thread David Schwartz  Saturday, 14 November 2009 o godz. 11:29 pm #  Add karma Subtract karma  +2

Anyone who things SSDs don’t need to be defragmented should take a look at the performance specifications of their favorite SSD. Specifically, they should compare the random write speed to the sequential write speed.

For example, take the Intel X25-E. Sequential writes are almost 200MB/s. Random writes are more like 35. For 128GB Indilinx MLC drives, the sequential write speed is around 130MB/s, but the random write speed is around 10MB/s.

If free space on an SSD is heavily fragmented, then writing new files to the SSD will be at close to the random write speed. If an SSD is not fragmented, then writing new files to the SSD will be at close to the sequential write speed.

Even read speeds vary somewhat. Sequential read speeds range from around 120 to 280MB/s. Random read speeds range from around 25 to 60MB/s. So even filled space benefits from defragmentation.

That said, the algorithms used to defragment data stored on SSDs should be very different from those used to defragment data stored on rotating media.

(Comments wont nest below this level)
fold this thread JohnDoe  Wednesday, 27 January 2010 o godz. 2:52 am #  Add karma Subtract karma  +1

You also have to take cell size into account, since its contents can only be written all at ones. Imagine the ssd-cell is 16k and your file system sector size is 4k. Means you have four logical blocks on a single ssd-cell. If you want to write to only one of these sectors, the ssd-controller would have to read the cell into a cache, modify it’s content in-memory and flush it back to the cell afterwards. You would have needed to do en extra read and actually re-write 3 data 3 times the size of your payload.

A Sequential write is of course much simpler: Just write it directly to disk!

Wear-Levelling is actually making these things somewhat more complicated, especially since manufacturers don’t disclose their individual algorithms.

 
 
fold this thread Alex Atkin UK  Friday, 22 January 2010 o godz. 6:39 pm #  Add karma Subtract karma  +2

I’m pretty sure when SSD specs claim sequential and random access they are referring to accessing a large file vs accessing a lot of smaller ones.

The reason there is any difference at all between requests is that accessing a large file is considerably less IO requests than accessing a lot of small ones, and its those requests for read/write operations that seem to clog up SSDs.

Fact is, data is not allocated how you think on an SSD anyway. They are designed to fragment the data intentionally due to the wear levelling process. The REAL killer for SSDs is low disk space, as the wear levelling demands that every write operation can be allocated to a different part of the drive (the least used). So if you have very little free space, it struggles to operate correctly. But that is also why defragmenting an SSD is pointless, you would just be jumbling the data at random around the drive wasting write cycles of the flash chips.

(Comments wont nest below this level)
 
fold this thread Marco  Saturday, 23 January 2010 o godz. 9:40 pm #  Add karma Subtract karma  +2

I just switched from ext3 to ext4 and ‘benchmarked’ a little. Here are the results before the transition:
Boot time reported by Bootchart: 45 seconds
Using VCD gear to export a 700 MB MPEG of a VideoCD image: 51 seconds

Then I did that online conversion with e2fstune:
boot: 43s
VCD: 31s
I was impressed by the write speed for large files but the boot time was unsatisfying. I tried e4defrag, but it did NOT add the ‘e’ attribute as reported by lsattr to all my files. So I made a backup of my data and formated the root partition.

With a fresh ext4 file system and all data copied back with rsync I got this result:
boot: 34s (It was 26% slower before)
VCD: 30s

Now that’s how I like it! Wether there WAS considerable fragmentation on my ext3 fs or some feature of ext4 wasn’t active after the online conversion, I don’t know and don’t want to join the flame war. But if someone asked me I would advise them to reformat.

(Comments wont nest below this level)
 
fold this thread David Turner  Friday, 16 April 2010 o godz. 4:03 pm #  Add karma Subtract karma  +2

I use a certain P2P system, and at some point I noticed that copying files to a flash drive would run at about 2MB/s, whereas copying side-to-side between two flash drives would run at about 20MB/s. This was a mystery until I tried running filefrag on various large data files. The number of extents was in the hundreds of thousands. After some cleaning up, the speeds improved to normal. So claiming that Linux filesystems aren’t affected by fragmentation is, frankly, bullshit. Of course this was ext3 – I’ve since upgraded to ext4 and am waiting with interest to see how it handles my usage patterns.

(Comments wont nest below this level)
 
fold this thread NoName  Friday, 28 May 2010 o godz. 12:11 am #  Add karma Subtract karma  +0

David I totally agree with you! When I want to get my shiny new Ubuntu I only use Transmission to save bandwidth on the server side and to be sure I got the error-free file (due to internal hash checking). I noticed that my ext4 performance goes down to 15-20% of nominal one when using – lets call it – non-linear download or video editing. When I make a copy of file after downloading and remove the original the fragmentation rate goes down, but I don’t feel like doing it every time I download something. Ext4 essentially needs e4defrag!

(Comments wont nest below this level)
 
fold this thread agaida  Sunday, 13 June 2010 o godz. 12:13 pm #  Add karma Subtract karma  +0

Yesterday if figured it out to me. Size does matter. It make sense to rebuild the partitioning under Ubuntu. Finaly i got a 10 G ‘/’, a 10G ‘/usr’, 100G ‘/home’ and ‘/tmp’ in tempfs. Then ‘defragment’ the system with a few copys like this:
-build a rescue system
-make a 100G partition ‘dirty’, mount as ‘/defrag’
- cp -ax /source/* /defrag
- rm -rf /source/*
- cp -ax /defrag/* /source
- rm -rf /target/*

In the end i have no fragmentation left and it speeds up the system a lot. But it will be nice to have a ext4defrag.

(Comments wont nest below this level)
fold this thread cmol  Monday, 20 December 2010 o godz. 3:54 pm #  Add karma Subtract karma  +0

@agaida, nice trick for small filesystems but i can’t imagine doing that with my 6TB data. i’d like a defrag tool :-) unless i do it sequentially, folder by folder eg 1TB at a time… will that work?

ps. i’m just an enthusiast/hobbyist, so forgive my ignorance :-)

interesting reading on the harddrive technologies.

 
 
fold this thread desherbeurs thermiques  Friday, 1 November 2013 o godz. 2:48 pm #  Add karma Subtract karma  +0

Quality articles is the crucial to invite the
viewers to pay a quick visit the web site, that’s what
this site is providing.

(Comments wont nest below this level)
 
fold this thread flexiblestorageplatform.mobi  Tuesday, 27 May 2014 o godz. 7:22 pm #  Add karma Subtract karma  +0

excellent issues altogether, you simply received a new reader.

What would you suggest in regards to your put up that you just made a few days in the past?
Any certain?

(Comments wont nest below this level)
 
fold this thread city bikes  Saturday, 9 August 2014 o godz. 6:49 pm #  Add karma Subtract karma  +0

We absolutely love your blog and find a lot of your post’s to be just what I’m looking for.
Do you offer guest writers to write content to suit your needs?
I wouldn’t mind producing a post or elaborating on a
few of the subjects you write in relation to here.
Again, awesome weblog!

(Comments wont nest below this level)
 
fold this thread Jayson  Tuesday, 12 August 2014 o godz. 12:46 am #  Add karma Subtract karma  +0

Remarkable things here. I’m very glad to look your article.

Thanks a lot and I’m having a look ahead to contact you.
Will you please drop me a e-mail?

(Comments wont nest below this level)
 
fold this thread video games  Thursday, 18 September 2014 o godz. 10:23 pm #  Add karma Subtract karma  +0

Excellent post. I’m going through some of these issues as well..

(Comments wont nest below this level)
 
fold this thread 生活禁忌  Sunday, 28 September 2014 o godz. 9:35 am #  Add karma Subtract karma  +0

好久不来,过来看看你的博客

(Comments wont nest below this level)
 
Name (required)
E-mail (required - never shown publicly)
URI

Adjust field size: shrink | enlarge)


You can use simple HTML in your comments. Some examples are as follows:
  • A hyperlink: <a href="polishlinux.org">GNU/Linux for everyone!</a>,
  • Strong text: <strong>Strong text</strong>,
  • Italic text: <em>italic text</em>,
  • Strike: <strike>strike</strike>,
  • Code: <code>printf("hello world");</code>,
  • Block quote: <blockquote>Block quote</blockquote>

New AdTaily ads!

Are you a film buff?

film buffs community, movie recommendations and reviews

RSS: Comments

You can follow the comments to this article through a special channel RSS 2.0 .

Related articles: CLI

 more »

PolishLinux Top Content


Become our fan on Facebook!

PolishLinux.org on Facebook

Follow PolishLinux on Twitter!

Follow polishlinux on Twitter

Google Ads