Defragmentation of Linux Filesystems

[ Friday, 23 May 2008, optimizationkit ]


There is a myth that “linux filesystems don’t need to be defragmented.” As it may be truth in general, it still can be dispelled by a simple script, which creates a certain number of directories and in each of them creates and deletes a certain number of files – in, let’s say, two passes. So, does your filesystem need defragmentation?

(!!!WARNING!!! In this article we use the fsck.ext3 program to check the degree of filesystem fragmentation – before using this program on your own better get familiar with its documentation. An unskilled use of fsck on a mounted filesystem can cause data failure. !!!WARNING!!!)

Let’s make an experiment – we’re going to start with creating a filesystem image:

dd if=/dev/zero of=filesystem.img bs=256M count=1

Next we’re going to create a new filesystem (I suggest ext2 or ext3, because it’s easy to obtain information about the degree of filesystem fragmentation for those).

/sbin/mkfs.ext3 filesystem.img

Now we can check the fragmentation degree of the newly-created filesystem:

/sbin/fsck.ext3 -nfv filesystem.img
e2fsck 1.40.4 (31-Dec-2007)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

11 inodes used (0.02%)
1 non-contiguous inode (9.1%)
number of inodes with blocks ind/dind/tind: 0/0/0
18561 blocks used (7.08%)
0 bad blocks
0 large files

0 regular files
2 directories
0 character device files
0 block device files
0 fifos
0 links
0 symbolic links (0 fast symbolic links)
0 sockets
--------
2 files

The information about the fragmentation degree contains the line:

1 non-contiguous inode (9.1%)

This is how the situation looks like on a “fresh” filesystem. It’s very easy to simulate how it would look like on an frequently used filesystem (on which we delete and save new files a lot). Let’s execute a the frag.sh script

It’s enough to mount a test-filesystem-image:

mount -o loop filesystem.img /mnt/loop0/

and next give to the script the directory name, in which the filesystem image is mounted:

frag.sh /mnt/loop0/

The script will show many notifications, like:

Creating file /mnt/loop0//8/file-128
28444+0 records read
28444+0 records saved
85332 bytes (85 kB) copied, 0,173525 s, 492 kB/s
step = 6
Deleting file /mnt/loop0//8/file-0

Next we have to unmount our test-filesystem:

umount /mnt/loop0/

and again check the fragmentation degree:

/sbin/fsck.ext3 -nfv filesystem.img

Because the script creates randomly sized files and deletes randomly chosen files, the data can significantly differ from the given below

e2fsck 1.40.4 (31-Dec-2007)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

895 inodes used (1.37%)
315 non-contiguous inodes (35.2%)
number of inodes with blocks ind/dind/tind: 735/0/0
77202 blocks used (29.45%)
0 bad blocks
0 large files

875 regular files
11 directories
0 character device files
0 block device files
0 fifos
0 links
0 symbolic links (0 fast symbolic links)
0 sockets
--------
886 files

As you can see — 35.2% of the inodes are not placed continuously — the fragmentation degree of files in our image is very high.

To have a point of reference for further experiments I suggest to make a backup of this image:

cp filesystem.img filesystem-backup.img

The ok_defrag program that we’re going to use for defragmenting can be downloaded from here.

In order to for the program to work, the python-dialog package is required, so install it using your package manager of choice.

The working principles of this program are based on the ones shown in Con Kolivas’ defrag. However, ok_defrag doesn’t fumble at the filesystem’s data structures, as professional programs like Diskeeper or Windows Defrag do.

The program is to be executed as follows:

ok_defrag -l log.txt -d /mnt/loop0/

The flags -l and -d are obligatory. The log of the program work will let us recover data, if something goes wrong, e.g. if the power-supply would fail during the job. The -d flag shows the directory with files, which we want to defragment.

First ok_defrag creates a list of directories, which are placed in the pointed directory — the list is sorted by directory size – from the largest to the smallest. Next for each directory of the list, the program creates a list of files, which are placed in those directories – it’s also sorted by size, from the largest to the smallest. Such a file sorting method should provide the best results during defragmentation.

Each file is copied to the /tmp/ok_defrag_tmp file, and then the content of this file is moved to the original location – such a simple method repeated a few times for each file (3 times by default) should provide the desired result. Let’s see, how it works out.

We have to mount our test-filesystem again:

sudo mount -o loop filesystem.img /mnt/loop0/

Then we run ok_defrag:

ok_defrag -l log.txt -d /mnt/loop0/

Now we have to wait a while (!!!WARNING!!! The defragmentation process should not be interrupted !!!WARNING!!!)… When the process finishes, we can unmount the filesystem:

umount /mnt/loop0/

and check the final result:

/sbin/fsck.ext3 -nfv filesystem.img
e2fsck 1.40.4 (31-Dec-2007)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

895 inodes used (1.37%)
84 non-contiguous inodes (9.4%)
number of inodes with blocks ind/dind/tind: 735/0/0
77202 blocks used (29.45%)
0 bad blocks
0 large files

875 regular files
11 directories
0 character device files
0 block device files
0 fifos
0 links
0 symbolic links (0 fast symbolic links)
0 sockets
--------
886 files

It was able to decrease fragmentation level from 35.2% to 9.4%. Another run of the process should decrease the fragmentation level even more. Instead of running ok_defrag a couple of times, you can add a -f flag to the program command line — then the defragmentation process will be repeated a specified number of times.

Opposite to Con Kolivas’ defrag, ok_defrag has a paranoic approach to data safety. Each defragment operation is divided into the following stages:

  • writing down the time of the last file modification,
  • writing down the file’s checksum,
  • copying the file to /tmp/ok_defrag_tmp,
  • testing the checksum of the /tmp/ok_defrag_tmp file,
  • comparing both checksums — if they differ from each other, it means you have problems with your hardware or our system — the defragmentation process for this file is interrupted,
  • another testing of the modification time for the original file — if it has been modified after the copy had been made, the defragmentation process for this file is interrupted,
  • if all of the above stages are completed successfully, the content of the /tmp/ok_defrag_tmp file is moved to the original file,
  • another testing of the file’s checksum – if the checksum differs from the original, it means something went really wrong – the program terminates.

An attentive user will notice that on the seventh stage, when moving the content of /tmp/ok_defrag_tmp to the original file, /tmp/ok_defrag_tmp is not being blocked for writing to other programs. It’s a defect, which (I hope) can be eliminated, because it hinders the defragmentation process in some directories, in which the data is still being written, e.g. logs.

If something during the data defragmentation process went wrong, e.g. unexpected power failure occurred – you need to run fsck on the filesystem. Filesystems with journaling should handle this problem – because ok_defrag doesn’t really make any abracadabra, but only moves files.

But if a file got damaged, you need to check the log – there is information about the performed tasks and checksums of files from all stages.

Checking /mnt/loop0/8/file-91 md5sum
md5sum = b8b828bda1b12173f8f8b0b87d8cd872
Copying /mnt/loop0/8/file-91 to /tmp/ok_defrag_tmp
Done.
Checking /tmp/ok_defrag_tmp md5sum
md5sum = b8b828bda1b12173f8f8b0b87d8cd872
Moving /tmp/ok_defrag_tmp to /mnt/loop0/8/file-91
Done.
Checking /mnt/loop0/8/file-91 md5sum
md5sum = b8b828bda1b12173f8f8b0b87d8cd872

If, e.g. a log breaks at “Moving /tmp/ok_defrag_tmp to /foo/bar/bas” it means, the file hasn’t realy been fully moved — its copy is placed in /tmp/ok_defrag_tmp — you just need to replace the file.

Some systems have a functionality to remove files located in /tmp during bootup. The best thing would be to turn it off for the time of defragmentation.

To check the effects of defragmentation the best thing is to use seekwatcher (it requires blktrace, CONFIG_BLK_DEV_IO_TRACE and CONFIG_DEBUG_FS options in kernel, and python-matplotlib with its dependencies).

A simple test:

seekwatcher -t find.trace -o find.png -p
'sync; echo 1 > /proc/sys/vm/drop_caches; for file in `find /mnt/loop0/ \
-type f`; do cat $file > /dev/null; done '
-d /dev/

shows the difference in file reading speed between a fragmented filesystem (filesystem-copy.img)

and a defragmented filesystem (filesystem.img)

I suggest to perform the defragmentation of directories, where the system can write data, after the system bootup to /bin/sh — parameter init=/bin/sh in the kernel command line. This should guarantee a trouble-free operation without worries about any data failure.

When is there no need to perform a defragmentation? Generally speaking, if fsck returns non-contiguous inode below 10%.

How long does the defragmentation process take? It depends on the disk speed, the amount and size of files — each file is copied and moved several times, so this operation can take a very long, long time if you have thousands of files.

I also have to remind of one more thing – it’s worth to check, how much free space we have on /tmp — the lack of free space won’t cause file damage — the checksums are tested the whole time — but it’s still worth doing. Remember that the defragmented disk should always have a reasonable amount of free space. If this is always the case, it may happen that you indeed will never need to perform the process. But it’s still good to be aware of the fact that you may occassionally do it on Linux as well.

This is a translation of the article Defragmentacja linuksowych systemów plików. Translated by Adam Dziuba. Proof-read by michuk.

Subscribe to RSS feed for this article!

70 Comments

fold this thread Rambo Tribble  Friday, 23 May 2008 o godz. 4:57 pm #  Add karma Subtract karma  +0

As a control it might be worthwhile to run your fragmentation script, run the system normally for a period, say a week or so, then check the fragmentation level to see if the fragmentation has been decreased by the normal filesystem operation.

(Comments wont nest below this level)
 
fold this thread chlue  Friday, 23 May 2008 o godz. 10:52 pm #  Add karma Subtract karma  +0

I still wonder if it would not be easier to just copy a whole partition to another partition/harddrive, wipe the original data and then copy it back. (make two copies and checksums for added security)

Only real problem I see is, that you need way more diskspace to do that. Is there something more that I miss?

(Comments wont nest below this level)
fold this thread optimizationkit  Friday, 23 May 2008 o godz. 11:01 pm #  Add karma Subtract karma  --1

“Is there something more that I miss?”

No, this is the main problem.

 
fold this thread Chris Lees  Saturday, 24 May 2008 o godz. 6:17 am #  Add karma Subtract karma  +0

That brings back old memories of my Macintosh and the very roomy (for the time) 1 gigabyte external hard disk I owned. I couldn’t be bothered buying Norton Utilities, so I just defragmented by copying everything over to the external drive, formatting the internal, and copying back.

 
fold this thread AzP  Monday, 26 May 2008 o godz. 3:25 pm #  Add karma Subtract karma  +0

This is also the only way of defragging a ReiserFS drive. And they fragment like a mutha******. At first they are great, but after about a year the’re so sluggish it’s very very very painful.

 
 
fold this thread EdwardOCallaghan  Saturday, 24 May 2008 o godz. 4:56 am #  Add karma Subtract karma  +1

How about ZFS in opensolaris ?

(Comments wont nest below this level)
 
fold this thread roger64  Saturday, 24 May 2008 o godz. 10:01 am #  Add karma Subtract karma  +0

I tried to launch ok_defrag but failed -no such command).

Previously, I downloaded the python-dialog packet and I did an image disk of my partition which needs defrag after four successive upgrades.

Could there be some more explicit information about it ?

Thanks for help.

(Comments wont nest below this level)
fold this thread michuk  Saturday, 24 May 2008 o godz. 10:29 am #  Add karma Subtract karma  +1

Have you downloded ok_defrag from here http://download.tuxfamily.org/optimization/OptimizationKit/v0.1/tmp/ and installed it and put on your $PATH?

 
fold this thread optimizationkit  Saturday, 24 May 2008 o godz. 10:37 am #  Add karma Subtract karma  +1

“I tried to launch ok_defrag but failed -no such command). ”

ok_defrag uses the following commands: find, md5sum, mv and cp.

fold this thread B. Katz  Tuesday, 27 May 2008 o godz. 2:11 am #  Add karma Subtract karma  +0

swbobcat@ What I’d like to know is *WHERE* I can download ok_defrag. When I tried to download the utility I got 404 Error Message.

 
 
 
fold this thread Jon  Sunday, 25 May 2008 o godz. 11:15 am #  Add karma Subtract karma  +2

“When is there no need to perform a defragmentation? Generally speaking, if fsck returns non-contiguous inode below 10%.”

Why? Not “Why is higher fragmentation *possibly* worse than lower fragmentation” – I already know this.

I’m asking: why did you pick the nice, round, base-10 friendly number “10%” as your recommended threshold?

Could it be because you’re picking numbers out of thin air? If not – if you’ve actually performed some real-world, performance-oriented benchmarks on filesystems with different fragmentation levels – then post these results. They’d be useful.

Having read your article, I’m quite confident you haven’t. It lacks any kind of rigour. For instance, why say that “35.2% of the inodes are not placed continuously — the fragmentation degree of files in our image is very high”? Surely “99.9%” is “very high”, and “35.2%” is rather lower?

It’s *relative* and, depending on the access patterns of the files that are fragmented, may be utterly unimportant in determining the performance of the containing system.

TL;DR Don’t give up the day job.

(Comments wont nest below this level)
fold this thread optimizationkit  Sunday, 25 May 2008 o godz. 12:16 pm #  Add karma Subtract karma  +0

“Could it be because you’re picking numbers out of thin air?”

No – because it is difficult to go down below 10% with current ok_defrag file sorting method.

Also, many users will never exceed 10%.

 
fold this thread Name (required)  Sunday, 25 May 2008 o godz. 3:01 pm #  Add karma Subtract karma  +2

I’m going to agree with jon. This smacks of noobishness. Using shell scripts and python to defrag a filesystem? Copying files in and out of /tmp? This seems like a rediculous idea, and I would not let this near any system with data on it that I care about.

Additionally, I really don’t care about manufactured cases of fragmentation … I have never seen the levels of fragmentation mentioned in this article in real world experience using ext2/3.

So, yes, if you force bad things to happen to your filesystem … bad things will happen to your filesystem. Shocking, I know.

This is completely unneccesary.

fold this thread A Real Professional  Sunday, 17 January 2010 o godz. 3:22 am #  Add karma Subtract karma  +1

To Mr “Name Required” and Jon, clearly you 2 are noobs and don’t have a real clue. The author of the article makes excellent points and spends his time sharing his expertise and detailed, while you useless individuals sneer and produce nothing.

You State:
“Additionally, I really don’t care about manufactured cases of fragmentation … I have never seen the levels of fragmentation mentioned in this article in real world experience using ext2/3.”

That’s probably a reflection of your lack of experience and competence in the field of computing.

Now WRT to defragmentation, REAL Engineers like myself have seen horrendous fragmentation problems with linux ext2/ext3 filesystems both in serious mission critical environments and in enduser environments. For example we sold professional carrier grade voice-mail systems to some of the largest mobile operators in Asia and the United States, these voice mail systems that serve millions of users each in large Asian cities, get heavily fragmented extremely fast (i.e. 1 month) and suffer crippling slow downs; we had to write our own proprietary (unfortunately) defragger for linux ext2 and ext3 in order to solve the problem. Note equivalent systems that used NTFS (YUCK!) didn’t suffer these problems, because of their online defragmentation capability. The only reason that our customer chose our open-systems linux based systems over proprietary systems was that for the same hardware we could deliver twice the capacity of a Windows system (windows is slow and bloated); imagine the embarrassment of the technical team when our linux based messaging systems started becoming slower than the Windows based systems that the mobile operator was trying to get rid of. Investigation revealed that the problem was file fragmentation; defragging the system ‘cured’ the problem. Fragmentation is real problem with ext2/ext3 and shoving you head up you ass isn’t going make the problem go away. Congratulations to the fine authors of this article for showing some solutions that any end user could benefit from.

 
 
 
fold this thread Jon  Sunday, 25 May 2008 o godz. 12:29 pm #  Add karma Subtract karma  +0

“Also, many users will never exceed 10%.”

Sounds like another fact, pulled from thin air!

If they won’t generally exceed 10%, and this method can’t take you much below 10%, and a newly created FS is at ~10% defragmentation, what does that tell you?

Perhaps that “10%” is *meaningless*! The only thing that matters is the performance of the system that contains the FS.

Professionally, as a *nix sysadmin who maintains many hundreds of Linux, *BSD and Solaris boxes for a tier-2 UK ISP, I’ve never defragmented any system. Ever. The time it’ll take to do, coupled with the risk inherent in moving *all* your data off one partition, onto another, and back again, coupled with the fact that this *only* helps with frequently read, infrequently written data, makes me doubt its overall usefulness in anything other than very very specific circumstances.

YMMV.

(Comments wont nest below this level)
fold this thread optimizationkit  Sunday, 25 May 2008 o godz. 1:12 pm #  Add karma Subtract karma  +0

“Perhaps that “10%” is *meaningless*!”

I don’t want to start another flame war (see the discussion under the original article). You have your arguments, I’ve got my arguments.

EOD from my side.

 
 
fold this thread Greg  Sunday, 25 May 2008 o godz. 12:31 pm #  Add karma Subtract karma  +1

In terms of native Linux defrag support:
Ext4 will have (online) defragmentation built in. Currently XFS has defrag support. I think you have to take the disk offline first.

(Comments wont nest below this level)
fold this thread optimizationkit  Sunday, 25 May 2008 o godz. 1:17 pm #  Add karma Subtract karma  +0

“Ext4 will have (online) defragmentation built in”

Defragmentation in Ext4 is not a built-in. It is a set of patches

ext4-online-defrag-exchange-blocks-between-two-inodes.patch
ext4-online-defrag-relocate-file-data.patch
ext4-online-defrag-alloc-contiguous-blks.patch
ext4-online-defrag-ext4-defrag.patch
ext4-online-defrag-for-relevant-files.patch
ext4-online-defrag-check-for-freespace-fragmentation.patch
ext4-online-defrag-move-victim-files.patch

www dot kernel dot org/pub/linux/kernel/people/tytso/ext4-patches/2.6.26-rc2-git5-ext4-1/broken-out.tar.bz2

and proffesional defragmentation program ext4-online-defrag-command.patch

 
 
fold this thread dexen deVries  Sunday, 25 May 2008 o godz. 12:58 pm #  Add karma Subtract karma  +1

Just as a sidenote, XFS comes with a tool for performing on-line defragmentation by means of copying files: xfs_fsr

(Comments wont nest below this level)
 
fold this thread Crismon  Sunday, 25 May 2008 o godz. 1:48 pm #  Add karma Subtract karma  +0

Hi guys I have some problem during the procedure. Actually when I try to launch the command /sbin/fsck.ext3 -nfv filesystem.img there are several errors. In particular: /sbin/fsck.ext2 -nfv filesystem.img
e2fsck 1.40.2 (12-Jul-2007)
Impossible to find the block ext2 trying backup blocks…
/sbin/fsck.ext2: Bad magic number in super-block during filesystem.img opening

Super block is not readable, or is not describing filesystem ext2
If the device is valid and contains a filesystem ext2
(e non swap, ufs o altro),the super block is corrupted, you could try to execute e2fsck with an alternativ super block:
e2fsck -b 8193

what am I supposed to do now? thanks for your help

(Comments wont nest below this level)
fold this thread optimizationkit  Sunday, 25 May 2008 o godz. 1:58 pm #  Add karma Subtract karma  +1

Did you created ext{2,3} fs on this image?

 
 
fold this thread Crismon  Sunday, 25 May 2008 o godz. 2:11 pm #  Add karma Subtract karma  +0

Yes with this command:

/sbin/mkfs.ext3 filesystem.img

(Comments wont nest below this level)
fold this thread optimizationkit  Sunday, 25 May 2008 o godz. 2:35 pm #  Add karma Subtract karma  +0

Strange, it might be the problem with your e2fstools.

 
 
fold this thread carolinason  Sunday, 25 May 2008 o godz. 2:13 pm #  Add karma Subtract karma  +0

I never defrag my linux boxes, but I think about ‘defraging’ sometimes, since I’m using windows alot :-(

(Comments wont nest below this level)
 
fold this thread jugular  Sunday, 25 May 2008 o godz. 2:16 pm #  Add karma Subtract karma  +0

It’s funny how so many people can take very simple comparisons and completely screw them up to make their case. Anyone looking at the graphs for comparisons will quickly see differences, bu if you look at the numbers, the scales are different, and maybe the numbers aren’t quite so different.

Can we see the graphs with identical scales for both situations?

Thanks

(Comments wont nest below this level)
fold this thread optimizationkit  Sunday, 25 May 2008 o godz. 2:33 pm #  Add karma Subtract karma  +0

“Can we see the graphs with identical scales for both situations?”

Ask Chris Mason for a better scales in seekwatcher.

fold this thread Rob Lazzurs  Thursday, 29 May 2008 o godz. 11:05 am #  Add karma Subtract karma  +0

Might it be worth posting a spreadsheet with the original data then so other people can see the results and judge them?

Nice article, thanks. I have to say that I am not sure how useful this would be in my environment but I will be checking.

I think a lot of people replying need to remember YMMV, just because this does not apply to your environment does not mean this is not a worthwhile article.

Take care all.

 
 
 
fold this thread Leon  Sunday, 25 May 2008 o godz. 3:17 pm #  Add karma Subtract karma  +0

IMO there isn’t much that this blog post can make conclusion of.

And that’s not only because I feel that the difference in performance is so unsubstantial.

The very fact that the test is conducted on a loop device makes all the numbers invalid. There is high likelyhood that the .img file itself is defragmented on the host fs. Before and after moving data, the performance of the host fs may in its way affect the data collected for this “test”.

For example, there’s chance that before defrag, the files in the loop device somehow were written across defragmentation points of the img file itself and luckily after defragmentation the files ended up on a continuous part of the img on the host device. This definitely affects throughput measured.

If you ask me, Jon had a point. The recommendations were given pretty baselessly, and needless to say, without any substantial evidence or reasoning to back up the claim that those were the optimal numbers. optimizationkit tried to defend by throwing more numbers around but again, they were to be marked “citation needed”.

Possibly if anyone has free time, do some tests on fs on real disks, with accountability for caches, begin/end of disk, max/min, types/sizes of files, relation to everyday usage of disk etc. Now that will more likely help us server administrators. =)

So far I have not noticed any difference between speeds of my own fragged/defragged every-day filesystems (I used the copy out all, verify, delete all, copy in method), but hey, that’s just me and only yours truely.

(Comments wont nest below this level)
 
fold this thread optimizationkit  Sunday, 25 May 2008 o godz. 4:13 pm #  Add karma Subtract karma  +1

“This seems like a rediculous idea”

“I have never seen the levels of fragmentation mentioned in this article in real world experience using ext2/3.”

et all

Publication of this article was a _bad_ idea, even worse idea was to translate it into English.

Why? Guess what – I _really_ _don’t_ _care_ about your opinion.

I wrote this tool for myself – like an other OK tools. I did not want to cause fscking holly wars. Since I do not want to cause further flame wars, I will not publish any future OK daemon/tools versions or articles about it.

(Comments wont nest below this level)
fold this thread scorp123  Sunday, 25 May 2008 o godz. 10:19 pm #  Add karma Subtract karma  --12

We’re very thankful for that and we don’t really need your crappy tools and your BS FUD articles. Seriously. WTF are you thinking??? Any newbie on UbuntuForums could have done a better job at researching the facts before posting this pile of nonsense that you dare to call “article”. Just to make sure you don’t make an idiot of yourself again, you might want to read on FHS (Wikipedia is your friend), you might want to research why “UNIX style” partitioning is a good idea and why one should keep partitions such as /var and /home that get plenty of write-access on separate places than partitions such as /boot, /usr and /opt that mostly get lots of read-access and why this helps a great deal to keep fragmentation down to a minimum. After you have read this stuff you might get an idea why your tools are so totally not needed and why your article is utter nonsense.

fold this thread damentz  Monday, 26 May 2008 o godz. 1:16 am #  Add karma Subtract karma  +8

We’re very thankful for you’re crappy reply and your BS and FUD opinions. Seriously, WTF are you accomplishing by bashing OK? Any noob reading your comment can see through your intentions like purified glass.

Just to make sure you don’t look like an idiot again, you might want to google some howto’s not -HOW NOT TO BE AN @SSHOLE- and why some random jack*ss cares that you’re bullying someone who published this information with good intentions.

And hey, dumbass, we know about distributing data storage across multiple partitions, but holy shit! this is actually a story on single partition (omg!) fragmentation as we’re isolating the fragmentation level, not reinventing the wheel.

After you have read this stuff, you might get an idea why you’re post was meaningless and only full of sound and fury.

fold this thread scorp123  Monday, 26 May 2008 o godz. 8:09 am #  Add karma Subtract karma  --12

Just like the author of this crap article: Get a life! And stay the hell out of issues you don’t understand, yes?

And besides: YES, I AM AN A$$HOLE. I am not even trying to hide it. LOL.

 
fold this thread A Real Professional  Sunday, 17 January 2010 o godz. 3:31 am #  Add karma Subtract karma  +0

You State:
And besides: YES, I AM AN A$$HOLE. I am not even trying to hide it. LOL.

You forgot mention that you’re also incompetent and probably an arteest; Since you are a self-admitted unproductive and incompetent asshole, do the world some good and go kill yourself (to further make up for you sorry shithole existence, please make to sign your organ donor card prior to offing yourself AND make sure to write a note to inform the medical team that no part of your diseased brain should be used as medical material).

 
 
 
 
fold this thread Blaque  Sunday, 25 May 2008 o godz. 4:18 pm #  Add karma Subtract karma  +2

Now I’m no Linux guru (I do know some however) but it does seem that this article is bent on fabricating a problem with Linux that does not cause problems on the level of a Windows system. The deciding factor should be performance not the fact that fragmentation exists.

(Comments wont nest below this level)
 
fold this thread Anon  Sunday, 25 May 2008 o godz. 4:28 pm #  Add karma Subtract karma  +0

If you are only interested in the fragmentation of a specifically small set of files on an ext3 system you can use the filefrag command as root to find out how many extents a file is taking up.

(Comments wont nest below this level)
 
fold this thread hirni  Sunday, 25 May 2008 o godz. 4:38 pm #  Add karma Subtract karma  +0

if you’re using XFS – you can do all that ONLINE:
-xfs_fsr -v /mountpoint
((defrags the filesystem while mounted))
-echo “frag\n” | xfs_db -r /dev/sda
((displays fragmentation ONLINE))

you can clearly see by that, that not every fragmentation is bad:
If a huge file has 2 extents instead of 1, no big deal.
If there are a total of 200.000 files in /, what’s the
problem – if 10000 of them are non-ideal ?
etc. etc.

I don’t think fragmentation plays any significant role nowadays, where you have megabytes of disk-cache and an efficient block-buffering in (gigabytes of) RAM.

(Comments wont nest below this level)
 
fold this thread blah  Sunday, 25 May 2008 o godz. 4:50 pm #  Add karma Subtract karma  --1

It’s only a myth if someone knows nothing about disk storage.
The real advantage Linux, or any U**X derivative, has is that the defrag process is inherentaly faster than Windows. While Windows attempts to move around a bunch of files and place the contents in the correct order, any U**X operating system needs only to move a set of the files to another drive, then move back. You don’t need to copy the entire drive at once, only large chunks…say the size of your biggest file or 10% of drive space.
Even with a 300GB drive, this can be accomplished in a matter of hours, while a Windows system would probably take a couple of days.

(Comments wont nest below this level)
 
fold this thread rjb  Sunday, 25 May 2008 o godz. 5:05 pm #  Add karma Subtract karma  +1

the / partition on my laptop is 21 G, contains 245089 files, is 89.61% used (in terms of blocks), and has been heavily used for many months (since last reformatted), with many files changed and thrown around (like by a dist-upgrade). The fragmentation in term of non-contiguous inodes is 4.4%.

I am not convinced that I need to worry about fragmentation on ext3.

(Comments wont nest below this level)
 
fold this thread bill  Sunday, 25 May 2008 o godz. 7:38 pm #  Add karma Subtract karma  +1

Very nice article.

I just did not understand what the contiguous inodes mean.

Keep up the good work.

Thank you very much for the information.

(Comments wont nest below this level)
 
fold this thread James Justin Harrell  Sunday, 25 May 2008 o godz. 7:53 pm #  Add karma Subtract karma  error

Refusal of the ext filesystem developers to accept that defragmentation is needed reminds me of the refusal of Apple to bundle in a mouse with more than one button.

(Comments wont nest below this level)
 
fold this thread ropers  Sunday, 25 May 2008 o godz. 8:59 pm #  Add karma Subtract karma  +0

It says above:

> Some systems have a functionality to remove files located in /tmp
> during bootup. The best thing would be to turn it off for the time of
> defragmentation.

Why not make the defrag script use /var/tmp instead?

/var/tmp, according to the Linux Filesystem Hierarchy Standard (FHS), is preserved between boots.

(Comments wont nest below this level)
 
fold this thread scorp123  Sunday, 25 May 2008 o godz. 10:10 pm #  Add karma Subtract karma  --6

This article is idiotic, sorry to say so. In 12 years of UNIX + Linux administration I have never ever seen a system suffering from fragmentation. This entire “Linux needs a defrag tool as well” nonsense is just that: nonsense.

(Comments wont nest below this level)
fold this thread AzP  Monday, 26 May 2008 o godz. 3:30 pm #  Add karma Subtract karma  +0

What about ReiserFS?

 
fold this thread B. Katz  Wednesday, 28 May 2008 o godz. 1:09 am #  Add karma Subtract karma  +0

While I have never seen any need to defrag my system, every once in a while BAD things happen: I get a power surge, power outage, etc., (yes I do have both a surge protector and a UPS, its interesting see what is left of a surge protector after a MAJOR voltage spike… And when the power goes down in the middle of the night… well you get the picture). and some files end up corrupted for no apparent reason, at while point something happens behind the seems and I sit for a minute or two twiddling my thumbs at which point it says that that whatever was wrong is now “PASSED”, “FIXED” or “OK” (I suspect this has something to do with the inodes). That was my major interest in this. I have NEVER had a problem per se UNLESS something like a power surge, outage etc., and then it is not one computer, its my entire network (of three computers) that seem to be whacked. Thank God I make periodic BACKUPS

fold this thread Yaro  Sunday, 1 March 2009 o godz. 9:11 pm #  Add karma Subtract karma  +0

Yes, but that’s file system errors, not fragmentation. Two completely different things.

No, Fragmentation is not a problem on Linux. I’ve never once cleared 5% non-contiguous space in all my years of using Linux. I’ve known people who’ve done decades of work on UNIX who have said they’ve never had fragmentation problems.

Existence of fragmentation is not even a problem. Simply becaus Linux, unlike Windows, actually runs like an operating system should run: RAM is used efficiently, resulting in swap being used rarely in large-RAM systems, only maybe a few megabytes for binary image purposes for the most part. And of course, hard disks are managed in a good way.

Compare it to Windows: The memory model sucks because Windows likes o use way too much RAM for itself. It uses its swap file heavily even when given lots of RAM. And don’t get me started on the delicate and horrible way FAT and NTFS fragment.

This article seems poorly written, almost as of the author is either FUDing, pulling “facts” out of thin air, or else, he’s a n00b coming from Windows trying to attribute one of Windows’ greatest failures to Linux as well.

Here’s a few facts:

1. Yes, there is fragmentation in Linux.

2. No, said fragmentation is not a problem.

3. Said fragmentation rarely, if ever, seems to reach over 10% of a decently-sized partition.

4. Defragging tools do exist for every Linux file system, but almost all of them are off-line. These defraggers are part of the *fstool packages, usually as part of their respective fscks.

5. These defragging features and tools are used for partitioning work. For example: You want to shrink a 300 GiB partition to a 150 GiB, GPartEd will run fsck before and after maniulating the partition to manage both errors and any non-contiguous space it can.

6. There is no real performance hit from fragmentation in Linux, as the physical arrangement of files, coupled with excellend buffering, caching, and othe rbuzzworded techniques provided by the Linux kernel basically negate it.

7. Serious fragmentation only happens in Linux when a file system reaches over 90% capacity. YES I INCLUDE REISERFS IN THAT, TOO!

8. If your Linux FSes are over 90% full, you’ve got bigger problems to worry about then fragmentation. Thank the Programmer Linux uses a separate partition for swap.

 
 
fold this thread A Real Professional  Sunday, 17 January 2010 o godz. 3:38 am #  Add karma Subtract karma  +0

DEAR SCORP123 or is it CRAP123; Your idiot comment goes as follows:

“This article is idiotic, sorry to say so. In 12 years of UNIX + Linux administration I have never ever seen a system suffering from fragmentation. This entire “Linux needs a defrag tool as well” nonsense is just that: nonsense.”

Right…. and I guess YOU know more than Theodore Ts’o, the primary developer of ext3 and ext4 WHO HAS BUILT A DEFRAGMENTATION UTILITY FOR EXT4. I wonder why he did that? Did he do it for fun? No… Did he do it because he was high? … no… HE DID IT BECAUSE IT’S NEEDED! …… YOU MORON!

 
 
fold this thread Henaway  Monday, 26 May 2008 o godz. 1:27 am #  Add karma Subtract karma  +0

I could be COMPLETELY wrong here, but my understanding of the difference between linux and Windows filesystems here (ext2/3 vs FAT/NTFS) is that linux writes an entire file in a contiguous block whenever possible whereas Windows will write a little piece of a file in scattered blocks all over the hard drive until it’s finished writing the file. This means that, by default, linux is FAR less prone to file fragmentation than Windows. Windows fragments by default. Linux does not.

So fragmentation on linux would only become a problem if an insanely huge number of tiny files were written and deleted on a regular basis over a very long period of time. (To the point where all the contiguous blocks have been used and now the filesystem has to write to blocks freed up by deleted files.) At THAT point a defrag would be beneficial to the system … but not until the entire disk had essentially been filled.

With terabyte drives now on the market for a couple hundred bucks … how often is that going to happen?

That is MY understanding and I could be wrong. So take it with a grain of salt.

(Comments wont nest below this level)
 
fold this thread JT  Monday, 26 May 2008 o godz. 5:05 am #  Add karma Subtract karma  --2

Why does this issue keep coming up despite the fact that numerous sources and numerous debated going back more than a decade show that degramentation is not a problem? First, start with understanding what “non-contiguous inode” means in ext3. Then understand how that number will rarely, if ever exceed 10%. Then look back at the hundreds of debates on this issue and see that each time it has been proven defragmentation in Linux is not a problem. This article is irresponsible and ignortant, you should remove it and offer a retraction.

(Comments wont nest below this level)
 
fold this thread Bbob  Monday, 26 May 2008 o godz. 5:17 am #  Add karma Subtract karma  --1

B.S.

(Comments wont nest below this level)
 
fold this thread solca  Monday, 26 May 2008 o godz. 5:56 am #  Add karma Subtract karma  +1

(sorry my bad english)

optimizationkit: I have read your nice tools and this article and decided to check my partitions, my laptop barely 4%, desktop 3%, I’m sysadmin in my University for ~ 20 servers and there is statistical facts in my case that can give you more info in your investigations:

* in our servers filesystems with lots of free space it barely fragment in ext{2,3} or xfs.
* inversely, saturated filesystems fragment a log (73% in our homes server).

Our homes server have 1.5TB SAS storage for all our students and is (ab)used a lot, several times have gone to 0% free space, in order to maximize free space we chose to reserve 0% to root when making the fs which (IMO) can affect greatly the fragmentation when full.

I’m not a fs expert by any means but I think your defrag tool could be improved by:

* stat and checksum old file and know it’s size.
* create a new empty file with exact size so it’s blocks are contiguos.
* overwrite new file from old file.
* checksum file.
* if ok new file
delete old file
else
delete new file
find out why fail and report back.
* check all error paths and fail softly.

You should ask kernel gurus about how to create a contiguos file, that’s the main problem, maybe you can check the rtorrent tool, it does some magic when it creates the file the whole size and then begins the download.

Anyway I (and others) like your article, as sysadmin I can tell you that it would be very interesting for us if you repeat it on real hardware and with different free spaces checking in what point of % free space fragmentation became a problem. Normally is not a problem but it can be as I know now thanks to your article.

(Comments wont nest below this level)
fold this thread optimizationkit  Monday, 26 May 2008 o godz. 11:03 am #  Add karma Subtract karma  +1

“create a contiguos file”

There are two C functions for this posix_fallocate and fallocate – I did not know how to use it from Python.

“defrag tool could be improved”

I’m not interested in improving ok_defrag.

The first reason is that I start to use Ext4 which has its own defragmentation program – e4defrag.

The second reason is that I don’t want to be flamed again by band of “experts”/trolls/etc. I wonder why trolls don’t attack XFS, Ext4 developers with equal wrath – they created defragmentation programs for their file systems.

 
 
fold this thread etienne  Tuesday, 27 May 2008 o godz. 11:35 am #  Add karma Subtract karma  +1

Unlike most people here, I still think Linux has a problem
with fragmentation. My experience is that when I absolutely need a contigous file (because this file will be accessed at bootloader time), file size of just few hundred Kbytes to few Mbytes, it can take 10-16 tries (write all file at once) to get it on ext2/ext3. Software available on project Gujin at sourceforge, obvious objections already been answered.
When blocks have been allocated for a file, it is “for life” i.e. a fragmented file will stay fragmented until re-written.
I still wonder if that is due to the extended attributes of files created by xattr or SELinux, so distribution specific…

(Comments wont nest below this level)
 
fold this thread Peter  Friday, 30 May 2008 o godz. 11:08 pm #  Add karma Subtract karma  +0

note that there are multiple types of fragmentation:

1. files in a directory [seek time for directory listings,
nowadays _mostly_ fixed in the various filesystems, e.g.
by trees: see e.g. mkfs.ext3: dir_index]

2. blocks of a file: continuous or scattered:
this concerns both the placement of inodes AND the data blocks.
And fsck does only tell part of the story: “non-contiguous inode”
… . (still too simple: block groups, etc “should”(?) also be
considered; as should be striping, physical disk layout,
raid issues, bad block remapping by the drive, etc, pp)

3. tail packing or fragments: placing multiple partial blocks into a
single block. Ext3 doesn’t fully support this, so its
a non-issue for ext3. Aix4′s JFS is an example of such
a FS. [basically this always violates #2 by trading
a tiny bit of disk space for an extra read and a
possible extra head movement. Which should make this
another obselete issue, I'd assume]

Basically the article only is only a very simple (and IMHO valid)
experiment concerning type 2 fragmentation.

Ignoring the loop issue, the improvements shown aren’t really worthwhile with *server-style* filesystems, though one rule of thumb might be worthwile to remember:

Do not allow a filesystem with _*changing*_ data to fill up beyond say 90% (or delete some huge files if you did). That’s basically the only way to get hit with REALLY significant degradation.

Another note: consider traditional usage cases and development constraints when comparing fragmentation issues for different filesystems on different operating systems:

Windows filesystems and their fragmentation issues are from a desktop heritage (i.e. outage is cheap, powerdown may happen at any moment), while Unix filesystems always were server filesystems, so reducing severe degradation and allowing long uptimes was always a far more important influence for unix filesystem design.

(Comments wont nest below this level)
 
fold this thread Aaron  Sunday, 1 June 2008 o godz. 8:43 am #  Add karma Subtract karma  +2

What we’re failing to understand here is how ext3 writes data to disk. I’ll draw up an analogy for simplification.

Consider a number of file cabinets on your office. You have a file that you need to store in your cabinets. Unfortunately, the file is far to large to fit in the first cabinet. So, at this point, you have two options:

If you’re going to take the road of the NTFS/FAT family of filesystems, then you immediately begin pulling papers from your file, and placing them in the first available spot in the cabinets, until your file is exhausted, meaning your file will probably exist in several cabinets, if they were already filled.

If you’re going to take the road of the ext2, ReiserFS, XFS, JFS, etc. family of filesystems, then you scan the cabinets looking for the largest chunk of contiguous cabinet space to fit the file. Chances are high, that you’ll find space to fit the entire file together, rather than splitting it out across separate cabinets.

Now, let’s look at the two algorithms’ results. In the first (NTFS/FAT), a single file is fragmented across the disk. As you continue to write and delete data, the filesystem becomes more and more fragmented until I/O performance becomes a serious concern. In the second algorithm, the inodes are fragmented, but individual files are not. This becomes less likely of a concern, as inodes have a lookup table in the filesystem superblocks, making it easy to locate the data blocks on disk. Although the inodes may not be contiguous, the file data blocks are. It is substantially more difficult to achieve poor I/O performance in this manner.

Lastly, take a look at the role of the e4defrag command. Not only does it defrag data blocks (fragmented files), but it also defrags inodes which is the more common scenario (this is what fsck is reporting, not fragmented files).

While the intention of your article is good, the understanding of lower level operations needs to be studied. The ext2 family of filesystems, and other popular OSS filesystems, do not generally suffer from block fragmentation, just inode fragmentation. This is generally fine, as inode fragmentation just won’t lead to poor I/O performance, keeping the user and system admin happy. However, with that said, a full disk will ultimately lead to block fragmentation in the end, even on ext2 based filesystems.

(Comments wont nest below this level)
 
fold this thread E@zyVG  Monday, 2 June 2008 o godz. 10:28 pm #  Add karma Subtract karma  +0

The extfs (extension 2 file system), as an example, that Linux OS operates under is resistant to fragmented files. Fragmenting happens when a file (of any size) will not ‘fit’ in a single space on the hard drive.

To put it very simply, while not entirely accurate, Linux will NOT split a file over a disk surface, instead, it finds the most appropriate space.

Check the following post:
http://blog.linuxoss.com/2007/04/15/why-doesnt-linux-need-defragmenting/

(Comments wont nest below this level)
 
fold this thread Morgauxo  Tuesday, 1 July 2008 o godz. 4:25 pm #  Add karma Subtract karma  +0

Optimization Kit is your friend

Significant fragmentation resulting in a noticable performance hit is unlikely in Linux.

Unlikely is not the same as impossible.

With all the new and varied ways Linux is being used, from super-computers to Tivos and even wristwatches it’s a safe bet that someone somewhere is running something that is constantly saving and deleting files of odd sizes on a mostly full hard drive.

Bringing a defrag script into the Linux world only makes Linux a little better, even if it doesn’t do a huge amount of good for the average user it certainly hurts no-one.

So why flame someone whom took the time to write a script and share it for the benefit of others?

Trolls only discourage the writing and sharing of new open source software. Even if the author being flamed doesn’t take it to heart, any potential new OSS programmer could read posts like that and decide it’s better not to participate.

(Comments wont nest below this level)
 
fold this thread al-Quaknaa  Sunday, 10 August 2008 o godz. 8:01 pm #  Add karma Subtract karma  +0

Hi,
a little late, I’d like to ask for the frag.sh script? Where can I get it, please? if there is the author (or anyone who knows where it is) and doesn’t want to publish the link publicly, Just say something and I will pu my email here, I just don’t want to do that if there’s no chance of getting the script :) Thanks in advance,
al-Quaknaa

(Comments wont nest below this level)
 
fold this thread heydar  Thursday, 14 August 2008 o godz. 6:25 am #  Add karma Subtract karma  +0

i need source of linux

(Comments wont nest below this level)
 
fold this thread Luc  Thursday, 29 October 2009 o godz. 1:42 pm #  Add karma Subtract karma  +0

@optimizationkit

Hi, I hope you somehow still get this, although I would not be surprised if you just “forgot” all about this thread… and I dont blame you if you did…

What some people seem to forget is that you try to improve our world. And some of the idots here should first learn some manners in how to deal with others… I guess it says more about them than about you..

But on the main subject of this thread and in support of you : forget the idots that say that fragmentation is “not an issue”. I know they try to intimidate you to shut up but ignore them, for they don’t know what they are talking about.

I am working 30 years in computers and electronics and software so I am going to say to all imbecieles that argue the fragmentation issue:

” YOU DO NOT EVEN HAVE A BASIC UNDERSTANDING OF HOW A HARD DISK WORKS! ”

Otherwise you would understand why fragmentation occurs in ANY operating system! The fact that you let a system run without defrag-ing does not mean it isn’t there and if you dont notice it, then you simply have a fast enough system. Still does not mean that it is not occuring!

IGNORING defrag does NOT make it go away!

SO BEFORE ANY OF YOU TELL OPTIMIZINGKIT TO LEARN SOFTWARE, YOU SHOULD FIRST LEARN ABOUT SOME BASIC HARDWARE (hint: start with how a harddisk works!)..

Optimizingkit: good luck and keep working at a better world..

(Comments wont nest below this level)
 
fold this thread Lan Tester   Wednesday, 20 October 2010 o godz. 9:52 pm #  Add karma Subtract karma  +0

we always use surge protectors at home because the electricity is very ustable`:’

(Comments wont nest below this level)
 
fold this thread Hair Loss Treatment %0B  Tuesday, 25 January 2011 o godz. 5:09 am #  Add karma Subtract karma  +0

;”~ I am very thankful to this topic because it really gives great information ‘,’

(Comments wont nest below this level)
 
fold this thread abbas_qamar  Thursday, 3 February 2011 o godz. 6:32 pm #  Add karma Subtract karma  +0

optimizationkit,
This is a great article you have written. I appreciate all the hard work you’ve put into this article.

(Comments wont nest below this level)
 
fold this thread vip loan shop  Monday, 1 April 2013 o godz. 10:01 am #  Add karma Subtract karma  +0

This web site is known as a walk-through for all of the data you wished about this and didn’t know who to ask. Glimpse here, and you’ll positively discover it.

(Comments wont nest below this level)
 
fold this thread Lorinda Gent  Sunday, 14 April 2013 o godz. 7:14 am #  Add karma Subtract karma  +0

When you put together a computer system, one piece of standard equipment you’ll probably buy is a surge protector. Most designs serve one immediately obvious function — they let you plug multiple components into one power outlet. With all of the different components that make up a computer system, this is definitely a useful device.”

Our personal blog site
<="http://www.caramoan.ph/caramoan-package/

(Comments wont nest below this level)
 
fold this thread www.promosoundcloud.com  Saturday, 21 December 2013 o godz. 1:36 am #  Add karma Subtract karma  +0

Howdy! I know this is kinda off topic but I’d figured I’d ask.

Would you be interested in trading links or maybe guest authoring a
blog article or vice-versa? My blog discusses a
lot of the same subjects as yours and I feel we could greatly benefit from
each other. If you happen to be interested feel free
to shoot me an email. I look forward to hearing from
you! Superb blog by the way!

(Comments wont nest below this level)
 
fold this thread WAN  Sunday, 23 February 2014 o godz. 3:14 pm #  Add karma Subtract karma  +0

I got this website from my friend who informed me concerning this web site and now this time I am browsing this web page and reading
very informative articles or reviews at this place.

(Comments wont nest below this level)
 
fold this thread achat PhéRomones  Friday, 27 June 2014 o godz. 4:29 pm #  Add karma Subtract karma  +0

The idea of women using pheromones to attract their attention seems a little hard to come to terms with for
most men. The law of attraction, revealed by Abraham-Hicks is the most powerful force
in the universe. Considering the variety of products which might be presented,
needless to say, you must be watchful to choose which of the best products
to utilize.

(Comments wont nest below this level)
 
fold this thread Albert à l’ouest Télécharger  Thursday, 17 July 2014 o godz. 7:18 pm #  Add karma Subtract karma  +0

Have you ever thought about adding a little
bit more than just your articles? I mean, what you say
is fundamental and everything. But think of if you added some great pictures or video clips to give your posts more, “pop”!
Your content is excellent but with pics and video
clips, this site could definitely be one of the greatest
in its niche. Very good blog!

(Comments wont nest below this level)
 
fold this thread hack facebook address  Saturday, 13 September 2014 o godz. 7:20 am #  Add karma Subtract karma  +0

I’ve similar experiences with purchasers whose experienced
DETRIMENTAL ROI on FB ad campaigns, whereas Google adwords resulted in a 4000%
ROI.

(Comments wont nest below this level)
 
fold this thread facebook security check required to proceed  Saturday, 13 September 2014 o godz. 7:55 pm #  Add karma Subtract karma  +0

If you use internet affiliate marketing you can get paid generally to enjoy-you
receives a commission for doing stuff you love doing and
you also were going to do in any case.

(Comments wont nest below this level)
 
Name (required)
E-mail (required - never shown publicly)
URI

Adjust field size: shrink | enlarge)


You can use simple HTML in your comments. Some examples are as follows:
  • A hyperlink: <a href="polishlinux.org">GNU/Linux for everyone!</a>,
  • Strong text: <strong>Strong text</strong>,
  • Italic text: <em>italic text</em>,
  • Strike: <strike>strike</strike>,
  • Code: <code>printf("hello world");</code>,
  • Block quote: <blockquote>Block quote</blockquote>

New AdTaily ads!

Are you a film buff?

film buffs community, movie recommendations and reviews

RSS: Comments

You can follow the comments to this article through a special channel RSS 2.0 .

Related articles: CLI

 more »

PolishLinux Top Content


Become our fan on Facebook!

PolishLinux.org on Facebook

Follow PolishLinux on Twitter!

Follow polishlinux on Twitter

Google Ads