Environment
SUSE Linux Enterprise Server 10
SUSE Linux Enterprise Server 11
Novell Open Enterprise Server 2 (OES 2) Linux
Novell Open Enterprise Server 11 (OES 11) Linux
SUSE Linux Enterprise Server 11
Novell Open Enterprise Server 2 (OES 2) Linux
Novell Open Enterprise Server 11 (OES 11) Linux
Situation
On an XFS filesystem, containing a large number of small files (i.e. <= 4KiB in size), the adding and deleting of files can cause the free space to get fragmented. This fragmentation can cause a false "out of space" message despite there being a significant amount of free space available.
Further, a severely fragmented filesystem can cause the inability to create new files as outlined in 7014318 - Error: "no space available" on XFS filesystem
Further, a severely fragmented filesystem can cause the inability to create new files as outlined in 7014318 - Error: "no space available" on XFS filesystem
Resolution
Monitor your Allocation Groups (AGs) periodically -- starting with a baseline measurement and comparing periodically.
Additional Information
If you're not already familiar with xfs_info, you will want to run this first against a mounted XFS filesystem. It will return data as follows:
meta-data=/dev/evms/csm/fence4 isize=256 agcount=288, agsize=8388607 blks
= sectsz=512 attr=0
data = bsize=4096 blocks=2409365488, imaxpct=25
= sunit=0 swidth=0 blks, unwritten=1
naming =version 2 bsize=4096
log =internal bsize=4096 blocks=32512, version=1
= sectsz=512 sunit=0 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0
Of interest for our purposes is:
Knowing how many AGs we have, one can manually run a simple command line to generate a report on each AG and list it out to a file. The following is such a command line that will report on each AG to a separate file:
for AGNO in `seq 0 287`;do /usr/sbin/xfs_db -r -c "freesp -s -a $AGNO" /dev/evms/csm/fence4 1>ag${AGNO}.txt
where:
from to extents blocks pct
1 1 6 6 0.00
8 15 31220 467872 26.33
1048576 2097151 1 1308895 73.67
total free extents 31227
total free blocks 1776773
average free extent size 56.8986
It shows there are:
- only 6, 1 block "orphans" -- meaning there are no other free blocks around each block.
- there are 31220 clusters of 8-15 contiguous blocks
- there is 1 cluster of 1048576-2097151 contiguous blocks
Conversely, here is an AG that is greatly fragmented:
from to extents blocks pct
1 1 979870 979870 68.79
2 3 205140 444668 31.21
total free extents 1185010
total free blocks 1424538
average free extent size 1.20213
This shows that 68.79% of the free space on the filesystem is made up of only single blocks; and 31.21% is made up of 2-3 block clusters.
meta-data=/dev/evms/csm/fence4 isize=256 agcount=288, agsize=8388607 blks
= sectsz=512 attr=0
data = bsize=4096 blocks=2409365488, imaxpct=25
= sunit=0 swidth=0 blks, unwritten=1
naming =version 2 bsize=4096
log =internal bsize=4096 blocks=32512, version=1
= sectsz=512 sunit=0 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0
Of interest for our purposes is:
- the "agcount" which is 288 AGs -- numbered 0 - 287
- the volume in question is right after "metadata=", or /dev/evms/csm/fence4
Knowing how many AGs we have, one can manually run a simple command line to generate a report on each AG and list it out to a file. The following is such a command line that will report on each AG to a separate file:
for AGNO in `seq 0 287`;do /usr/sbin/xfs_db -r -c "freesp -s -a $AGNO" /dev/evms/csm/fence4 1>ag${AGNO}.txt
where:
- the "287" is the agcount value
- the "/dev/evms/csm/fence4" is the XFS filesystem
- ag${AGNO}.txt is the file capturing the output.
from to extents blocks pct
1 1 6 6 0.00
8 15 31220 467872 26.33
1048576 2097151 1 1308895 73.67
total free extents 31227
total free blocks 1776773
average free extent size 56.8986
It shows there are:
- only 6, 1 block "orphans" -- meaning there are no other free blocks around each block.
- there are 31220 clusters of 8-15 contiguous blocks
- there is 1 cluster of 1048576-2097151 contiguous blocks
Conversely, here is an AG that is greatly fragmented:
from to extents blocks pct
1 1 979870 979870 68.79
2 3 205140 444668 31.21
total free extents 1185010
total free blocks 1424538
average free extent size 1.20213
This shows that 68.79% of the free space on the filesystem is made up of only single blocks; and 31.21% is made up of 2-3 block clusters.