Investigate disk usage

Recently I had to investigate the usage of a very big volume… with a lot of data-files, owned by several users.

I started with “agedu”, but somehow I was not able to get the information I needed.. so I started using find with stat and put everything into MySQL.

So the first step was to do the following find command:

# find /nfs/bigfiler -exec stat –format=”%F:%n:%s:%U:%u:%G:%g:%X:%Y:%Z” {} ; > /scratch/big-filer.info

Create a table in MySQL:

CREATE TABLE `pieter.bigfiler_content` (
`fileid` int(10) unsigned NOT NULL AUTO_INCREMENT,
`file_type` char(32) NOT NULL,
`filename` varchar(512) NOT NULL,
`size` int(11) NOT NULL,
`user` char(16) DEFAULT NULL,
`uid` char(16) DEFAULT NULL,
`groupname` char(16) DEFAULT NULL,
`gid` char(16) DEFAULT NULL,
`time_access` datetime DEFAULT NULL,
`time_mod` datetime DEFAULT NULL,
`time_change` datetime DEFAULT NULL,
PRIMARY KEY (`fileid`),
KEY `idx_file_type` (`file_type`) USING BTREE,
KEY `idx_user` (`user`) USING BTREE
)

Load the data into MySQL:

LOAD DATA INFILE '/scratch/big-filer.info'
INTO TABLE pieter.bigfiler_content
FIELDS TERMINATED BY ':'
LINES TERMINATED BY 'n'
(file_type, filename, size, user, uid, groupname, gid, @time_access, @time_mod, @time_changed)
SET time_access = FROM_UNIXTIME(@time_access),
time_mod = FROM_UNIXTIME(@time_mod),
time_change = FROM_UNIXTIME(@time_change);

And now you can run nice queries to analyse the data 😀 

Fedora directory server

Yesterday evening I start playing with Fedora Directory Server

So first I setup Fedora Core 8 as a VMWare-instance… But after some playing around, I had the next message:

“Server failed to start !!! Please check errors log for problems”

And guess what… no information at all in the logs :-( So removed the packages and the next directories:

/etc/dirsrv
/etc/sysconfig/dirsrv
/var/lock/dirsrv
/var/lib/dirsrv

So no information… then strace will be your best friend :-D

So I started:

[root@fedora-ds debug]# strace -o ~/debug/setup -ff /usr/sbin/setup-ds.pl

And guess what… I had the error again… So I went to the ~/debug folder on another terminal and did:

[root@fedora-ds debug]# grep “failed” *
setup.31676:read(4, “Server failed to start !!! Pleas”…, 4096) = 64
setup.31676:write(2, “Server failed to start !!! Pleas”…, 64) = 64
setup.31711:write(1, “Server failed to start !!! Pleas”…, 64) = 64
[root@fedora-ds debug]#

When I digged into setup.31711 I found:
read(255, “if test ! -f $STARTPIDFILE ; the”…, 2220) = 663
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
stat64(“/var/run/dirsrv/slapd-fedora-ds.startpid”, 0xbfcd8eb8) = -1 ENOENT (No such file or directory)
rt_sigprocmask(SIG_SETMASK, [], NULL, 8-) = 0
fstat64(1, {st_mode=S_IFIFO|0600, st_size=0, …}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f7c000
write(1, “Server failed to start !!! Pleas”…, 64) = 64

So this is a nice clue… /var/run/dirsrv… and guess what… the owner of this directory was fedora-ds (a user I set up initially for testing purposes for the Directory Server :-( ) With the comment chown I corrected the owner of this folder… and now service dirsrv start works :-P

Conclusion… strace is your best friend :-D