Investigate disk usage | Just my blog

/dev/random

Investigate disk usage

Recently I had to investigate the usage of a very big volume... with a lot of data-files, owned by several users.


I started with "agedu", but somehow I was not able to get the information I needed.. so I started using find with stat and put everything into MySQL.


So the first step was to do the following find command:


# find /nfs/bigfiler -exec stat --format="%F:%n:%s:%U:%u:%G:%g:%X:%Y:%Z" {} ; > /scratch/big-filer.info


Create a table in MySQL:



CREATE TABLE `pieter.bigfiler_content` (
`fileid` int(10) unsigned NOT NULL AUTO_INCREMENT,
`file_type` char(32) NOT NULL,
`filename` varchar(512) NOT NULL,
`size` int(11) NOT NULL,
`user` char(16) DEFAULT NULL,
`uid` char(16) DEFAULT NULL,
`groupname` char(16) DEFAULT NULL,
`gid` char(16) DEFAULT NULL,
`time_access` datetime DEFAULT NULL,
`time_mod` datetime DEFAULT NULL,
`time_change` datetime DEFAULT NULL,
PRIMARY KEY (`fileid`),
KEY `idx_file_type` (`file_type`) USING BTREE,
KEY `idx_user` (`user`) USING BTREE
)


Load the data into MySQL:



LOAD DATA INFILE '/scratch/big-filer.info'
INTO TABLE pieter.bigfiler_content
FIELDS TERMINATED BY ':'
LINES TERMINATED BY 'n'
(file_type, filename, size, user, uid, groupname, gid, @time_access, @time_mod, @time_changed)
SET time_access = FROM_UNIXTIME(@time_access),
time_mod = FROM_UNIXTIME(@time_mod),
time_change = FROM_UNIXTIME(@time_change);



And now you can run nice queries to analyse the data :D 


  • Social

  • By continuing to use the site, you agree to the use of cookies. more information

    The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible and enable advertising to provide you free content. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

    Close