Koozali.org: home of the SME Server

diference between quota reported and space used on disk

Offline larieu

  • ****
  • 214
  • +0/-0
diference between quota reported and space used on disk
« on: April 17, 2008, 06:55:34 AM »
It happend second time

first time on old machine
the diference between quota reported and real amount of data in user folder is notabel (Up to 7-800M) for an quota of 1G with maximum 2G

on the old server I used quotacheck with several parameters after I stopped some services which could change files on user folders
that workaround had good results on that time

on new machine the problem start to be annoing also
I have at least 5 user from 100 with notable diferences
for examle one new user acccount has from start around 750M used reported by quota menu in sme admin interface
but if I look with
du -h /home/e-smith/files/users/username
I receive almost 0 (what is expected for new user)
any work around it will be helpfull

Thanks
if everybody's life around you is better, probably yours will be better
just try to improve their life

Offline jumba

  • ****
  • 291
  • +0/-0
  • Donations: July 2007 - $ 20.00
    • Smeserver på svenska!
Re: diference between quota reported and space used on disk
« Reply #1 on: April 18, 2008, 09:46:27 PM »
Are you sure that user didn't save any files in other locations like ibays???

Quota should show all space occupied by the user on the disk, not only in the user home directory...
« Last Edit: April 18, 2008, 10:20:23 PM by jumba »

Offline larieu

  • ****
  • 214
  • +0/-0
Re: diference between quota reported and space used on disk
« Reply #2 on: April 22, 2008, 10:25:33 PM »
I am positive
Initially I suspected something on the group of that user (from other user) and I calculate manually for several users the amount of files but no apparent mistake
Also if that happend the difference must be counted for all users in that group (at least this is expected)

I used the contrib diskusage and the reported amount of data was pretty same as the  du command for that user (1-5 k not even 1M)
of course very different from the quota

I tried in many  ways with quotacheck but no apparent result

in this moment all amount of data reported by du command
du -h /home/e-smith/files/users/username/
is 360M
the quota reported is 819M and the user is set to 800M with maximum 1G
that user don't have any access to any iBays or samba folders, ONLY EMAIL

if somebody know a method to find all files "owned" by that user and counted by quota it will help for debug

Many Thanks
if everybody's life around you is better, probably yours will be better
just try to improve their life

Offline jumba

  • ****
  • 291
  • +0/-0
  • Donations: July 2007 - $ 20.00
    • Smeserver på svenska!
Re: diference between quota reported and space used on disk
« Reply #3 on: April 22, 2008, 10:31:17 PM »
To list all files owned by a user, you could try something like:

Code: [Select]
ls /home/e-smith/files -lR | grep " accountname " > name_of_container_file

Offline larieu

  • ****
  • 214
  • +0/-0
Re: diference between quota reported and space used on disk
« Reply #4 on: April 23, 2008, 05:34:36 AM »
Exactly as I expected
all files are in
/home/e-smith/files/users/"username" (360M)
plus one system file in
/home/e-smith/files/samba/profiles/"username" (4k)

total amount of data 360M (read by du in that two location)

the quota reported 820M
if everybody's life around you is better, probably yours will be better
just try to improve their life

Offline jumba

  • ****
  • 291
  • +0/-0
  • Donations: July 2007 - $ 20.00
    • Smeserver på svenska!
Re: diference between quota reported and space used on disk
« Reply #5 on: April 23, 2008, 09:25:02 AM »
I've never in my life seen such a thing!

Do you have users with very similar user names?

Do you have old, deleted users with same user name as newly created ones?

I'm only guessing here.....

Is the server upgraded from previous versions???

If you read /etc/password can you find any strange connections between the uuid's?

The lines should reda something like:

Code: [Select]
username:x:5001:5001:firstname lastname:/home/e-smith/files/users/username:/usr/bin/rssh
(In this case, that users uuid is 5001)

Otherwise, please file a bug report...

Offline m

  • ****
  • 276
  • +0/-0
  • Peet
Re: diference between quota reported and space used on disk
« Reply #6 on: April 24, 2008, 10:57:08 PM »
You most likely have files owned by the user outside of /home/e-smith/files

Consider the following scenario:
Create a new new user jsixpack. The uid assigned is e.g. 5001. Let the user create a file in his home dir. Then login as root and copy (cp -a) or hardlink (ln) the file to /var
Delete the account and immediately create a user ssorglos. This user will be assigned again uid 5001. The still existing file in /var is now owned by ssorglos and quota will report the size.



if somebody know a method to find all files "owned" by that user and counted by quota it will help for debug
Code: [Select]
echo `find / -type f -user ACCOUNTNAME -exec du {} \; | awk '{s+=$1}END{print s}'`+`find / -type d -user ACCOUNTNAME | wc -l`*4|bc

This prints the total size in kbytes of all files owned by user ACCOUNTNAME. It should be the same number as the reported blocks of the quota command.

Code: [Select]
quota -v ACCOUNTNAME

(if hardlinked files exist the first command reports a higher value)
 

Offline larieu

  • ****
  • 214
  • +0/-0
Re: diference between quota reported and space used on disk
« Reply #7 on: May 06, 2008, 06:15:07 PM »
Thanks mweinber

but seems to be deeper
I followed your tips
this is the results

for the command
Quote
echo `find / -type f -user ACCOUNTNAME -exec du {} \; | awk '{s+=$1}END{print s}'`+`find / -type d -user ACCOUNTNAME | wc -l`*4|bc

find: /proc/1067: No such file or directory
find: /proc/2468/task: No such file or directory
find: /proc/31448: No such file or directory
1473152
- I appreciate as over 1.2G (if it is 1M it has no sense)

this is for quota
Quote
quota -v ACCOUNTNAME
Disk quotas for user ACCOUNTNAME (uid 5053):
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
/dev/mapper/main-root
                 991568  1048576 1536000            7636       0       0

this seems to be AROUND 900M (which is the same with the web interface)

and the last
the du command
Quote
du -h /home/e-smith/files/users/ACCOUNTNAME
return
509M

this is deeper than I can image
seems to be a dead end
what is strange is that after a while (half a year) this behaviour seems to change the user
if everybody's life around you is better, probably yours will be better
just try to improve their life

Offline stdean

  • *
  • 37
  • +0/-0
Re: diference between quota reported and space used on disk
« Reply #8 on: July 29, 2008, 05:12:06 AM »
Is the a solution to this problem?

I am seeing the same thing happen on my machine, which I suspect is caused by the uuid's. I recently removed an account with 1.8Gb space taken up and created a new account no more than 10 minutes later. It is now reporting the new account is taking up close to 1.8Gb. The uuid of the new user appears to be quite low compared to other users so I'm guessing they have gotten the id of the deleted account.

How do I get it to display the correct quota usage for the new user. It seems strange that it will report the file usage for a deleted account. Do I need to re-start the the machine?

Conor

Offline larieu

  • ****
  • 214
  • +0/-0
Re: diference between quota reported and space used on disk
« Reply #9 on: September 26, 2008, 09:14:25 AM »
I found the root of the troubleHurray! 8-)
to better understand I'll write a "short history"

01. server 1 - old hardware (P3/600 256MRAM 32G/UDMA66 - not important)
02. server 2 - new hardware (P4D/2.4 2GRAM 320G/SATA2 - also not important)
03. using this great cotrib http://wiki.contribs.org/Moving_SME_to_new_Hardware I moved everything to the new hardware (or almost all)
04. starting from this moment I found that difference  between quota repported by quota and counted files using du command
Code: [Select]
du /home/e-smith/files/users/username- using find command I verified all files used by the user with the problem (in my case was the UID 5053)
Code: [Select]
find / -user 5053 > /tmp/repportfile by parsing this file (repportfile) I found that this user has files in /var/affa and several other places
in this moment you need to analyze all places and delete that ones which is not more needed

it could also happen that the "name" of the UID is changed between the moment of backup and the moment you try to find files - because of this is better to look after UID not after name
for example:

you make backup
delete user with name userxa and UID 5050
after some while create another user with name userxb and the server allocate the same UID 5050
the quota difference is reported!

if you'll look after name ( as I deed) you're lost

hope this will help somebody to don't wait or try to solve the trouble from the root
« Last Edit: September 30, 2008, 07:55:59 AM by larieu »
if everybody's life around you is better, probably yours will be better
just try to improve their life

Offline m

  • ****
  • 276
  • +0/-0
  • Peet
Re: diference between quota reported and space used on disk
« Reply #10 on: September 26, 2008, 10:22:45 AM »
I found that this user has files in /var/affa and several other places
in this moment you need to analyze all places and delete that ones which is not more needed
That's why the Wiki document states "/bin/rm -rf /var/affa" in the "Cleaning up" chapter. ;-)

The directory /var/affa contains the entire backup archive. All files are hardlinked to their original locations.
Strange quota reports will also happen when a user deletes files, as the files still physically exists in /var/affa. See my previous post for details.

Offline larieu

  • ****
  • 214
  • +0/-0
Re: diference between quota reported and space used on disk
« Reply #11 on: September 30, 2008, 08:02:40 AM »
Thanks for remmarks mweinber

I don't know exactly why I didn't use that at the end
I think when I used that tutorial that command wasn't there
or to be more honnest
I used for about a month the both servers in parallel for automatically backup one to each other
and probably when I uninstalled affa was my mistake to delete all  that files
probably at that time I supposed  (wrong) that if you uninstall "something" you'll delete also this kind of "additional" files

and my problem was that I wasn't able to find a way to "see" all files used by the userx
the "find" command do this job but unfortunately is not to well documented on web
« Last Edit: September 30, 2008, 08:06:09 AM by larieu »
if everybody's life around you is better, probably yours will be better
just try to improve their life

Offline bcalder01

  • **
  • 47
  • +0/-0
Re: diference between quota reported and space used on disk
« Reply #12 on: February 10, 2009, 01:35:23 PM »
I have been trying to figure out why my user's current useage is so much higher than what's in their home directories.

> You most likely have files owned by the user outside of /home/e-smith/files

Does this mean that if they own a file in a departmental ibay, it "counts" against their quota?? That makes no sense, if so!

Is there a way to set a quota on their home drive & anything they create/save in an departmental ibay is quota-less?

Offline larieu

  • ****
  • 214
  • +0/-0
Re: diference between quota reported and space used on disk
« Reply #13 on: February 10, 2009, 03:00:54 PM »
I'm not pretty sure about what you sad
but is an easy way to find out

first of all find the UID of the "trouble" user
in my "short" story you'll find the "find command" where you can change UID and destination file
and after that you can look into that "destination file" (use page down, and for sure it will punch you the root of the file path - that one which you don't expect to have for that user)

I agree with you
in my wish list is to have separate quota for user, ibay, ftp/user, roaming/user
if everybody's life around you is better, probably yours will be better
just try to improve their life

Offline bcalder01

  • **
  • 47
  • +0/-0
Re: diference between quota reported and space used on disk
« Reply #14 on: February 10, 2009, 03:19:46 PM »
Thanks for the response, Larieu, sorry for being inarticulate.

I found the UID for one of my "problem" users & you were right:
Disk quotas for user username (uid 5081):
/dev/mapper/main-root

The issue is that everyone writes to the ibays, but I only want to set quotas on home drives. Let's hope enough people ask for this modification.