Do you have clamav installed? If so, do you have it setup to do a scheduled filesystem scan? If so, change that to never. It had been reported a while back by someone that if you have this set that the file permissions were being changed. If not, what other contribs do you have installed?
JB