Yeah you could set this up. After your main server downloads the files you could create a repository on that server and point your other servers that that repository. Create an ibay with your yum files, then do a yum-arch /path_to_your_files/. Modify your other servers' yum.conf to point to your server and they should download the files from there.
That's a high-level overview. I did this a while back on some test servers and didn't have any issues.
JB