[Exherbo-dev] new distfiles mirror setup

Kim Højgaard-Hansen kim at khh.dk
Tue Nov 24 13:17:51 GMT 2009


I'm working on setting up a new distfiles.exherbo.org box, but before 
doing a lot of work i would like to discuss how to do it.

The current distfiles.exherbo.org setup "works" but has some problems 
which would be nice to avoid in the new setup.

1. accerso use the same set of repositories as paludis/cave on the 
machine, meaning we often have to reinstall paludis by hand since the 
repositories are synced before we react to Ciaran mass breakages
2. when new repositories are added they have to be added manually to 
distfiles.exherbo.org
3. the script to run accerso is somewhat nasty, in my oponion mostly 
because accerso is a NoConfig client, meaning we have to specify all 
master repositories on the cmdline
4. accerso does not respect the "listed-only" label for downloads, which 
means we fetch stuff like flash etc. Currently those are removed 
manually at the end of the fetch script
5. unused distfiles are not removed automatically
6. when distfiles are removed from distfiles.exherbo.org at least all 
the GPL ones should be kept (forever?), this is also handled manually 
and eating up my server space :) we have ~14GB of old distfiles now

There is also a couple of things that could be nice to have:
- i would like to mirror unofficial repositories also if it can be done 
sanely, the new box has plenty of space
- it could perhaps be nice to use the BUGS_TO information for error 
mails instead of putting the fixing burden on a few people only?

Solution ideas:
1. should be quite easy to fix by having another set of repositories and 
syncing those with a script
2. i've talked to ingmar about using the information from unavailable to 
automate this
3. A SomeConfig client might be nice, but that seems non-trivial to 
implement, suggestions?
4. i'll try to fix that if it's a bug in accerso/paludis
5. find_unused_distfiles.rb could be used, but i don't think it makes 
sense to remove all unused distfiles. Instead i would keep X GB of free 
space, and just remove the oldest unused distfiles to keep the free 
space. Personally i would like that to be a part of the mirror client
6. I've been thinking about how to handle this sanely, but haven't come 
up with anything. Suggestions please

/Kim




More information about the Exherbo-dev mailing list