[Exherbo-dev] new distfiles mirror setup

Ingmar Vanhassel ingmar at exherbo.org
Tue Nov 24 13:31:10 GMT 2009

Excerpts from Kim Højgaard-Hansen's message of Tue Nov 24 14:17:51 +0100 2009:
> I'm working on setting up a new distfiles.exherbo.org box, but before 
> doing a lot of work i would like to discuss how to do it.
> The current distfiles.exherbo.org setup "works" but has some problems 
> which would be nice to avoid in the new setup.
> 1. accerso use the same set of repositories as paludis/cave on the 
> machine, meaning we often have to reinstall paludis by hand since the 
> repositories are synced before we react to Ciaran mass breakages
> 2. when new repositories are added they have to be added manually to 
> distfiles.exherbo.org
> 3. the script to run accerso is somewhat nasty, in my oponion mostly 
> because accerso is a NoConfig client, meaning we have to specify all 
> master repositories on the cmdline
> 4. accerso does not respect the "listed-only" label for downloads, which 
> means we fetch stuff like flash etc. Currently those are removed 
> manually at the end of the fetch script
> 5. unused distfiles are not removed automatically
> 6. when distfiles are removed from distfiles.exherbo.org at least all 
> the GPL ones should be kept (forever?), this is also handled manually 
> and eating up my server space :) we have ~14GB of old distfiles now
> There is also a couple of things that could be nice to have:
> - i would like to mirror unofficial repositories also if it can be done 
> sanely, the new box has plenty of space
> - it could perhaps be nice to use the BUGS_TO information for error 
> mails instead of putting the fixing burden on a few people only?
> Solution ideas:
> 1. should be quite easy to fix by having another set of repositories and 
> syncing those with a script

s/with a script/cave sync/
with sync_options = --reset (but the playboy repository configuration
already sets that for you)

> 2. i've talked to ingmar about using the information from unavailable to 
> automate this

Yup, this requires either point 3, or a bit of a hack to pass
configuration options to accerso (& NoConfig clients in general)

> 3. A SomeConfig client might be nice, but that seems non-trivial to 
> implement, suggestions?
> 4. i'll try to fix that if it's a bug in accerso/paludis
> 5. find_unused_distfiles.rb could be used, but i don't think it makes 
> sense to remove all unused distfiles. Instead i would keep X GB of free 
> space, and just remove the oldest unused distfiles to keep the free 
> space. Personally i would like that to be a part of the mirror client

As you said on IRC, most users of accerso will probably want to have
their mirror cleaned up from distfiles that aren't used anymore. So it
makes sense to add this in & optionally run it after mirroring

> 6. I've been thinking about how to handle this sanely, but haven't come 
> up with anything. Suggestions please

No idea, how do other distributions handle this? Does Gentoo provide
distfiles back to the beginning? Does Ubuntu allow access to all their
uploads from all their development releases?

> /Kim

Exherbo KDE, X.org maintainer

More information about the Exherbo-dev mailing list