Scope and purpose
This story aims to cover use case where:
- site administrators (in pink below) control the network (hence connexion to external servers) and infrastructures (bandwidths, connexion, etc)
- mirror admin (in blue below) control which git repositories are replicated (from the allowed servers)
- regular project administrators and developers (in red below) use standard Tuleap features (fork & clone / push) to interact with the sources
Use case
Typical scenario:
- Site admin declare one new server (eg. eclipse.org) with:
- A name
- A base url (either git://, http:// or https://)
- http(s) might require a proxy
- git support depends on company network capabilities
- ssh is not supported
- A mirroring frequency (cron style)
- Site admin grants a specific project "external mirroring" capability
- This is a regular Tuleap project
- It's recommended that one project is dedicated for the mirroring purpose solely
- Mirror administrators (eg. Eclipse Mirror Project) can then create a new git repository (eg. papyrus)
- At creation, admin can choose to mirror a repository from the servers listed by site admin
- As soon as one serveur is granted, mirror admins can replicate any repository from the serveur (eg. once github.com is granted, any repository from github can be mirrored)
- All features of regular Tuleap git repositories are supported (notifications, permissions, etc)
- Mirror projects and git repositories should be public (so anyone can clone from them)
- Regular Project administrators can then fork from "Mirror project" inside their own project (eg papyrus@mycomp)
- The plus size of this approach is that Tuleap can make a "hard link clone" server side (hence save place)
- Developers will:
- Clone 'papyrus@mycomp'
- Add "papyrus mirror" as remote
- Then, typical development cycle will be:
- Fetch from "papyrus mirror"
- Push to papyrys@mycomp
How it works ?
- This is a new plugin (depends on git)
- There is a site admin configuration panel to CRUD servers (name, urls, cron frequencies)
- Project granted as mirrors are managed through existing "enable plugin for project feature"
- There is a new option for git repository creation: replicate from mirror:
- string/text field with full path to the repo
- tuleap check that base url corresponds to selected serveur
- if possible, check if it's a valid git repo
- There is a new setting in git repository with mirror info:
- Mirror url
- Last sync job + status
- Notification email in case of failure
- There is a backend job (dedicated queue) that:
- Fetch all repositories to be synchronized
- Iterate over them according to polling frequencies
- When sync fail, an email is sent to notification list defined before
Performances and scalability
In this first story, the purpose is to design and implement the global behaviour. No special considerations are taken for management of very big repositories.
For one serveur, repositories synchronization are done as a sequence in order to limit the risk of server overload. This means that:
- if one serveur is slow, all mirroring events for this servers will be impacted
- but repositories from other servers will not be impacted
- if one repository on a given server is slow, this will slowdown all subsequent mirrors
Smarter mirroring strategies might be needed in the future but this will be covered in other stories