Skip to main content

Public

Project privacy set to public. By default, its content is available to everyone (authenticated or not). Please note that more restrictive permissions might exist on some items.

  •  
     
    story #7516 have my repositories replicated on selected mirror
Actions
Summary
site admin
have my repositories replicated on selected mirror
Empty
Empty
By Loïc Lefort (lefort)
(2 B)
manifest_template.js
Manifest template
By Loïc Lefort (lefort)
(31 B)
manifest_template.js.gz
Compressed manifest template
loic.lefort@st.com, Nouha Terzi (terzino), Denis PILAT (denis_pilat)
Status
Empty
Done
Development
  • [ ] Does it involves User Interface? 
  • [ ] Are there any mockups?
  • [ ] Are permissions checked?
  • [ ] Does it need Javascript development?
  • [ ] Does it need a forge upgrade bucket?
  • [ ] Does it need to execute things in system events?
  • [ ] Does it impact project creation (templates)?
  • [ ] Is it exploratory?
Empty
Details
#7516
Manuel Vacelet (vaceletm)
2014-10-21 12:27
2014-10-09 10:19
7522

References
Artifact links
Empty
Referencing story #7516

Artifact Tracker v5

rel #7433 7.6
epic #7321 Git mirroring with grokmirror

Git commit

tuleap/tuleap/stable

On mirroring, grant mirror user read permission on repository c9434872b1
User avatar Manuel Vacelet (vaceletm) , 2014-10-10 14:14
All mirrors should be allowed to read gitolite admin repo 4b278b9076
User avatar Manuel Vacelet (vaceletm) , 2014-10-10 14:15
Generate one manifest per mirror f602bba71f
User avatar Manuel Vacelet (vaceletm) , 2014-10-10 14:15
Display settings

Follow-ups

User avatar
Nouha Terzi (terzino)2014-10-21 12:27
I was missing the grokmirror path in the git config. It is ok now. thank you
  • Category set to
User avatar
Manuel Vacelet (vaceletm)2014-10-17 13:04
So the manifests are not generated, please note that their will be 1 manifest per mirror:
manifest_mirror_1.js.gz
manifest_mirror_2.js.gz
...

You need to lower the log level, in local inc, you should add:
$sys_logger_level = 'debug';

And activate again mirroring on a repo
User avatar
Nouha Terzi (terzino)2014-10-17 12:29
[root@crx19007 grokmirror]# ll
total 16
-rw-r----x 1 gitolite gitolite 9878 Oct 17 10:29 manifest.js
-rw-r----- 1 gitolite gitolite 2198 Oct 16 15:08 manifest.js.gz

In /var/log/codendi/codendi_syslog, just my system_event with the status done.
User avatar
Manuel Vacelet (vaceletm)2014-10-17 12:07
What do you have in /var/lib/tuleap/gitolite/grokmirror ?
And do you see any useful info in /var/log/codendi/codendi_syslog ? (look at "grok" or "post-recieve")
User avatar
Nouha Terzi (terzino)2014-10-17 10:59
Hello Manuel,

I am testing tuleap 7.5.99.72

1- I run grokmanifest to have the manifest:
grok-manifest -m /var/lib/codendi/gitolite/grokmirror/manifest.js -t /var/lib/codendi/gitolite/repositories


2- I added a new mirror

3- I configured the mirror with the forge user and added it to the " Retrieve User Membership Information" group

4- I created a new git repository

5- i selected it for the mirroring ( all my system events are PASSED)

However, I don't see any modification on the manifest.

could you please tell me what is wrong with my use case?


thank you for your support.


regards,
Nouha
User avatar
Loïc Lefort (lefort)2014-10-13 09:42
For point 1, grokmirror generates a full list when manifest file does not exist.
I would recommend to use an empty template (you can find them attached) when creating a new mirror then when adding a git, only git in argument is added in manifest.
User avatar
Nouha Terzi (terzino)2014-10-10 15:55
For point 1, I think we can make use of the --check-export-ok flag (ps: we haven't tested it).
It will not remove the repositories belonging to other mirror, but at least it reduces the exported repositories' number to manifests.
User avatar
Loïc Lefort (lefort)2014-10-10 15:22
I would add it in paragrah "8.14.1. Setup tuleap-gitolite-membership", at the end of step 3.
User avatar
Manuel Vacelet (vaceletm)2014-10-10 15:14

For @terzino :
  So $UNSAFE_PATT seems to have sovled 'bad config' issue

For @lefort:
  I was not aware of this gitolite command, at which step should it be added in the doc ?

User avatar
Loïc Lefort (lefort)2014-10-10 15:02
Do you have used following command to disable write access from mirror ?
gitolite writable @all off
User avatar
Nouha Terzi (terzino)2014-10-10 14:47
In .gitolite.rc, do you have this section after %RC = ( ...........); and before 1; :
$UNSAFE_PATT = qr();
User avatar
Manuel Vacelet (vaceletm)2014-10-10 14:32

Ok so as of Tuleap 7.5.99.58, you should be able to test with Tuleap that generates manifests.
I need your review of the admin doc because I'm not really confident about the mirror part (esp. integration with gitolite)

Esp, when I run grok-pull, I get several errors:

Error 1:

Traceback (most recent call last):
  File "/usr/bin/grok-pull", line 592, in <module>
    sect_retval = pull_mirror(section, config, opts)
  File "/usr/bin/grok-pull", line 501, in pull_mirror
    set_owner_description(toplevel, gitdir, owner, desc)
  File "/usr/bin/grok-pull", line 62, in set_owner_description
    if description is not None and repo.description != description:
  File "/usr/lib/python2.6/site-packages/git/repo/base.py", line 163, in _get_description
    return file(filename).read().rstrip()
IOError: [Errno 2] No such file or directory: u'/var/lib/gitolite3/repositories/mozilla/u/vaceletm/firefox.git/description'

Error 2: (in gitolite logs)

2014-10-09.14:03:22     701             system,gitolite compile
2014-10-09.14:03:22     701     die     bad config value 't=%s; git show --name-status --pretty='format:URL: https://red.tuleap-aio-dev.dev.docker/plugins/git/mozilla/firefox?p=firefox.git&a=commitdiff&h=%%H%%nAuthor: %%an <%%ae>%%nDate: %%aD%%n%%n%%s%%n%%b' $t'
2014-10-09.14:03:22     701             system() failed,gitolite compile,-> 6400

How did you work around those issues ?

In addition to that, when I run other grok-pull (after the inital one) I have to add -r switch otherwise I get following error:

-sh-4.1$ /usr/bin/grok-pull -p -c /etc/grokmirror/repos.conf
Found existing repository in /var/lib/gitolite3/repositories/gitolite-admin.git
Run with -r to use existing repos

Did you get that too ?

Finally we identified 2 major issues:

  1. Even if we separate the mirrors (different grokmirror-manifest calls per repo / mirrors), each time ALL the repositories are listed in ALL manifests
  2. When we manage to replicate gitolite-admin repo, the final permissions that applies on the mirror are the same than master. It means that users have WRITE access to mirror!

For the first issue, there is an alternative: tuleap can generate a grokmirror compatible manifest.

For the second issue it's still open...

What do you think ?

 

  • Status changed from Ready (stalled) to Done
  • CC list set to Denis PILAT (denis_pilat), Nouha Terzi (terzino), loic.lefort@st.com