•  
      request #13842 update broke nginx resolve 502 Bad Gateway
    Infos
    #13842
    Anton KULIK (d00AK)
    2019-09-11 10:33
    2019-09-03 12:02
    15055
    Details
    update broke nginx resolve 502 Bad Gateway
    The update on Sunday from 11.4 to 11.5 broke ngnix on CentOS 6 resolve into 502 Bad Gateway
    tuleap.noarch 11.5.99.83-1.el6

    The end of php7.2 has nothing to do with the issue since all migration to 7.3 procedures listed in:
    https://docs.tuleap.org/deployment-guide/intro.html#tuleap-11-4
    and nginx config update to:

    upstream tuleap-apache {
    server 127.0.0.1:8080;
    }
    upstream tuleap-php-fpm {
    server 127.0.0.1:9000;
    }
    upstream tuleap-php-fpm-long-running-request {
    server 127.0.0.1:9002;
    }

    were performed on earlier upgrades to 11.4 and were working OK.

    However, on Sunday, an update of PHP7.3 packages and Tuleap to 11.5
    led to successful start of Tuleap and nginx, but with brocken paths (502 Bad Gateway):

    before upgrade script:
    service tuleap stop && service nginx stop

    after upgrade script:
    /usr/lib/forgeupgrade/bin/forgeupgrade --config=/etc/tuleap/forgeupgrade/config.ini update; /usr/share/tuleap/tools/utils/php73/run.php --module=nginx; service nginx start; service tuleap start

    Neither system reboot, nor reapplying:

    service php73-php-fpm stop
    /usr/share/tuleap/tools/utils/php73/run.php --module=fpm
    service php73-php-fpm start
    service tuleap restart
    service nginx restart

    fixes 502 Bad Gateway. But all above commands perform OK.











    Upgrade process
    11.5
    CentOS 6
    • [ ] enhancement
    • [ ] internal improvement
    Empty
    Stage
    Empty
    Declined
    2019-09-10
    Attachments
    References
    References list is empty

    Follow-ups

    User avatar
    Anton KULIK (d00AK)2019-09-11 10:33
    Yep, that is what it did, I figured out the troubleshooting procedure myself.
    But that would have been hardly possible without your hints guys! Thanks for that!

    Being a full-stack dev is quite challenging in terms of learning curve and time spent on digging into various details.
    That is why I can not afford myself diving into PHP. At this point hints from your side are of paramount importance :)
    User avatar

    Good to know the problem is solved

    @2@You have to check your fpm configuration and fix them or adapt you nginx conf.

    That is out of my expertise. I try to avoid everything PHP (no offence here, I'm focused on Node/VanillaJS stack).

    When you are running Tuleap Community you actually are on your own and you will somehow have to deal with php stuff, no offence ;)

    User avatar
    Anton KULIK (d00AK)2019-09-10 18:56
    last edited by: Anton KULIK (d00AK) 2019-09-10 18:57
    SOLVED:
    changed ports 9001 and 9003 were found in:
    /etc/opt/remi/php73/php-fpm.d/tuleap.conf
    and
    /etc/opt/remi/php73/php-fpm.d/tuleap-long-running-request.conf

    FULL TROUBLESHOOTING PROCEDURE:
    (a) check if PHP-FPM ports align
    in nginx config:
    /etc/nginx/conf.d/tuleap.conf
    with php tuleap configs:
    /etc/opt/remi/php73/php-fpm.d/tuleap.conf
    /etc/opt/remi/php73/php-fpm.d/tuleap-long-running-request.conf
    (b) correct ports in php tuleap configs if needed
    (c) reboot server
    (d) enjoy proper 9000 and 9002 php-fpm ports (no HACKs needed).
    User avatar
    Anton KULIK (d00AK)2019-09-10 18:09
    @1@That's not a hack your platform is running fpm on different ports than 100% of other plarforms.

    I think I caused the port switch accidentally by running php72 on Sep. 1 2019:
    /usr/share/tuleap/tools/utils/php72/run.php --module=nginx
    instead of php73
    /usr/share/tuleap/tools/utils/php73/run.php --module=nginx

    in the tuleap post upgrade sequence. Then correcting 72 to 73 and running again.
    If both 72 and 73 where running, 73 could have got nearest next ports available, i.e. 9001 and 9003.

    @2@You have to check your fpm configuration and fix them or adapt you nginx conf.

    That is out of my expertise. I try to avoid everything PHP (no offence here, I'm focused on Node/VanillaJS stack).

    @3@From what we can see there are far more stuff running on your server than tuleap (all those php-cgi and /php-loop.pl), you are on your own with that.

    I have no idea what that PHP stuff running is. Nothing PHP related could have ever been installed on my server if not required by Tuleap.

    Again, thank's for the help with figuring the routing problem.
    This request can be considered as resolved now, and can be closed.
    User avatar

    That's not a hack your platform is running fpm on different ports than 100% of other plarforms.

    You have to check your fpm configuration and fix them or adapt you nginx conf.

    From what we can see there are far more stuff running on your server than tuleap (all those php-cgi and /php-loop.pl), you are on your own with that.

    User avatar
    Anton KULIK (d00AK)2019-09-10 17:46
    Back to service now. This is a hack, will it survive future upgrades?!

    Just in case I added comments to tuleap.conf:

    upstream tuleap-apache { # netstat -lntp | grep 'nginx\|httpd\|php-fpm'
    server 127.0.0.1:8080; # 8080:httpd; 443:nginx
    }

    upstream tuleap-php-fpm { # netstat -lntp | grep 'nginx\|httpd\|php-fpm'
    server 127.0.0.1:9001; # changed to 9001 => was 9000 before 2019.09.01 update 502 error
    }

    upstream tuleap-php-fpm-long-running-request { # netstat -lntp | grep 'nginx\|httpd\|php-fpm'
    server 127.0.0.1:9003; # changed to 9003 => was 9002 before 2019.09.01 update 502 error
    }
    User avatar

    Try to update your 2 last nginx upstream with

    upstream tuleap-php-fpm {
      server 127.0.0.1:9001;
    }
    
    upstream tuleap-php-fpm-long-running-request {
      server 127.0.0.1:9003;
    }

    and restart your nginx server and your php-fpm service.

    And to know why php-fpm listen to these ports instead of 9000 and 9002, I don't know.

    User avatar
    Anton KULIK (d00AK)2019-09-10 17:24
    # netstat -lntp | grep 'php-fpm'
    tcp 0 0 127.0.0.1:9001 0.0.0.0:* LISTEN 28581/php-fpm
    tcp 0 0 127.0.0.1:9003 0.0.0.0:* LISTEN 28581/php-fpm
    #

    what to do?
    User avatar
    So it seems that php-fpm does not listen on 9000 and 9002, which are the ports of 2 nginx upstreams.

    I've got on my side:
    tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 1818/nginx
    tcp 0 0 127.0.0.1:9000 0.0.0.0:* LISTEN 1851/php-fpm
    tcp 0 0 127.0.0.1:9002 0.0.0.0:* LISTEN 1851/php-fpm
    tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 1884/httpd

    Maybe you have to check something about this.
    User avatar
    Anton KULIK (d00AK)2019-09-10 17:20
    # netstat -lntp | grep '443\|8080\|9000\|9002'
    tcp 0 0 5.189.133.180:443 0.0.0.0:* LISTEN 15530/nginx
    tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 2416/httpd
    #
    User avatar
    Anton KULIK (d00AK)2019-09-10 17:05
    # ps aux | grep nginx
    avegaon 2514 0.0 0.0 127584 3744 ? S Sep01 0:00 /usr/bin/perl /usr/bin/php-loop.pl /usr/bin/php-cgi -b /var/php-nginx/14065648283943.sock/socket
    avegaon 2521 0.0 0.1 396572 14060 ? Ss Sep01 0:00 /usr/bin/php-cgi -b /var/php-nginx/14065648283943.sock/socket
    515 2523 0.0 0.0 127584 3740 ? S Sep01 0:00 /usr/bin/perl /usr/bin/php-loop.pl /usr/bin/php-cgi -b /var/php-nginx/14712982522448.sock/socket
    515 2531 0.0 0.0 127584 3740 ? S Sep01 0:00 /usr/bin/perl /usr/bin/php-loop.pl /usr/bin/php-cgi -b /var/php-nginx/14279059069671.sock/socket
    515 2532 0.0 0.1 400936 14156 ? Ss Sep01 0:00 /usr/bin/php-cgi -b /var/php-nginx/14712982522448.sock/socket
    515 2541 0.0 0.1 396572 14056 ? Ss Sep01 0:00 /usr/bin/php-cgi -b /var/php-nginx/14279059069671.sock/socket
    avegaon 2542 0.0 0.0 396572 5892 ? S Sep01 0:00 /usr/bin/php-cgi -b /var/php-nginx/14065648283943.sock/socket
    avegaon 2543 0.0 0.0 396572 5892 ? S Sep01 0:00 /usr/bin/php-cgi -b /var/php-nginx/14065648283943.sock/socket
    avegaon 2544 0.0 0.0 396572 5892 ? S Sep01 0:00 /usr/bin/php-cgi -b /var/php-nginx/14065648283943.sock/socket
    avegaon 2545 0.0 0.0 396572 5892 ? S Sep01 0:00 /usr/bin/php-cgi -b /var/php-nginx/14065648283943.sock/socket
    515 2550 0.0 0.0 396572 5888 ? S Sep01 0:00 /usr/bin/php-cgi -b /var/php-nginx/14279059069671.sock/socket
    515 2551 0.0 0.0 396572 5888 ? S Sep01 0:00 /usr/bin/php-cgi -b /var/php-nginx/14279059069671.sock/socket
    515 2552 0.0 0.0 396572 5888 ? S Sep01 0:00 /usr/bin/php-cgi -b /var/php-nginx/14279059069671.sock/socket
    515 2553 0.0 0.0 396572 5888 ? S Sep01 0:00 /usr/bin/php-cgi -b /var/php-nginx/14279059069671.sock/socket
    root 15530 0.0 0.0 86132 4296 ? Ss 16:14 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
    nginx 15531 0.0 0.1 92856 14132 ? S 16:14 0:00 nginx: worker process
    nginx 15532 0.0 0.1 92856 14140 ? S 16:14 0:00 nginx: worker process
    root 26576 0.0 0.0 103320 896 pts/0 S+ 17:01 0:00 grep nginx
    515 28014 0.0 0.7 460688 61136 ? S Sep09 1:03 /usr/bin/php-cgi -b /var/php-nginx/14712982522448.sock/socket
    515 29646 0.0 0.8 469480 69928 ? S Sep09 1:02 /usr/bin/php-cgi -b /var/php-nginx/14712982522448.sock/socket
    515 29647 0.0 0.8 469448 69900 ? S Sep09 1:02 /usr/bin/php-cgi -b /var/php-nginx/14712982522448.sock/socket
    515 29648 0.0 0.7 462224 62672 ? S Sep09 1:03 /usr/bin/php-cgi -b /var/php-nginx/14712982522448.sock/socket
    # service nginx status
    nginx (pid 15530) is running...
    #
    User avatar
    Anton KULIK (d00AK)2019-09-10 17:01
    # ps aux | grep fpm
    root 25684 0.0 0.0 103316 888 pts/0 S+ 16:58 0:00 grep fpm
    root 28581 0.0 0.0 509348 7632 ? Ss Sep04 0:26 php-fpm: master process (/etc/opt/remi/php73/php-fpm.conf)
    497 28582 0.0 0.0 509288 7440 ? S Sep04 0:00 php-fpm: pool tuleap-long-running-request
    497 28583 0.0 0.0 509288 7432 ? S Sep04 0:00 php-fpm: pool tuleap-long-running-request
    497 28584 0.0 0.0 509288 7432 ? S Sep04 0:00 php-fpm: pool tuleap-long-running-request
    497 28585 0.0 0.0 509288 7436 ? S Sep04 0:00 php-fpm: pool tuleap-long-running-request
    497 28586 0.0 0.0 509288 7436 ? S Sep04 0:00 php-fpm: pool tuleap-long-running-request
    497 28587 0.0 0.0 509288 7440 ? S Sep04 0:00 php-fpm: pool tuleap
    497 28588 0.0 0.0 509288 7440 ? S Sep04 0:00 php-fpm: pool tuleap
    497 28589 0.0 0.0 509288 7440 ? S Sep04 0:00 php-fpm: pool tuleap
    497 28590 0.0 0.0 509288 7448 ? S Sep04 0:00 php-fpm: pool tuleap
    497 28592 0.0 0.0 509288 7448 ? S Sep04 0:00 php-fpm: pool tuleap
    User avatar
    Tuleap is still fully tested on CentOS 6.

    We have multiple other instances running 11.5 on CentOS 6 without any issue.
    User avatar
    Anton KULIK (d00AK)2019-09-10 16:27
    last edited by: Anton KULIK (d00AK) 2019-09-10 16:29
    yes, those are harmless warnings that existed before (see last logs zip).
    nginx works like swiss watch. And, at time of 502 upgrade (around 30-31 August till 1-2 September) no special nginx errors logged.
    Just to check stapling cause I did disable 'stapling: on' in tuleap config, but this had no effect on 502 Bad gateway (wich was expected).

    Does the 502 mean that Tuleap development in not being tested on CentOS 6 any more, and is only tested on CentOS 7 before deploy?

    User avatar
    Anton KULIK (d00AK)2019-09-10 16:05
    if needed, I can clean-up tuleap.conf from commented out lines for greater readability.
    User avatar
    Anton KULIK (d00AK)2019-09-10 15:59
    # /usr/share/tuleap/tools/utils/php73/run.php --module=nginx
    info Configure Tuleap for PHP 7.3 / FPM and Nginx
    info [nginx] Start configuration
    info [nginx] Deploy configuration chunks in /etc/nginx
    info [nginx] Deploy configuration chunks in /etc/nginx/conf.d/tuleap.d
    info [nginx] Deploy configuration chunks in /etc/nginx/conf.d/tuleap-plugins
    info [nginx] Configuration done!
    info Configuration completed
    User avatar
    What is the output of /usr/share/tuleap/tools/utils/php73/run.php --module=nginx on your server ?

    In addition, is it possible to attach the /etc/nginx/conf.d/tuleap.conf file ?
    User avatar
    Anton KULIK (d00AK)2019-09-10 14:56
    Nope, nginx is working fine. Returning Tuleap favicon without problems on a 502 Bad Gateway page.
    The only warning in error in log is:
    2019/09/08 03:37:21 [warn] 29823#29823: "ssl_stapling" ignored, issuer certificate not found for certificate "/home/congineer/ssl.cert"

    This is either a routing issue for page assets in PHP config or the side effect of the only error logged in /var/log/tuleap/statistics_syslog

    #7 /usr/share/tuleap/src/common/plugin/Plugin in /usr/share/tuleap/src/vendor/paragonie/easydb/src/Factory.php on line 49

    The error is sent once a week by Cron, last one sent on 08.09.2019:

    Subject:
    Cron <codendiadm@hCODE02> /usr/bin/tuleap process-system-event grokmirror

    Mail Body:
    PHP Fatal error: Uncaught ParagonIE\EasyDB\Exception\ConstructorFailed: Could not create a PDO connection. Please check your username and password. in /usr/share/tuleap/src/vendor/paragonie/easydb/src/Factory.php:99
    Stack trace:
    #0 /usr/share/tuleap/src/common/DB/DBCreator.php(45): ParagonIE\EasyDB\Factory::fromArray(Array)
    #1 /usr/share/tuleap/src/common/DB/DBConnection.php(46): Tuleap\DB\DBCreator->createDB()
    #2 /usr/share/tuleap/src/common/DB/Compat/Legacy2018/CompatPDODataAccess.php(63): Tuleap\DB\DBConnection->getDB()
    #3 /usr/share/tuleap/src/common/dao/include/DataAccessObject.class.php(111): Tuleap\DB\Compat\Legacy2018\CompatPDODataAccess->query('SELECT * FROM f...', Array)
    #4 /usr/share/tuleap/src/common/Config/ConfigDao.php(27): DataAccessObject->retrieve('SELECT * FROM f...')
    #5 /usr/share/tuleap/src/common/Config/ConfigValueDatabaseProvider.php(38): ConfigDao->searchAll()
    #6 /usr/share/tuleap/src/common/Config/ForgeConfig.php(46): ConfigValueDatabaseProvider->getVariables()
    #7 /usr/share/tuleap/src/common/Config/Fo in /usr/share/tuleap/src/vendor/paragonie/easydb/src/Factory.php on line 99
    User avatar
    Anton KULIK (d00AK)2019-09-10 14:29
    Any ideas? Or shall I consider my Tuleap server a dead system?
    User avatar
    Anton KULIK (d00AK)2019-09-05 10:32

    All logs are OK except for /var/log/tuleap/statistics_syslog (I replaced password with 'UNREADABLE_PASSWORD' below)

    2018-03-26T13:13:12+02:00 [15496] [debug] Processing statistics queue.
    2018-03-26T13:13:12+02:00 [15496] [debug] Restore ownership to /var/tmp/tuleap_cache
    2018-03-26T13:13:12+02:00 [15496] [debug] Restore ownership to /var/tmp/tuleap_cache/lang
    2018-03-26T13:13:12+02:00 [15496] [debug] Restore ownership to /var/tmp/tuleap_cache/hooks.json
    2018-03-26T13:14:25+02:00 [15586] [debug] Processing statistics queue.
    2018-03-26T13:14:25+02:00 [15586] [debug] Restore ownership to /var/tmp/tuleap_cache
    2018-03-26T13:14:25+02:00 [15586] [debug] Restore ownership to /var/tmp/tuleap_cache/lang
    2018-03-26T13:14:25+02:00 [15586] [debug] Restore ownership to /var/tmp/tuleap_cache/hooks.json
    2018-05-20T00:45:01+02:00 [13958] [error] [SystemEventProcessor] An error happened during execution of post actions: Unable to run command "/usr/share/tuleap/src/utils/php-launcher.sh$
    Stack trace:
    #0 /usr/share/tuleap/src/common/DB/DBFactory.php(41): ParagonIE\EasyDB\Factory::create('mysql:host=loca...', 'codendiadm', 'UNREADABLE_PASSWORD')
    #1 /usr/share/tuleap/src/common/DB/DBFactory.php(31): Tuleap\DB\DBFactory::getDB('tuleap')
    #2 /usr/share/tuleap/src/common/DB/DataAccessObject.php(34): Tuleap\DB\DBFactory::getMainTuleapDB()
    #3 /usr/share/tuleap/plugins/git/include/gitPlugin.class.php(1800): Tuleap\DB\DataAccessObject->__construct()
    #4 /usr/share/tuleap/plugins/git/include/gitPlugin.class.php(1796): GitPlugin->getGitDao()
    #5 /usr/share/tuleap/plugins/git/include/gitPlugin.class.php(1775): GitPlugin->getRepositoryFactory()
    #6 /usr/share/tuleap/plugins/git/include/gitPlugin.class.php(398): GitPlugin->getGitSystemEventManager()
    #7 /usr/share/tuleap/src/common/plugin/Plugin in /usr/share/tuleap/src/vendor/paragonie/easydb/src/Factory.php on line 49

    Fatal error: Uncaught exception 'ParagonIE\EasyDB\Exception\ConstructorFailed' with message 'Could not create a PDO connection. Please check your username and password.' in /usr/share$
    Stack trace:
    #0 /usr/share/tuleap/src/common/DB/DBFactory.php(41): ParagonIE\EasyDB\Factory::create('mysql:host=loca...', 'codendiadm', 'UNREADABLE_PASSWORD')
    #1 /usr/share/tuleap/src/common/DB/DBFactory.php(31): Tuleap\DB\DBFactory::getDB('tuleap')
    #2 /usr/share/tuleap/src/common/DB/DataAccessObject.php(34): Tuleap\DB\DBFactory::getMainTuleapDB()
    #3 /usr/share/tuleap/plugins/git/include/gitPlugin.class.php(1800): Tuleap\DB\DataAccessObject->__construct()
    #4 /usr/share/tuleap/plugins/git/include/gitPlugin.class.php(1796): GitPlugin->getGitDao()
    #5 /usr/share/tuleap/plugins/git/include/gitPlugin.class.php(1775): GitPlugin->getRepositoryFactory()
    #6 /usr/share/tuleap/plugins/git/include/gitPlugin.class.php(398): GitPlugin->getGitSystemEventManager()
    #7 /usr/share/tuleap/src/common/plugin/Plugin in /usr/share/tuleap/src/vendor/paragonie/easydb/src/Factory.php on line 49

     

    This corresponds to getting an email form Cron Deamon once a week since then with:

    PHP Fatal error:  Uncaught ParagonIE\EasyDB\Exception\ConstructorFailed: Could not create a PDO connection. Please check your username and password. in /usr/share/tuleap/src/vendor/paragonie/easydb/src/Factory.php:99
    Stack trace:
    #0 /usr/share/tuleap/src/common/DB/DBCreator.php(45): ParagonIE\EasyDB\Factory::fromArray(Array)
    #1 /usr/share/tuleap/src/common/DB/DBConnection.php(46): Tuleap\DB\DBCreator->createDB()
    #2 /usr/share/tuleap/src/common/DB/Compat/Legacy2018/CompatPDODataAccess.php(63): Tuleap\DB\DBConnection->getDB()
    #3 /usr/share/tuleap/src/common/dao/include/DataAccessObject.class.php(104): Tuleap\DB\Compat\Legacy2018\CompatPDODataAccess->query('SELECT * FROM f...', Array)
    #4 /usr/share/tuleap/src/common/Config/ConfigDao.php(26): DataAccessObject->retrieve('SELECT * FROM f...')
    #5 /usr/share/tuleap/src/common/Config/ConfigValueDatabaseProvider.php(36): ConfigDao->searchAll()
    #6 /usr/share/tuleap/src/common/Config/ForgeConfig.php(45): ConfigValueDatabaseProvider->getVariables()
    #7 /usr/share/tuleap/src/common/Config/Fo in /usr/share/tuleap/src/vendor/paragonie/easydb/src/Factory.php on line 99

    last email sent on 2019.08.25

    User avatar
    Anton KULIK (d00AK)2019-09-05 09:44
    Hi, nothing strange on nginx side. It returns the favicon, but nothing more from
    upstream tuleap-php-fpm {
    server 127.0.0.1:9000;
    }

    How to investigate PHP FPM? I'm not a PHP coder.
    User avatar
    Thomas Gerbet (tgerbet)2019-09-05 09:40
    It's strange that nginx returns you a 502 but you got nothing in the error.log. You probably want to investigate it. Maybe also try to take a look at the logs of PHP FPM.
    User avatar
    Anton KULIK (d00AK)2019-09-04 09:11
    Hi, just did Tuleap upgrade for this morning packages. Run post-upgrade scripts successfully. The accessed page and got Bad Gateway error again.
    Now most recent nginx logs:

    Neither error nor access nginx logs show anythins wrong (NOTE: Tuleap favicon is downloading without problems):

    1) tail -20 /var/log/nginx/error.log
    2019/09/04 08:56:35 [warn] 25480#25480: "ssl_stapling" ignored, issuer certificate not found for certificate "/home/congineer/ssl.cert"

    2) tail -20 /var/log/nginx/access.log

    ... Requests from other IPs
    31.10.137.61 - - [04/Sep/2019:08:54:57 +0200] "GET / HTTP/2.0" 502 552 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36 OPR/63.0.3368.66" "-"
    31.10.137.61 - - [04/Sep/2019:08:54:57 +0200] "GET /favicon.ico HTTP/2.0" 200 15086 "https://alm.congineer.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36 OPR/63.0.3368.66" "-"
    User avatar
    Anton KULIK (d00AK)2019-09-04 08:47
    Hi, what logs other than codendi should I check?
    Here is the codendi_syslog:

    tail 20 /var/log/tuleap/codendi_syslog
    tail: cannot open `20' for reading: No such file or directory
    ==> /var/log/tuleap/codendi_syslog <==
    2019-09-04T03:54:37+02:00 [300] [info] [system_check] Running git gc on gitolite admin working copy.
    2019-09-04T04:08:18+02:00 [4282] [info] [system_check] Running git gc on gitolite admin working copy.
    2019-09-04T04:35:48+02:00 [9410] [info] [system_check] Running git gc on gitolite admin working copy.
    2019-09-04T05:01:32+02:00 [14301] [info] [system_check] Running git gc on gitolite admin working copy.
    2019-09-04T05:54:42+02:00 [24169] [info] [system_check] Running git gc on gitolite admin working copy.
    2019-09-04T06:13:37+02:00 [27683] [info] [system_check] Running git gc on gitolite admin working copy.
    2019-09-04T06:33:19+02:00 [31342] [info] [system_check] Running git gc on gitolite admin working copy.
    2019-09-04T07:05:06+02:00 [5134] [info] [system_check] Running git gc on gitolite admin working copy.
    2019-09-04T07:57:09+02:00 [14177] [info] [system_check] Running git gc on gitolite admin working copy.
    2019-09-04T08:28:49+02:00 [19697] [info] [system_check] Running git gc on gitolite admin working copy.
    User avatar
    Thomas Gerbet (tgerbet)2019-09-04 08:40
    Hello,

    Are all the services actually running? Do you have some information in the logs?