•  
      request #14409 Memory limit exhausted while uploading git-lfs
    Infos
    #14409
    Guilhem Bonnefille (CS) (gbonnefille)
    2020-02-21 14:06
    2020-01-15 17:51
    15589
    Details
    Memory limit exhausted while uploading git-lfs
    While uploading files via git-lfs, the git command failed:
    - with generic error: "Failed to push some refs
    - without error in gitolite logs
    - with the following error in nginx

    2020/01/13 17:17:45 [error] 3677#0: *399559 FastCGI sent in stderr: "PHP message: PHP Fatal error: Allowed memory size of 205520896 bytes exhausted (tried to allocate 199237632 bytes) in /usr/share/tuleap/plugins/gitlfs/vendor/league/flysystem/src/Adapter/Local.php on line 159" while reading response header from upstream, client: 172.27.40.104, server: odin.si.c-s.fr, request: "PUT /uploads/git-lfs/objects/021be08a0874d4accf1a2e5b0be36f10ff10654549674b74e0228a5a36f0475e HTTP/1.1", upstream: "fastcgi://127.0.0.1:9002", host: "tuleap.example.fr"
    2020/01/13 17:17:45 [error] 3677#0: *399559 FastCGI sent in stderr: "PHP message: PHP Warning: Error while sending STMT_CLOSE packet. PID=3634 in Unknown on line 0" while reading upstream, client: 172.27.40.104, server: odin.si.c-s.fr, request: "PUT /uploads/git-lfs/objects/021be08a0874d4accf1a2e5b0be36f10ff10654549674b74e0228a5a36f0475e HTTP/1.1", upstream: "fastcgi://127.0.0.1:9002", host: "tuleap.example.fr"

    The concerned file is 211 MB.

    The concerned project is configured without any limit on file syze, in the Git.

    The Git-LFS limit is set to 512 MB on the platform.

    The PHP memory-limit is set to 256 MB.

    (Effective platform's version is 11.7)
    SCM/Git
    All
    Empty
    • [ ] enhancement
    • [ ] internal improvement
    Empty
    Stage
    Thomas Gerbet (tgerbet)
    Closed
    2020-02-21
    Attachments
    Empty
    References

    Follow-ups

    User avatar
    Integrated into Tuleap 11.11.99.129

    • Status changed from Under review to Closed
    • Connected artifacts
    • Close date set to 2020-02-21
    User avatar
    Thomas Gerbet (tgerbet)2020-02-21 08:53
    The issue is fixed when using the latest PHP packages from the remi repository.

    The minimal version of PHP required by Tuleap will be raised to 7.3.15: gerrit #17867.

    • Status changed from Verified to Under review
    User avatar
    Thomas Gerbet (tgerbet)2020-02-20 18:22
    PHP 7.3.15 has been released with the fix. I will confirm tomorrow but with my last build from a few days ago of the 7.3 branch, the issue was resolved.
    User avatar
    Thomas Gerbet (tgerbet)2020-02-06 14:39
    The issue is caused by a bug in PHP present since PHP 7.3.10/7.4.0 (or to be more precise since https://github.com/php/php-src/commit/05560b67bc87a2bcbfd5b48a48443a62f3311e7d). The bug has been fixed (https://bugs.php.net/bug.php?id=78902) but is not yet present in one of the released version.

    ATM I have no idea on how work around it while keeping the feature usable. While I do not recommend it, it is possible to increase the memory limit for those requests (see /etc/opt/remi/php73/php-fpm.d/tuleap-long-running-request.conf) so the file you are trying to upload can fit in memory.


    I will keep the request open until the fix is available in one of the PHP version supported by Tuleap.

    • Reported in version changed from 11.8 to All
    User avatar
    Thomas Gerbet (tgerbet)2020-02-06 10:51
    @gbonnefille Would you be able to provide us the version of PHP you are currently running (can be obtained with /opt/remi/php73/root/usr/bin/php -v)?
    User avatar
    Thomas Gerbet (tgerbet)2020-02-05 14:34
    Hum in fact this one is a bit strange. I have been able to reproduce on two distinct production instances but it seems to work just fine on a my dev instance...
    User avatar
    Thomas Gerbet (tgerbet)2020-02-05 13:44
    I have been able to reproduce. There is indeed an issue here.

    • Status changed from Acknowledged to Verified
    • Platform cleared values: CentOS 6
    User avatar
    Thomas Gerbet (tgerbet)2020-01-15 18:40
    Hi,

    Thanks for the report, I will try to reproduce.

    • Status changed from New to Acknowledged
    • Assigned to changed from None to Thomas Gerbet (tgerbet)