•  
     
    story #12404 download folder as zip
Summary
Empty
download folder as zip
Empty

Functional overview

  • Only files & embedded documents are downloaded. Other doc types are ignored.
  • Only the last version of the document is downloaded (the document history is not part of the archive).
  • On download action, if the amount of files to d/l (uncompressed) is bigger than a given threshold (50MB?) there is a warning + confirm
  • As content is streamed and zipped on the fly, the end users cannot have a progress bar to inform them the time it will take.
  • Download works on root folder too
  • Site administrators (site admin / Document) can set a max size:
    • as as warning threshold to warn that people are about to d/l something large. Default is set to 50MB
    • as a blocker to avoid download of big archives. Default is set to 2GB
  • Embedded are downloaded without templating (body & raw content).
  • When some files can't be added to the archive for some reason (file corrupted, not
    found, not readable, memory overflow), a file named TULEAP_ERRORS.txt will be created at the root of the archive. This file will list all the impacted files and their paths. Errors will be properly logged in codendi_syslog to ease the debugging.

Technical overview

  • Zip is generated on the fly and streamed from the backend (with PHP) to avoid creation of fat archives on the backend.
  • The lib to be used will be ZipStream
  • Zip will only be used as a container but content won't be actually compressed inside in order to be as light as possible on the server CPU
    • This will also allow to compute the size of the data to download (as it's the sum of the recorded filesize)
  • There are some limitations to take into account
    • Archive meant to be used on OSX cannot be bigger than 4GB or 64k files (known issue). If either condition is detected before download, a warning is raised to users (if such a big archive is allowed).

Implementation warnings

  • Action menu on folders is not unique. It depends on context (creation & co). This needs to be reviewed carefully.
  • Fetch of total file size is made in a dedicated REST route (/!\ permissions)
Empty
Denis PILAT (denis_pilat)
Status
Empty
Done
Development
  • [ ] Does it involves User Interface? 
  • [ ] Are there any mockups?
  • [ ] Are permissions checked?
  • [ ] Does it need Javascript development?
  • [ ] Does it need a forge upgrade bucket?
  • [ ] Does it need to execute things in system events?
  • [ ] Does it impact project creation (templates)?
  • [ ] Is it exploratory?
Empty
Details
#12404
Manuel Vacelet (vaceletm)
2020-06-01 10:43
2018-10-17 11:47
15906

References
Referencing story #12404

Git commit

tuleap/tuleap/stable

Define threshold in site admin 1c253677f4
New fa-tlp-zip-download icon 937089ee3d
Download the folder as a ZIP archive 11c38e8065
Log ZipStream errors 58097eefa2
Corrupted file break the process c497fcb26c
Add a text file that lists problematic files in the archive 4dcc04c412
Get the size of a folder 609ff527d0
Improve translations in site admin b88d70e963
Show error modal when the archive exceeds size threshold eb22ca66d0
Show warning modal when the archive exceeds warning threshold 374fb95695
Simplify unit tests for modal loading 34e60b2d59
Show confirm modal 48fc875a9d
Add a cypress test to verify zip download 4a6c0a1c53
Error during the retrieval of the folder's size shows a modal 11ecfc61b2
Warn OSX user when archive weights 4GB or contains 64k+ files 7e4e6facae
Use fa-circle-o-notch as spinner c7b2f22c9c
Fix threshold field id so the label is clickable 811105df6f
Remove download confirmation modal c17ec797fe
Send a mail notification when Files in error 7c4bce1684
Reject download of the zip when its size exceeds the maximum limit f7ccc25138
Allow download of big folders to finish its execution b88404deea

Follow-ups

User avatar
Thomas Gorka (tgorka)2020-05-19 10:33
  • Acceptance criteria
    Something went wrong, the follow up content couldn't be loaded
    Only formatting have been changed, you should switch to markup to see the changes
User avatar
Joris MASSON (jmasson)2020-05-15 11:12
Warning threshold has been lowered to 50MB

  • Acceptance criteria
    Something went wrong, the follow up content couldn't be loaded
    Only formatting have been changed, you should switch to markup to see the changes
User avatar
  • Acceptance criteria
    Something went wrong, the follow up content couldn't be loaded
    Only formatting have been changed, you should switch to markup to see the changes
User avatar
  • Acceptance criteria
    Something went wrong, the follow up content couldn't be loaded
    Only formatting have been changed, you should switch to markup to see the changes
User avatar
  • Acceptance criteria
    Something went wrong, the follow up content couldn't be loaded
    Only formatting have been changed, you should switch to markup to see the changes
  • CC list set to Denis PILAT (denis_pilat)