Skip to main content

Public

Project privacy set to public. By default, its content is available to everyone (authenticated or not). Please note that more restrictive permissions might exist on some items.

    •  
      request #36483 CSV values mixed from one line to other
    Actions
    Infos
    #36483
    Guilhem Bonnefille (CS) (gbonnefille)
    2024-03-07 16:00
    2024-02-01 15:00
    38068
    Details
    CSV values mixed from one line to other

    When exporting a CSV report, some information are mixed from on line to the following. The beginning of a line contains the values related to the current artifact, but the end of the line concerns the previous or next artifact. But this isn't systematic, this seems related to some data only.

    We are still unable to reproduce on a simple test case and we cannot share our data.

    This behaviour seems to occur when the number of requested column exceed 20. But it is not only related to this: we tried to create a tracker from scratch and we do not observed the behaviour.

    In order to understand the behaviour, we noticed that the "20" columns seems related to the threshold in buildOrderedQuery to split the query into sub-queries. We also hacked the code in exportToCSV as following:

                        $subrow = $result->getRow();
                    if ($subrow['id'] != $row['id']) {
                        // TODO Warning
                    }
                    $row = array_merge($row, $subrow);

    With such code, we were able to understand that there are "permutations" :

    subrow[id]=403 row[id]=2
subrow[id]=2 row[id]=403
subrow[id]=404 row[id]=3
subrow[id]=3 row[id]=404
subrow[id]=405 row[id]=4
subrow[id]=4 row[id]=405
subrow[id]=406 row[id]=5
subrow[id]=5 row[id]=406
subrow[id]=501 row[id]=6
subrow[id]=6 row[id]=501
subrow[id]=623 row[id]=7
subrow[id]=7 row[id]=623
subrow[id]=915 row[id]=8
subrow[id]=8 row[id]=915
subrow[id]=935 row[id]=9
subrow[id]=9 row[id]=935
    Trackers
    15.3
    Empty
    • [ ] enhancement
    • [ ] internal improvement
    Empty
    Stage
    Empty
    Closed
    2024-03-07
    Attachments
    Artifact links

    Releases

    Fixed in

    Artifact ID Project Tracker Summary Status Last Update Date Submitted By Assigned to
    rel #35498 Tuleap Releases 15.6 Delivered 2024-04-03 15:49 Manuel Vacelet (vaceletm)
    Empty
    References
    Referencing request #36483

    Git commit

    tuleap/tuleap/stable

    fix: Ensure data are always returned in the same order for each chunked queries 37d50e920f
    User avatar Aurélien Tisné (atisne) , 2024-03-07 10:14
    Referenced by request #36483

    Artifact Tracker v5

    rel #35498 15.6

    Other

    gerrit #30386
    Display settings

    Follow-ups

    User avatar
    Thomas Gerbet (tgerbet)2024-03-07 16:00

    Merged gerrit #30386 (Tuleap 15.6.99.50), thanks for the investigation and dealing with the patch.

    • Status changed from New to Closed
    • Connected artifacts
    • Close date set to 2024-03-07
    User avatar
    Aurélien Tisné (atisne)2024-02-02 15:40

    FYI, we hacked the code of buildOrderedQuery to add an ORDER BY c.id on queries. This seems to avoid the issue.

    User avatar
    Guilhem Bonnefille (CS) (gbonnefille)2024-02-01 17:44

    Concerning the bug itself, the only idea I have is to enforce the contract expected around buildOrderedQuery by adding an ORDER BY c.id in order to ensure that any sub-request returns mergeable sub-results.

    User avatar
    Aurélien Tisné (atisne)2024-02-01 17:42

    I would like to know if you have ideas on what we can catch (adding traces) or try to go further in investigation.

    User avatar
    Guilhem Bonnefille (CS) (gbonnefille)2024-02-01 17:40

    We already have a client to request the REST API in order to create CSV from batch. But we still have customers using the Web UI CSV export. I think my patch reduce the memory pressure without any side-effect. It certainly worth an integration.

    User avatar
    Manuel Vacelet (vaceletm)2024-02-01 17:28

    Thanks for the patch @gbonnefille. FTR, the right way of fixing that would be to have the same approach we had for XLSX generation: do it client side based on REST API (lower the pressure on server, no risk of memory issue).

    Do you have any suggestion on how to track the issue?

    What do you mean @atsine ?