request #36483 CSV values mixed from one line to other
    Guilhem Bonnefille (CS) (gbonnefille)
    2024-03-07 16:00
    2024-02-01 15:00
    CSV values mixed from one line to other

    When exporting a CSV report, some information are mixed from on line to the following. The beginning of a line contains the values related to the current artifact, but the end of the line concerns the previous or next artifact. But this isn't systematic, this seems related to some data only.

    We are still unable to reproduce on a simple test case and we cannot share our data.

    This behaviour seems to occur when the number of requested column exceed 20. But it is not only related to this: we tried to create a tracker from scratch and we do not observed the behaviour.

    In order to understand the behaviour, we noticed that the "20" columns seems related to the threshold in buildOrderedQuery to split the query into sub-queries. We also hacked the code in exportToCSV as following:

                        $subrow = $result->getRow();
                        if ($subrow['id'] != $row['id']) {
                            // TODO Warning
                        $row = array_merge($row, $subrow);

    With such code, we were able to understand that there are "permutations" :

    subrow[id]=403 row[id]=2
    subrow[id]=2 row[id]=403
    subrow[id]=404 row[id]=3
    subrow[id]=3 row[id]=404
    subrow[id]=405 row[id]=4
    subrow[id]=4 row[id]=405
    subrow[id]=406 row[id]=5
    subrow[id]=5 row[id]=406
    subrow[id]=501 row[id]=6
    subrow[id]=6 row[id]=501
    subrow[id]=623 row[id]=7
    subrow[id]=7 row[id]=623
    subrow[id]=915 row[id]=8
    subrow[id]=8 row[id]=915
    subrow[id]=935 row[id]=9
    subrow[id]=9 row[id]=935
    • [ ] enhancement
    • [ ] internal improvement
    Referencing request #36483
    Referenced by request #36483

    Artifact Tracker v5

    rel #35498 15.6


    User avatar
    Thomas Gerbet (tgerbet)2024-03-07 16:00

    Merged gerrit #30386 (Tuleap, thanks for the investigation and dealing with the patch.

    • Status changed from New to Closed
    • Connected artifacts
    • Close date set to 2024-03-07
    User avatar

    FYI, we hacked the code of buildOrderedQuery to add an ORDER BY c.id on queries. This seems to avoid the issue.

    User avatar

    Concerning the bug itself, the only idea I have is to enforce the contract expected around buildOrderedQuery by adding an ORDER BY c.id in order to ensure that any sub-request returns mergeable sub-results.

    User avatar

    I would like to know if you have ideas on what we can catch (adding traces) or try to go further in investigation.

    User avatar

    We already have a client to request the REST API in order to create CSV from batch. But we still have customers using the Web UI CSV export. I think my patch reduce the memory pressure without any side-effect. It certainly worth an integration.

    User avatar

    Thanks for the patch @gbonnefille. FTR, the right way of fixing that would be to have the same approach we had for XLSX generation: do it client side based on REST API (lower the pressure on server, no risk of memory issue).

    Do you have any suggestion on how to track the issue?

    What do you mean @atsine ?