Tuleap 16 est là ! Assistez à l'événement virtuel le 17 octobre à 10h30. Inscrivez-vous ici !

    •  
      epic #24164 Search in trackers
    Summary
    Search in trackers
    Empty

    Overview

    A long time a go, there was a full text search plugin for Tuleap. It was eventually removed from the code base because it became un-manageable for two main reasons:

    • the initial architecture tradeoff were not the good ones for permissions (it's not possible to actually store permissions for documents in the engine).
    • the hard dependency on ElasticSearch at a stage where it evolved really fast was not manageable.
    • ElasticSearch itself proved to be a tricky beast and unqualified teams for OnPrem deployments made it a nightmare to manage.

    The current strategy is quite different:

    • permissions should not be stored with indexation but applied afterward by Tuleap
      • The tradeoff here is that it might be slower to get the results and it will not be possible to give the number of matched elements (on this point, keep in mind the google doesn't give it either)
    • as a first step, the target will be to mainly index tracker's texts (string & texteara fields as well as follow-up comments)
    • the indexation engine should be designed for replacement with MySQL itself as primary target (then it might be replaced/enhanced with external engine like ElasticSearch or Algolia for instance).
      • for instance we certainly don't want to manage ourselves ElasticSearch instances for Tuleap Cloud.

    There are now 2 backends:

    Search on artifacts

    From a functional stand-point, search on artifacts would be accessible via the "Switch to" button.

    A mock-up of the interaction and how the existing behavior of "filter" is handled is accessible in codepen: https://cdpn.io/pen/debug/OJxpRMY/0266ad34436604c43b0b0ecfaf768406

    REST route

    Search is done via a REST route.

    As no permissions are associated with the indexed data, the REST route must filter the data to ensure that permissions are applied. There are 2 situations:

    • fields: need to take into account permissions on it self field (sic!), artifact (permission on artifact), tracker (esp. with permissions relative to artifact creator & assignee), project (visibility) and platform (anonymous, restricted, ...).
    • follow-up comments: same as field but without the field level

    Indexing

    By default, indexing is done in a dedicated MySQL table designed for search using FullTextSearch index.

    Index is done in an asynchronous way (need of redis for message queue) after events:

    • Artifact Create
    • Artifact Update
    • Artifact Delete
    • Tracker Delete
    • Field Delete
    • Project Delete
    • Follow-up Update

    Table format (proposal): Search (type: string, content: TEXT, metadata: JSON) with metadata like {art_id: INT, field_id: INT} for fields or {art_id: INT} for follow-up.

    Create initial index

    A dedicated tuleap command is added to manage the initial index of the platform. Maybe we should design this command to take slice of artifacts to allow progressive creation of the index on very large platform. That said, on platforms with 1M+ artifacts, it's unlikely that an admin will be behind their desk to watch and launch such a command so maybe we need to design something with daily root or weekly to automatically index the platform over the course of a week for instance.

    Get better results

    This section covers the thoughts about search experience and what is desirable in the future. It focuses on meilisearch backend because trying to improve db backend doesn't make much sense.

    Filtering out automated test exec

    Possible strategies:

    • do not index test_execution artifacts that have automated_tests not empty
    • do not index test_execution at all (but we would lack of test results)

    Take project and tracker shortname into account

    Given I have a tracker of type bug in a project called garden And issue name dependency 2022 When I search for garden bug 2022 Then I should be able to find this bug

    Another way to approach this would be to introduce special filters like in:project_name, in:bug, 'status:closed'

    Progress
    Empty
    Empty
    Closed
    Details
    #24164
    Manuel Vacelet (vaceletm)
    2024-09-26 10:50
    2021-11-22 14:01
    Attachments
    References

    Follow-ups

    User avatar
    • Description
      Something went wrong, the follow up content couldn't be loaded
      Only formatting have been changed, you should switch to markup to see the changes
    • Status changed from Planned to Closed
    User avatar
    • Description
      Something went wrong, the follow up content couldn't be loaded
      Only formatting have been changed, you should switch to markup to see the changes
    User avatar
    • Description
      Something went wrong, the follow up content couldn't be loaded
      Only formatting have been changed, you should switch to markup to see the changes
    User avatar
    • Description
      Something went wrong, the follow up content couldn't be loaded
      Only formatting have been changed, you should switch to markup to see the changes
    • Category set to
    User avatar
    • Description
      Something went wrong, the follow up content couldn't be loaded
      Only formatting have been changed, you should switch to markup to see the changes
    User avatar
    • Description
      Something went wrong, the follow up content couldn't be loaded
      Only formatting have been changed, you should switch to markup to see the changes
    User avatar
    • Description
      Something went wrong, the follow up content couldn't be loaded
      Only formatting have been changed, you should switch to markup to see the changes
    User avatar
    • Description
      Something went wrong, the follow up content couldn't be loaded
      Only formatting have been changed, you should switch to markup to see the changes
    User avatar
    • Description
      Something went wrong, the follow up content couldn't be loaded
      Only formatting have been changed, you should switch to markup to see the changes
    User avatar
    • Description
      Something went wrong, the follow up content couldn't be loaded
      Only formatting have been changed, you should switch to markup to see the changes
    • Permissions set to