•  
      epic #10407 Cross tracker search
    Summary
    Cross tracker search
    Cross tracker search

    Overview

    Provide a way to aggregate data across several trackers and several projects.

    It will be done via a Widget to be installed on a Dashboard (personal or project). Unlike current "Tracker report" widget, it will be possible to directly interact with the widget to refine a query. For instance the widget displays by default all open artifacts, then directly in the widget, any user will be able to add "and category = test" to run it's own query.

    This widget will rely on:

    • Tracker Query Language for the definition of the search
    • The selection of the columns to display will be done with a selector (as it's already done in tracker table reports)
    • The order of the columns and their size will be manipulated in drag'n drop like tracker table report. However the underlying libs will be upgraded to fix current bugs. In order to be consistent, the changes will be done on tracker table report too.

    On the technical side it will be a dedicated Angular app backed by REST routes. All the work on this widget report will be root for future redesign of tracker reports.

    Always There Fields and semantics

    Some fields are specials, we have 2 categories:

    Always There Fields, they are "always there behind the scenes" fields but that might not be used in one tracker (or renamed):

    • Submitted by
    • Submitted on
    • Last updated by
    • Last updated on
    • Artifact ID

    Semantics:

    • Title
    • Status
    • Assigned to
    • Description

    The proposal is to treat them like if they were normal fields and propose a unified syntax to query / display them. That is to say:

    • Regardtheless the actual field pointed by "Status" semantic (progress, stage, etc), in cross tracker search, people will always query on "status". Eg: status = OPEN() and start_date >= NOW() - 1w
    • Regardtheless the name or the label associated to "Submitted by" field (Creator, Starter, etc), in cross tracker search, people will always query on "submitted_on". Eg. submitted_on = '2017-04-24';

    Corresponding identifiers:

    • Submitted by: submitted_by
    • Submitted on: submitted_on
    • Last updated by: last_update_by
    • Last updated on: last_update_on
    • Artifact ID: id
    • Title: title
    • Status: status
    • Assigned to: assigned_to
    • Description: description

    Limits and tracker consistency

    With the given definition, we can end with configuration nightmare like having a field called (name) summary with "title" semantic and, in the same tracker a 'title' field that refers to something completly different. In this scenario, we cannot know if a search on title is for the semantic or the lone title field.

    So, in order to make life easier, the Runtime Check will ensure, before running the query that:

    • All the fields used in query term or in columns are present and accessible for read by the user
      • If I want to search on "submitted_on", the field must be active in all trackers I selected and I should have the right to read it
    • There are no overlaping field names
      • If I want to search on "submitted_on", the only "submitted_on" field name accepted in involved trackers is the one the correspond to "Submitted on" field. If someone created a lone date field with name "submitted_on" in one of the tracker, the query will not be executed and the user will get an error to align its trackers structures.

    Definition of boundaries

    On the bright side, you can do anything with the trackers. The problem is that people actually do anything with them. And most of the time they aren't even aware about the configuration details and subtle differences (like different names or permissions). We need to define clear boundaries on what we can search on. Either people align their trackers or they won't be able to search across them.

    As it's a tough problem due to the highly configurable nature of Tuleap trackers and the fine grain permissions, some preconditions are needed:

    • Limit the number of trackers on which the query would be applied (for instance 10)
      • limit the amount of data we need to search on
      • allow to resonate on a finite state (for instance in term of number of fields)
      • allow to run consistency check ahead of the query execution
    • Duck typing: if it has the same name and the same type, it's the same field
      • Given a set of tracker, if 2 different trackers have a "name" field but one is a String and the other one is a Select box, it will not be possible to search on this field or to display it as a column
    • Permissions handling
      • Given an already defined query, when a random user want to look at the result, there is a check in first hand to ensure that the user has the right to search on all the fields of the query in all invoved tracker.
      • Same goes for the columns, if at least one field of one tracker cannot be displayed to the current user, nothing is displayed at all.
    • Sorting
      • It will not be possible to sort on multiple columns in the same time (no multisort)
    • Allowed fields
      • It's possible to search or display only on fields that are common in all trackers according to the Duck Typing definition
      • There are "fake fields" available as columns: "project" and "tracker".

    Ways of working

    A possible scenario of usage:

    Configuration:

    1. Select the trackers that will be searched on (regardtheless of the project)
    2. Define the query (TQL)
    3. Select the columns, their positions, their size and the sort (UI)

    Runtime:

    1. Ensure field consistency and permissions across trackers
      1. The current user is allowed to access all fields used in the search query and displayed in the report. Those fields belongs to a tracker the user can access. Those tracker belongs to a project the user can see.
      2. The used fields (search and display) are consistent across tracker (same names/types).
    2. Perform the query and render the result

    Refinement:

    1. Based on this first execution, the user can modify the query for doing it's own searches

    Spikes & mockups

    Spikes

    • We need to spike the most efficient way to build and execute the query. Either "Monstro query" approach one huge query that embed all the search terms or a union of smaller queries. Investigation on real data set is needed.
    • Which libs shall replace the bogus one used to build the tracker report columns (drag'n drop and resize)

    Mock-ups

    • How do we select the trackers on which search will apply (across projects)
    • How do we select columns, place them and rezise (+ alignment of Tracker report view)
    • How do we interact with widget during the "Refinement of query" phase

    Steps

    Given the large amount of work that is needed, the following steps are identified to progressively come close the a full featured widget:

    1. Create a new widget (for project and user dashboards) where it's possible to select the trackers on which we want to run queries. At this step, the only selection possible is the trackers, everything else is hardcoded:
      • the query will be 'status = "Open"'
      • the sort will be on artifact id
      • the displayed columns will be hardoded (like artifact links v2 reports)
      • only the first N elements will be displayed
    2. Add runtime check for permissions and consistency
    3. Allow to define the query with sematic and "always there" fields (artifact id, submitted on, etc)
    4. Add pagination to "load more" artifacts
    5. Allow to select columns and resize with semantic and "always there" fields
    6. Allow to select on which column the sort is done
    7. Introduce duck typing for columns
    8. Introduce duck typing for query
    Progress
    2024-01-04
    2024-04-24 (79 working days)
    Closed
    Details
    #10407
    Manuel Vacelet (vaceletm)
    2024-07-18 17:03
    2017-07-05 15:44
    Attachments
    References

    Follow-ups

    User avatar
    User avatar
    Marine Pieux (nynoe)2017-09-29 12:33
    • Attachments 1-widget-multi-trackers-search-dropdown-columns.png, 2-widget-multi-trackers-search-hover.png, 3-widget-multi-trackers-search-edit.png, 4-widget-multi-trackers-search-save-report.png added
    User avatar

    All,

    Some of the elements of discussions that comes here (like being able to filter on artifact links, query sharing, etc) requires to have a first working version to iterate on. We need to validate the global way of working before defining more precisely those requirements. Moreover, the biggest questions that need to be answered are:

    • performances: how it will behave with the huge artifact base you have
    • what users will try to search and how they will interact with the widget

    For both questions, the easiest way to gather data is to work with a subset of the features, test with users and iterate (yay Agile!) so I updated the description with the steps we foresee that should allow to tackle all the functionnal and technical questions we have progressively.

     

     


    • Description
      Something went wrong, the follow up content couldn't be loaded
      Only formatting have been changed, you should switch to markup to see the changes
    User avatar
    My 2 cents:
    "boundaries" part sounds reasonable.

    "Ways of working:"
    Following separate discussion, I'm fine with TQL in a first time, but it could be a blocking point for the adoption/deployment of this cross tracker search. A long term click-able approach must be found once feature is stable.

    If report (so queries) already exist in tracker we need to collect data from, then I suggest Tuleap propose to reuse existing query, in the place of writing all from scratch.

    When field disapears (or become unvisible because access right change) from the source tracker, a feedback box should help user to identify why its cross tracker does not work anymore. A kind of "query checker" must be put in place with clever feedback to the user at the very begining of the development phase to me.

    About table that display the results, the name of column should be proposed by the UI, but customizable. Sort should be also possible like it is today.
    Of course these "cross search" have to be integrated in new project and personal dashboards, with idealy a way to share them easily (xml export/import). Let's imagine it become a very useful feature (why not ?), the query sharing between users could help a lot.

    Cheers
    Denis
    User avatar
    @vaceletm : That is correct. We would want to be able filter the results based on one or more of the commonly available fields, which is how we could restrict on status to get only open fields.

    Having a column for planned for would be really helpful as well, but that is a separate enhancement request.
    User avatar
    Laurence Terrien (ltn)2017-07-11 15:18

    Hi Manuel

    For example :

    1. I want to list all the artefacts in trackers T1, T2, T3.. (from different projects) having a link type "delived_in" or "fixed_in" to the artefact XXX from a  Release Tracker.
      => find all users storys/bugs/.. delivered in a Release  (one level link)
    2. I want to list all the artefcts having a link type "delived_in" or "fixed_in" to the artefacts that have themselves a link type "embedded_in" to the artefact XXXX from a  Release tracker.
      => find aiil users stories/bugs/..deliveered in sofwtare relese components which are embeded in an application release

     

     

    User avatar

    Ok so if understand correctly @stephanbill the point here is really to express a query that is able to fetch data across projects (eg. "all currently open user stories in project A, B and C") and display them in a consolidated table view with a list of selected fields.

    I added in epic description the precision about the usable fields as per Stephan and Laurence définition.

     


    • Description
      Something went wrong, the follow up content couldn't be loaded
      Only formatting have been changed, you should switch to markup to see the changes
    User avatar
    Looking at this user's projects for the releases, the fields seems to be the same.
    The Release tracker report shows these fields:
    - Release Number
    - Progress (status: planned, delivered to customer, ...) (status semantic)
    - Start Date
    - Duration -in days-
    - Forecasted Capacity
    - Capacity (auto computed field)
    - Initial effort (auto computed field)
    - Remaining effort (auto computed field)
    - ID

    I would think that we should be able to add "project" as one of the fields to be displayed as well, so we know from where these artifacts come from.

    For the MR and Epic trackers, the tracker list view is not as consistent. A subproject either has the MR or Epic tracker. Here are the fields that are displayed:
    - Epic ID or MR ID
    - Epic or Title (title semantic)
    - Progress (Status semantic, the trackers don't have all the same values)
    - Description
    - effort
    - Initial effort (autocompute field)
    - Remaining effort (autocompute field)
    - Links
    - ID

    For the story tracker, the fields displayed by the trackers vary a little. The multi-select fields has different options from one tracker to another.
    User avatar

    @stephanbill 

    Use case 2: We have an umbrella project. I would like to see a common report and filtering about some trackers in subprojects. Eg. 
    - X in subprojects

    Would it be possible to have a concrete example of what users expect on this (maybe a small scanned sketch might help) ?

    @ltn

    Could it be possible to proposed only common fileds (common means = (the couple 'label,name) is the same for all the trackers) are found in all  the trackers. One rule, wharever the type of field, in the semantic or not.., is easier to explaine. It will be up tu the administrators of the trackers to modify the fields definitions, in order to have more in the proposed list.

    I thought it would be too restrictive from end users point of view but it's perfectly feasible. It might be related to Stephan's comment "need appropriate error handling, informative error messages on the appropriate level of config and running.". This probably means that we need a tool to compare N tracker structures to highlight differences, however it's a bit out of the scope of the original intend.

    @ltn

    Its seem to me important to b able to search on links and type of links.

    Same question as Stephan, could you give me an example of query you would like to execute ?

    User avatar
    Laurence Terrien (ltn)2017-07-06 16:50

    Hello,

    My first remarks to Enalean proposal.

    Could it be possible to proposed only common fileds (common means = (the couple 'label,name) is the same for all the trackers) are found in all  the trackers. One rule, wharever the type of field, in the semantic or not.., is easier to explaine. It will be up tu the administrators of the trackers to modify the fields definitions, in order to have more in the proposed list.

    Its seem to me important to b able to search on links and type of links.

    For result display,  is the display already existing in the trackers (link by nature), could be the basis, with additional colomns for search criteria

    Artifact ID - Project - Tracker - Summary - Status - Last Update - Date Submitted By - Assigned to

    Regards

    Laurence

    User avatar
    We have some very large projects where the work is spread across several Tuleap projects. They would like to get an overview of activities.

    I asked for use cases and comments from some users in Ericsson. Here is one comment:

    Use case 1: In one project we would like to see and filter Stories, defects and system tasks in one report

    Use case 2: We have an umbrella project. I would like to see a common report and filtering about some trackers in subprojects. Eg.
    - Releases in subprojects
    - Epics and main requirements in subprojects
    - Stories in subprojects
    - Sprints in subprojects
    Important fields are status/progress, initial effort, remaining effort, capacity, type and flag (story)

    Some comments from the user:
    1. think about how can we handle well the resources (less priority for multi-tracker queries?)
    2. think about an "alias" attribute for fields which can be the same as field name by default, but allows to have common field name for trackers without changing the original field name. It can give us more flexibility but on the other hand it can be more complex solution. But it is worth to think a little bit about it.
    3. need appropriate error handling, informative error messages on the appropriate level of config and running.
    User avatar
    • Description
      Something went wrong, the follow up content couldn't be loaded
      Only formatting have been changed, you should switch to markup to see the changes
    User avatar
    • Description
      Something went wrong, the follow up content couldn't be loaded
      Only formatting have been changed, you should switch to markup to see the changes
    User avatar
    • Description
      Something went wrong, the follow up content couldn't be loaded
      Only formatting have been changed, you should switch to markup to see the changes