•  
     
    story #10710 search on fields with duck typing
Summary
Empty
search on fields with duck typing
Empty

Overview

Allow to search on fields (outside semantics & "Always There Fields") without too much overhead in the query definition or column selection.

Apply "duck typing" principle: if it has the same name and has a compatible type, let's assume it's the same thing.

Compatible types definition:

  • All numerics (initial_effort could be fload or int): float
  • All list with same bind (eg a select box with static values and check box with static values) including open lists
  • String and text fields
  • Date
  • Date time (to be clear: date and date time are not compatible types)

Are excluded of search:

  • rank: doesn't make sense to search on this value that is purely internal
  • computed fields: as computed fields are computed on display we cannot search on them.
  • permission on artifact: cannot be searched on in TQL as of today.

Searches

In the following example we uses the the term exist for a field being in a tracker. The complete definition is:

  • The field is technically present in the tracker
  • The user who runs the query can read the content of the field

Base rules

When a field exists in ALL trackers of the query
And there is at least one of the field that is not compatible
Then there is an error

When a field exists in SOME trackers of the query
and there is least one of the field that is not compatible
Then there is an error

When a field exists in SOME trackers of the query
and all the fields are compatible
Then the query is performed on the trackers that share the field

Base scenario

Given the query category="foo" on tracker T1, T2
And category exists in T1,T2
And have a compatible definition
Then the query applies on both tracker

Given the query category="foo" on tracker T1, T2
And category exists in T1,T2
And doesn't have a compatible definition
Then the query gives an error: field category is present in T1 and T2 but their types cannot be compared (T1.category = text/md, T2.category = text/html)

Given the query category="foo" on tracker T1, T2
And category only exists in T1
Then the query only applies on T1

Exist rule precision: It means that if category exists in T1 and T2 but the current user can only see it in T1, then the query only applies on T1.

Filter as much as possible

Given the query category="foo" on tracker T1, T2, T3
And no tracker has category as field
Then the query lead to an error: no tracker is matching the query

Given the query category="foo" on tracker T1, T2, T3
And only T1 have category field
the query will apply on T1 only

Given the query category="foo" on tracker T1, T2, T3
And T1 & T2 have category field with compatible definitions (see Overview)
the query will apply on those 2 trackers

Two terms

Given the query category = "foo" and priority = "bar" on tracker T1, T2, T3
And category exists in T1, T2 but not in T3
And priority exists in T3 but not in T1,T2
Then the query gives no tracker is matching the query

Given the query category = "foo" and priority = "bar" on tracker T1, T2, T3
And category exists in T1, T2,T3
And priority exists in T3 but not in T1,T2
Then the query applies only on T3

List fields

List values may not exist in all trackers included in the query. In the same spirit as "duck typing" for fields, we apply the same method for list values:

Given the query list_field = "foo" on trackers T1, T2, T3
And list_field exists in T1, T2 and T3
And the list value "foo" exists in T1, T2 but not in T3
Then the query does not raise an error, and applies only on T1 and T2

Given the query list_field = "bar" on trackers T1, T2, T3
And list_field exists in T1, T2 and T3
And the list value "bar" does not exist in any of the three trackers above
Then the query raises an error that the list value "bar" does not exist

This is the case for list fields bound to static values, to users or to user groups.

Impacts on "single-tracker" TQL expert query

Previously, comparing an integer field with a float value (for example int_field > 3.3) was forbidden and raised an error. This constraint will be lifted. Since integer and float fields are treated as a single "numeric" category in cross-tracker search, we will lift this constraint to ease this work. Both float fields and integer fields will now allow comparisons to float values (3.3)

Empty
Empty
Status
Cross tracker search
Done
Development
  • [ ] Does it involves User Interface? 
  • [ ] Are there any mockups?
  • [ ] Are permissions checked?
  • [ ] Does it need Javascript development?
  • [ ] Does it need a forge upgrade bucket?
  • [ ] Does it need to execute things in system events?
  • [ ] Does it impact project creation (templates)?
  • [ ] Is it exploratory?
Empty
Details
#10710
Manuel Vacelet (vaceletm)
2024-03-18 14:31
2017-09-29 15:15
30814

References
Referencing story #10710

Git commit

tuleap/tuleap/stable

feat: Add duck-typed fields feature flag 22fbe82c11
feat: Add check and display error for numeric fields c246402de0
InvalidSearchableCollectorParameters must be a builder and not a mock 3313937177
refactor: Use a dedicated FieldType business object 4542a0fc78
add missing tests for InvalidSearchableCollectorVisitor e65700308a
InvalidSearchablesCollection must be final 98ea2e6dfe
feat: Introduce duck-typed field 9c58d0905d
chore: Promoted property for QueryBuilderVisitor a20eab08eb
feat: Query for Equal comparison of fields 7527a548e2
feat: Query for field equal empty 90aa7e6a92
feat: Computed table aliases c3e665967a
issue: comparison to nothing can fail 26e7262016
refactor: RetrieveUsedFields takes tracker id into account 413d7eb45e
Add integration tests on `=` comparison query 1faf123443
Merge integration tests together 69c313fa98
[chore] DuckTypedField should retrieve itself its field type e4bcfc1e75
DB integration tests are flaky 483154563f
DB integration tests are flaky (v2) 1ab86a9b89
READ permissions must be checked on field dd7932e258
feat: Query for Not equal comparison for numeric fields 561c204fc0
feat: Query for Lesser than comparison for numeric fields ac2577b74a
refactor: Flatten comparison matching 21335d7faa
feat: Query for Greater than comparison for numeric fields 6164cc13a5
refactor: Group together Numeric SQL building a6c4cd2db6
feat: Query for Lesser Than Or Equal comparison for numeric fields 7ae179d65c
refactor: move DB setup at the top df9157afcc
feat: Query for Between comparison for numeric fields 85e313dc6a
feat: Add comparison checks 09aaf29ec6
comparisons between int and float should be valid in regular tql f5087e28cf
chore: Tracker integration test files should match PSR-4 a891870a45
feat: Add string/text field for XTS TQL 2e1f7bddc1
refactor: Add explicit types to DateTimeValueRounder 28f13b12fc
feat: XTS support = on date fields db350da848
feat: XTS support != on date fields 24bb493148
feat: XTS support < and <= on date fields 835e026485
feat: XTS support > and >= on date fields 6535d9b2f8
feat: XTS support BETWEEN on date fields 370969014b
feat: XTS support = on datetime fields 2653750926
feat: XTS support != on datetime fields 7eb1088b57
feat: XTS support < and <= operators on datetime fields 6a5ab20d9e
chore: Unit test invalid comparisons and values for float 1a1efbf97a
feat: XTS support > and >= on datetime fields 4d84975c8c
feat: XTS support BETWEEN() on datetime fields 84d5218878
refactor: Group Float field invalid checks together bf154ce583
chore: Rework Invalid fields check tests f16345cefe
refactor: Use Flat field checker for cross-tracker too 1727c03d88
refactor: Group Int field invalid checks together 7bfcfb1cbc
feat: XTS support = on static list fields 7bf2208c2c
refactor: Group String/Text invalid checks together 017cc6cd94
feat: XTS support != on static list fields 9efc6cac14
feat: XTS support IN() on static list fields ff3551b7d8
feat: XTS support NOT IN() on static list fields 0d3d91e1d2
refactor: Replace left join + is not null by inner join in XTS ef51f9fade
chore: Unit test invalid comparisons and values for dates b5be3e5e64
refactor: Group date invalid checks together a7bf0d8852
test: Add tests on field permissions for XTS 23a008021b
refactor: Group File invalid checks together 402289151e
feat: XTS support = on ugroup list fields d17cea32d6
feat: XTS support != on ugroup list fields e5c9dac1d1
refactor: Bind builder should not try to build its field by itself 6bac1b5697
feat: XTS support IN() on ugroup list fields bf6a41b8f8
feat: XTS support NOT IN() on ugroup list fields 86e146618e
feat: XTS support = on user list fields c0247fd041
feat: XTS support != on user list fields bacd62a4ce
refactor: Group List invalid checks together 7988c3d4ff
feat: XTS support IN() on user list fields 6b7907aa1e
feat: XTS support NOT IN() on user list fields 8ff92aee12
refactor: Group Subby + luby invalid checks together 4de9fc9147
feat: Reject comparison of submitter / last updater to empty string b94fc483b9
feat: XTS support myself() for user list fields eb67915cd7
chore: Add "permissions" DB tests on Numeric 2a29de969e
refactor: Drop InvalidFieldChecker interface c3dd3d8195
feat: Allow openlist for XTS d381a09c69
fix: DuckTypedFieldChecker don't throw all exceptions d53868cad7
refactor: Use yield from to compose data providers e91ac9f604
feat: XTS support open lists for all binds 42a527503e
feat: Drop XTS duck typing feature flag 6db1186a15
feat: Change label of suggestions and info text 4af5de819d

Follow-ups

User avatar
Joris MASSON (jmasson)2024-03-18 14:02

Explain the behaviour of list values when absent from some trackers of the query


  • Acceptance criteria
    Something went wrong, the follow up content couldn't be loaded
    Only formatting have been changed, you should switch to markup to see the changes
User avatar

Date & date time are not compatible.

A few examples:

Given ArtifactA.close_date(date) := 2024-02-14
When I search close_date = 2024-02-14 16:22
Then I cannot find the artifact, make sense

Given ArtifactA.close_date(date) := 2024-02-14
When I search close_date = 2024-02-14 00:00
Then I can find the artifact => it only works because technically it's stored as a timestamp but there is no way that functionnally it makes more sense than the first.

Given ArtifactA.close_date(date) := 2024-02-14
When I search close_date > 2024-02-14 16:22
Then I cannot find the artifact, that sounds sensible

Given ArtifactA.close_date(date) := 2024-02-14
When I search close_date < 2024-02-14 16:22
Then I cannot find the artifact, that sounds sensible

Given ArtifactA.close_date(date) := 2024-02-14
When I search close_date >= 2024-02-14 16:22 Or search close_date <= 2024-02-14 16:22 Then I cannot find the artifact => there is actually no way to find the artifact when I express the search term with date time.

However Given ArtifactA.close_date(date) := 2024-02-14
Given ArtifactB.close_date(date) := 2024-02-15
When I search close_date > 2024-02-14 16:22 or close_date >= 2024-02-14 16:22 Then I will find ArtifactB => that makes the feature quite unreliable


  • Acceptance criteria
    Something went wrong, the follow up content couldn't be loaded
    Only formatting have been changed, you should switch to markup to see the changes
User avatar

XTS on artifact ids is not about duck-typed searches. It got a dedicated story #36812


  • Acceptance criteria
    Something went wrong, the follow up content couldn't be loaded
    Only formatting have been changed, you should switch to markup to see the changes
  • Is related to
User avatar

Artifact IDs should be searchable too.


  • Acceptance criteria
    Something went wrong, the follow up content couldn't be loaded
    Only formatting have been changed, you should switch to markup to see the changes
User avatar
Joris MASSON (jmasson)2024-02-08 10:40
  • Acceptance criteria
    Something went wrong, the follow up content couldn't be loaded
    Only formatting have been changed, you should switch to markup to see the changes
User avatar
Joris MASSON (jmasson)2024-01-31 16:15

Artifact links can already be searched with cross-tracker search since epic #32276


  • Acceptance criteria
    Something went wrong, the follow up content couldn't be loaded
    Only formatting have been changed, you should switch to markup to see the changes
User avatar
  • Acceptance criteria
    Something went wrong, the follow up content couldn't be loaded
    Only formatting have been changed, you should switch to markup to see the changes
User avatar
  • Acceptance criteria
    Something went wrong, the follow up content couldn't be loaded
    Only formatting have been changed, you should switch to markup to see the changes
User avatar
  • Acceptance criteria
    Something went wrong, the follow up content couldn't be loaded
    Only formatting have been changed, you should switch to markup to see the changes
  • Status changed from Ready (stalled) to Selected
User avatar
  • Acceptance criteria
    Something went wrong, the follow up content couldn't be loaded
    Only formatting have been changed, you should switch to markup to see the changes
  • Category set to Cross tracker search
User avatar
  • Acceptance criteria
    Something went wrong, the follow up content couldn't be loaded
    Only formatting have been changed, you should switch to markup to see the changes
  • Permissions set to