Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add pattern matching #982

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from

Conversation

AlexandreKempf
Copy link

Description

👋 Hello here

In many real-world scenarios, training a detection model, deploying it, and getting predictions is only one half of the job. Processing of the predictions is the other half of data scientists' work, and Supervision is the only tool to help do that in object detection.
However, thinking about my previous jobs and what I needed, I realized that there was a part of post-processing that was not covered by Supervision: looking for patterns. We can think of it as a regex for object detection: Given a set of rules, give me the set of boxes that respect these rules.

For instance in this image with predictions:
g15

That we obtained with:

    import cv2
    import supervision as sv
    from ultralytics import YOLO

    image = cv2.imread("men_on_bicycle.jpg")
    model = YOLO('yolov8s.pt')

    classes = [person, bicycle, ...]

    result = model(image)[0]
    detections = sv.Detections.from_ultralytics(result)

We could add a pattern to find an individual with their bike. The rule to identify this pattern (the regex) would be:

  • there should be a person
  • there should be a bike
  • the person center and the bike should be aligned vertically
    pattern = sv.MatchPattern(
       [
           (lambda id: id == classes.index("bicycle") , ["Bike.class_id"]),
           (lambda id: id == classes.index("person") , ["Rider.class_id"]),
           (lambda bike_center, rider_center: abs(bike_center[0] -  rider_center[0]) <= 50),
               ["Bike.CENTER", "Rider.CENTER"]
           ),
       ]
   )
   matches = pattern.match(detections)

matches is a list of detections (List[sv.Detections]) that match the pattern. Once displayed, the 4 first matches from the list look like this:

g30

For each match, an additional data attribute named "match_name" is added to the sv.Detection to specify to which detection corresponds each object of the pattern ("Bike" and "Rider" in our case).

Additional notes

1 - I added a little sugar on top of the feature. It is possible to define a value instead of a function. In that case, it will ensure that the arguments match exactly the value you provide.

    pattern = sv.MatchPattern(
       [
           (1 , ["Bike.class_id"]),
           (0 , ["Rider.class_id"]),
           (lambda bike_center, rider_center: abs(bike_center[0] -  rider_center[0]) <= 50),
               ["Bike.CENTER", "Rider.CENTER"]
           ),
       ]
   )
   matches = pattern.match(detections)

2 - The arguments you can give to the function directly come from sv.Detections. The complete list is:

  • xyxy, mask, class_id, confidence, or tracker_id
  • one of the Position enum strings
  • a field from the data attribute of your detections for more custom use cases

3 - It we want to continue this feature, it would be nice to provide a small bank of function that the user can use out of the box for bounding boxes (iou, included_in, etc.) and segmentation (next_to, area_are_touching, etc.)

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

How has this change been tested, please provide a testcase or example of how you tested the change?

I added functional tests.

Docs

  • Docs updated? What were the changes? I wanted to update the document, but I couldn't find a way to host images in the long run. Your documentation images are hosted on https://media.roboflow.com, but I can't access them. How are you usually doing it for people who don't have access to internal resources?

@onuralpszr onuralpszr mentioned this pull request Mar 13, 2024
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant