SwiftUI + Vision Contour Request — Coin Detection in iOS | by Anupam Chugh

WWDC20

Apple boosts its pc imaginative and prescient ambitions with a bunch of latest Imaginative and prescient requests

Picture by Rahul Bhosale on Unsplash

Apple’s WWDC 2020 (digital-only) occasion kickstarted with a bang. There have been lots of new surprises from the world of SwiftUI, ARKit, PencilKit, Create ML, and Core ML. However the one which stood out for me was pc imaginative and prescient.

Apple’s Imaginative and prescient framework obtained bolstered with a bunch of thrilling new APIs that carry out some complicated and demanding pc imaginative and prescient algorithms in a reasonably easy means.

Beginning with iOS 14, the Imaginative and prescient framework now helps Hand and Physique Pose Estimation, Optical Stream, Trajectory Detection, and Contour Detection.

Whereas we’ll present an in-depth have a look at every of those another time, proper now, let’s dive deeper into one significantly attention-grabbing addition—the contour detection Imaginative and prescient request.

  • Understanding Imaginative and prescient’s contour detection request.
  • Operating it in an iOS 14 SwiftUI software to detect contours alongside cash.
  • Simplifying the contours by leveraging Core Picture filters for pre-processing the pictures earlier than passing them on to the Imaginative and prescient request. We’ll look to masks the pictures in an effort to cut back texture noise.

Contour detection detects outlines of the perimeters in a picture. Basically, it joins all the continual factors which have the identical shade or depth.

This pc imaginative and prescient job is helpful for form evaluation, edge detection, and is useful in situations the place you should discover related varieties of objects in a picture.

Coin detection and segmentation is a reasonably frequent use case in OpenCV, and now by utilizing Imaginative and prescient’s new VNDetectContoursRequest, we are able to carry out the identical in our iOS functions simply (with out the necessity for third-party libraries).

To course of photographs or frames, the Imaginative and prescient framework requires a VNRequest, which is handed into a picture request handler or a sequence request handler. What we get in return is a VNObservation class.

You should use the respective VNObservation subclass primarily based on the kind of request you’re working. In our case, we’ll use VNContoursObservation, which gives all of the detected contours from the picture.

We are able to examine the next properties from the VNContoursObservation:

  • normalizedPath — It returns the trail of detected contours in normalized coordinates. We’d must convert it into the UIKit coordinates, as we’ll see shortly.
  • contourCount — The variety of detected contours returned by the Imaginative and prescient request.
  • topLevelContours — An array of VNContours that aren’t enclosed inside any contour.
  • contour(at:) — Utilizing this operate, we are able to entry a baby contour by passing its index or IndexPath.
  • confidence — The extent of confidence within the general VNContoursObservation.

Notice: Utilizing topLevelContours and accessing youngster contours is helpful when you should modify/take away them from the ultimate commentary.

Now that we’ve obtained an concept of Imaginative and prescient contour detection request, let’s discover the way it may work it in an iOS 14 software.

To start out off, you’ll want Xcode 12 beta because the naked minimal. That’s about it, as you possibly can straight run Imaginative and prescient picture requests in your SwiftUI Previews.

Create a brand new SwiftUI software within the Xcode wizard and see the brand new SwiftUI App lifecycle:

You’ll be greeted with the next code when you full the mission setup:

@major
struct iOS14VisionContourDetection: App
var physique: some Scene
WindowGroup
ContentView()


Notice: Beginning in iOS 14, SceneDelegate has been deprecated in favor of the SwiftUI App protocol, particularly for SwiftUI-based functions. The @major annotation on the highest of the struct signifies it’s the place to begin of the appliance.

With a view to carry out our Imaginative and prescient request, let’s rapidly arrange a SwiftUI view, as proven beneath:

Within the above code, we’ve used the if let syntax that’s launched with SwiftUI for iOS 14. Ignore the preprocessImage state; for now, let’s straight soar onto the detectVisionContours operate that’ll replace the outputImage state upon the completion of Imaginative and prescient request:

Within the above code, we’ve set the contrastAdjustment (to boost the picture) and detectDarkOnLight (for higher contour detection as our picture has gentle background) properties on the VNDetectContoursRequest.

Upon working the VNImageRequestHandler with the enter picture (current within the Property folder ), we get again the VNContoursObservation.

Ultimately, we’ll draw the normalizedPoints as an overlay on our enter picture.

The code for the drawContours operate is given beneath:

The UIImage returned by the above operate is ready to the contouredImage SwiftUI state, and subsequently our view will get up to date:

The outcomes are fairly respectable contemplating we ran this on a simulator, however they will surely be higher if we ran this on a tool with iOS 14, with entry to the Neural Engine.

However nonetheless, there are far too many contours (largely on account of coin textures) for our liking. We are able to simplify (or fairly cut back) them by pre-processing the picture.

Core Image is Apple’s picture processing and evaluation framework. Although it really works high-quality for easy face and barcode detection duties, it isn’t scalable for complicated pc imaginative and prescient use instances.

The framework truly boasts of over 200 picture filters and is helpful in pictures apps in addition to for information augmentation in your machine studying mannequin coaching.

However extra importantly, Core Picture is a helpful software for pre-processing photographs which are then fed to the Imaginative and prescient framework for evaluation.

Now, in the event you’ve watched the WWDC 2020 Computer Vision APIs video, you’ve seen that Apple has leveraged Core Picture’s monochrome filter for pre-processing, whereas demonstrating their punchcard contour detection instance.

In our case, for coin masking, the monochrome impact wouldn’t give nearly as good outcomes. Particularly for cash which have an analogous shade depth that’s totally different from the background, utilizing the black and white shade filter for masking cash is a greater wager.

For every of the above pre-processing sorts, we’ve additionally set a Gaussian filter to smoothen the picture. Pay attention to how the monochrome pre-processing filter truly provides us considerably extra contours.

Therefore, it’s essential to pay heed to the sorts of photographs you’re coping with when doing pre-processing.

The outputImage obtained after the pre-processing is fed to the Imaginative and prescient picture request. The block of code for creating and making use of Core Picture filters is out there on this GitHub Repository, together with the total supply code.

Through the use of the VNGeometryUtils class, we are able to observe properties like diameter, bounding circle, space perimeter, and facet ratio of the contour. Merely move the contour, as proven beneath:

VNGeometryUtils.boundingCircle(for: VNContour)

This will open up new pc imaginative and prescient prospects in figuring out the totally different sorts of shapes out there in a picture.

Moreover, by invoking the polygonApproximation(withEpsilon:) methodology on a VNContour, we are able to additional simplify our contours by filtering out little noisy elements round an edge.

Laptop imaginative and prescient performs an enormous function in Apple’s combined actuality future. The introduction of hand and physique Pose APIs, which had been part of the ARKit framework, will open up new sorts of alternatives for constructing clever pc imaginative and prescient functions.

There’s lots of thrilling stuff that got here out of WWDC 2020. I’m excited concerning the new sorts of prospects for machine studying on cell. Keep tuned for extra updates, and thanks for studying.

More Posts