Simplifying IT, Empowering Businesses

We’ve picked up some of the most exciting announcements from Apple’s WWDC 2018 event where Apple has introduced some cutting-edge capabilities of their frameworks and APIs for the developers to leverage the benefits from the extremely powerful chipsets embedded in latest iOS devices.

With the announcement of iOS 12, the next generation Operating System for iOS devices, Apple has confirmed improvements in performance and has also included some brand new frameworks in the list.

Apart from all the announcements, Apple has reportedly upgraded ARKit to the next level and introduced ARKit 2, which has reportedly higher speed and processing power as compared to the initial version. Apple has also come up with the official measurement app bundled with iOS 12.

While addressing the crowd assembled at the McEnery Convention Center, Apple’s senior vice president of Software Engineering, Craig Federighi told that

“ARKit opens up the world of AR to hundreds of millions of users. Making it the world’s largest AR platform. By far.”

Craig also added: “We’re on a relentless pace of advancement with AR, and that continues today with ARKit 2. ARKit 2 delivers advances with improved face tracking, realistic rendering, and support for 3D object detection and persistence – which enables launching into AR experiences associated with a particular object or physical space. Like starting a game based around a physical toy, or having a physical classroom as the foundation for a lesson that was authored in AR.”

Here are some of the key improvements in ARKit 2 :

Persistent AR Experience

This will allow the user to save and load ARKit world-mapping data to allow users to return to previous AR experiences in the same real-world environment.

You would now be able to leave virtual AR Objects in the living scene and after that arrived back to them later in time.

Communicating with AR turns out to be easier as you would now be able to begin a game on a surface and return to it later in a similar state.

It would now be able to distinguish 3D objects like moles, sculptures, and can likewise apply reflections of this real-world onto AR objects.

Shared AR Experience

This enables users to transmit ARKit world-mapping data between nearby devices with the MultipeerConnectivity framework to create a shared basis for AR experiences.

ARKit 2 enables multiple users to play a game or collaborate on projects together, such as home renovation, team-specific games, and others.

It also supports a “spectator mode” that enables friends and family to see AR gameplay from another iOS device without participating.

Measure

A new inbuilt app developed by Apple for the users to help users in the measurement of the surface without any physical instrument required. This will be available in the device itself after iOS 12 for the AR compatible iOS devices.

New File Format – usdz

USDZ is a new file format for persisting and transmitting AR objects in iOS 12. It is based on Pixar’s existing Universal Scene Description specification, packaged in a zero-compression, non-encrypted ZIP file.

Support for creating and editing USDZ files in Adobe’s Creative Cloud suite of programs was announced at WWDC, as supported in Autodesk and Sketchfab.

ARKit 2 + Vision

Let’s take a deep dive into this by using Vision with ARKit 2:

Here, We’ll create an app in which we’ll display a real-time capture view, in our case we’ll use SpriteKit’s SKScene overlay with ARSKView on app launch. This is how ARSKView will capture

video and that can be tracked by method session(_: did update:) from the delegate.

This will bring captured frames to the delegate method and then we can send these frames to Vision’s classification method to identify objects inside the frame.

 The request handler for VNImageRequestHandler will then call a method to classify objects using Core ML model and it will predict the suitable and most confident label for the given frame or object by the Vision request handler.

So, now we have our results stored in a variable we need to identify where it should be displayed. For that, we can use ARKit’s hitTest(_: types:) method to estimate position which then will add anchor node at that place of interaction in the view.

After that, view(_:didAdd: for:) methods will be called and there we can add resulting data for the view to be displayed.

This is how we can display the text on ARKit’s view for prediction of Core ML achieved by Vision Framework’s frames which were captured using SpriteKit’s scene.

We can also detect and track human faces using ARKit and Vision in real-time to morph or for masking purposes. We can also use ARKit to develop real-time SnapChat like stickers app and in other areas.