ios

Seeing AI Photo Gallery

Microsoft Seeing AI

Seeing AI is a mobile app that narrates the world around you. It enables people with low vision to recognize faces, objects, text, bills, colors, and much more! Seeing AI was first announced at //build conference in 2016.

Over last few months I was working with Seeing AI team overnight on in-app Photo Gallery that allows you not to browse through photos that you have taken earlier. Before you could only take photo, analyze it, and save (without description). Now, you can save it with description that you can later retrieve. You can also analyze photos taken with your phone camera.

One of the challenges was to decide what details about the photo should we present. We have a lot of different channels (short text, document, person, scene, etc.). For now we decided to show scene description, place and date when photo was taken. Let us know if you have suggestions to improve this!

SeeingAI - photo gallery

Another problematic part was deciding how to distinguish between analyzed, and not analyzed photos. Initially we had a toggle to switch between recognized photos and all photos. Ultimately we decided to have only 1 view with all photos. Thoughts?

For non recognized photos we needed to provide mechanism to analyze them. Initially user had to open photo, and explicitly tap ‘analyze’. We changed this approach to automatically analyze not recognized photos when going to full screen view, and enable users to reanalyze them. The ‘reanalyze’ button might be useful in situations where we update app with new AI models. These models may do better job in recognizing photos.

Seeing AI is built with Xamarin iOS native. I was surprised that we had to build entire gallery from scratch. There was no plugin or open source sample we could use. Hint: opportunity for you!

You can download Seeing AI from App Store.


Adding biometrics authentication to Xamarin.iOS (Touch ID / Face ID) and Xamarin.Android (Fingerprint)

One of the top Azure App users requests was to add Touch ID support for additional security. In this post I will share the details of implementing biometrics authentication for iOS and Android with Xamarin.

There are three aspects of biometrics auth:
1. Enable user to turn biometrics authentication on and off. Users shouldn’t be forced to use this additional security feature.
2. Detecting when user should be asked for biometrics authentication, e.g., when app is coming from background, and when app is starting.
3. Authentication process. Includes detecting hardware capabilities (is touch or face id available?), and local setup (does user configured local authentication in system settings).

Enabling biometrics authentication usually can be controlled in settings (like in Outlook or OneDrive). We did the same in Azure App:

Require Touch ID Settings

iOS

Detecting when user is switching back to our app in iOS is pretty simple. Every time when user switch from background, method WillEnterForeground in AppDelegate is being called. We just need to override it with our custom implementation:

public override void WillEnterForeground(UIApplication application)
{
    // biometrics authentication logic here
}

You should also authenticate user when app is being launched. In that case authentication should be performed in your initial view controller.

In iOS we have 2 kinds of biometrics authentication:
1. Touch ID
2. Face ID (available from iPhoneX)

We can also fallback to passcode if touch/face ID is not configured, or user’s device does not support it.

The iOS Local Auth API is pretty straightforward, and well documented. I created simple helper to handle feature detection and authentication:

public static class LocalAuthHelper
{
    private enum LocalAuthType
    {
        None,
        Passcode,
        TouchId,
        FaceId
    }

    public static string GetLocalAuthLabelText()
    {
        var localAuthType = GetLocalAuthType();

        switch (localAuthType)
        {
            case LocalAuthType.Passcode:
                return Strings.RequirePasscode;
            case LocalAuthType.TouchId:
                return Strings.RequireTouchID;
            case LocalAuthType.FaceId:
                return Strings.RequireFaceID;
            default:
                return string.Empty;
        }
    }

    public static string GetLocalAuthIcon()
    {
        var localAuthType = GetLocalAuthType();

        switch (localAuthType)
        {
            case LocalAuthType.Passcode:
                return SvgLibrary.LockIcon;
            case LocalAuthType.TouchId:
                return SvgLibrary.TouchIdIcon;
            case LocalAuthType.FaceId:
                return SvgLibrary.FaceIdIcon;
            default:
                return string.Empty;
        }
    }

    public static string GetLocalAuthUnlockText()
    {
        var localAuthType = GetLocalAuthType();

        switch (localAuthType)
        {
            case LocalAuthType.Passcode:
                return Strings.UnlockWithPasscode;
            case LocalAuthType.TouchId:
                return Strings.UnlockWithTouchID;
            case LocalAuthType.FaceId:
                return Strings.UnlockWithFaceID;
            default:
                return string.Empty;
        }
    }

    public static bool IsLocalAuthAvailable => GetLocalAuthType() != LocalAuthType.None;

    public static void Authenticate(Action onSuccess, Action onFailure)
    {
        var context = new LAContext();
        NSError AuthError;

        if (context.CanEvaluatePolicy(LAPolicy.DeviceOwnerAuthenticationWithBiometrics, out AuthError)
            || context.CanEvaluatePolicy(LAPolicy.DeviceOwnerAuthentication, out AuthError))
        {
            var replyHandler = new LAContextReplyHandler((success, error) =>
            {
                if (success)
                {
                    onSuccess?.Invoke();
                }
                else
                {
                    onFailure?.Invoke();
                }
            });

            context.EvaluatePolicy(LAPolicy.DeviceOwnerAuthentication, Strings.PleaseAuthenticateToProceed, replyHandler);
        }
    }

    private static LocalAuthType GetLocalAuthType()
    {
        var localAuthContext = new LAContext();
        NSError AuthError;

        if (localAuthContext.CanEvaluatePolicy(LAPolicy.DeviceOwnerAuthentication, out AuthError))
        {
            if (localAuthContext.CanEvaluatePolicy(LAPolicy.DeviceOwnerAuthenticationWithBiometrics, out AuthError))
            {
                if (GetOsMajorVersion() >= 11 && localAuthContext.BiometryType == LABiometryType.TypeFaceId)
                {
                    return LocalAuthType.FaceId;
                }

                return LocalAuthType.TouchId;
            }

            return LocalAuthType.Passcode;
        }

        return LocalAuthType.None;
    }

    private static int GetOsMajorVersion()
    {
        return int.Parse(UIDevice.CurrentDevice.SystemVersion.Split('.')[0]);
    }
}

There are helper methods determining proper label (GetLocalAuthLabelText), icon (GetLocalAuthIcon) and authentication text (GetLocalAuthUnlockText) depending on available authentication type. There is also one liner IsLocalAuthAvailable checking if Local Authentication (face/touch ID or passcode) is available, and Authenticate method that performs authentication, which takes success and failure callbacks as parameters. It can be used in WillEnterForeground method as follows:

public override void WillEnterForeground(UIApplication application)
{
    if (!AppSettings.IsLocalAuthEnabled)
    {
        return;
    }

    LocalAuthHelper.Authenticate(null, // do not do anything on success
    () =>
    {
        // show View Controller that requires authentication
        InvokeOnMainThread(() =>
        {
            var localAuthViewController = new LocalAuthViewController();
            Window.RootViewController.ShowViewController(localAuthViewController, null);
        });
    });
}

We do not have to do anything on success. The popup shown by iOS will disappear and user will be able to use the app. On failed authentication though we should display some kind of shild (e.g., ViewController) that prevent user from using the app until authorization succeed. This is how it looks in Azure App:

Azure App - Unlock with Touch ID

Android

Detecting when app is coming from background in Android is tricky. There is no single method that is invoked only when app is coming back from background. The OnResume method is being called when app is coming back from the background, but it’s also called when you switch from one activity to another. Solution for that is to keep a time stamp with last successful authentication, and update it to DateTime.Now every time when activity is calling OnPause. This happen when app is going to background, but also when app is changing between activities. Thus we cannot simply set flag Background=true when OnPause is called. However, when difference between subsequent OnPause and OnResume is larger than some period of time (e.g., more than a few seconds) we can assume that app went to background. Below code should be implemented in some BaseActivity class that all activities inherit from:

public class BaseActivity
{
  public const int FingerprintAuthTimeoutSeconds = 5;
  public static DateTime LastSuccessfulFingerprintAuth = DateTime.MinValue;
    
  protected override void OnResume()
  {
    base.OnResume();

    if (IsFingerprintAvailable() && LastSuccessfulFingerprintAuth > DateTime.Now.AddSeconds(-FingerprintAuthTimeoutSeconds))
    {
      StartActivity(typeof(FingerprintAuthActivity));
    }
  }

  protected override void OnPause()
  {
    base.OnPause();

    if (IsFingerprintAvailable())
    {
      LastSuccessfulFingerprintAuth = DateTime.Now;
    }
  }
}

The basics of Fingerprint authentication are very well described in Xamarin docs.

Even better reference is a sample app FingerprintGuide from Xamarin.

The main disadvantage of adding fingerprint authentication in Android (over Face/Touch ID in iOS) is requirement to build your own UI and logic for the authentication popup. This includes adding icon, and handling all authentication results. iOS handles incorrect scan, and displays popup again with passcode fallback after too many unsuccessful tries. In Android you have to implement this entire logic by yourself.

Summary

Adding biometrics authentication is useful for apps that hold sensitive data, like banking apps, file managers (Dropbox, OneDrive), or an app that has access to your Azure Resources 🙂

Implementing local authentication in iOS is pretty straightforward, and iOS APIs provide authentication UI for free. In Android however, the APIs are only working with the backend, and UI has to be implemented by you.

Local authentication should be always optional. Some users may not need nor want it. Thus, it should be configurable in the app settings.

Try out biometrics auth in Azure App!

Download on the App Store
Get it on Google Play


Managing multiple accounts in Azure App

One of our top user’s feedback requests was to enable multiple account access without singing out and signing in.

It is now available on latest iOS and Android releases!

Quick overview

Azure App - Multiple Accounts

You can see all your accounts in hamburger menu. You can add new account by tapping ‘Add account’. To remove account you need to simply sign out.

Limitations

First limitation (AKA caveat): when you are adding second live account you, you may run into the following screen during authentication with Active Directory:

Azure App - already signed in

This is Active Directory limitation. Just tap ‘Sign out and sign in with a different account’, and we will sign you in to another account without signing you out from another.

Second limitation: you are not able to sign in into two accounts if they are associated with one email. Yes! One email can be used to sign in to more than 1 account. If you are seeing screen similar like below during authentication this is a case:

Azure App - Multi Accounts One Email

In this situation you need to sign out, and then sign in again choosing another account.

Summary

I personally love this feature as switching between my MSDN account and work account was a pain in the past. Now it’s seamless.

We are still exploring possibility to enable users to sign in into two accounts tied to the same email. We are also looking at improvements around removing and adding accounts.

Do you have feedback? Let us know! You can ping me or our team on twitter. You can also add or vote on existing ideas at our User Voice.

Happy Thanksgiving!


Trying iOS 11 with Xamarin

The triathlon season is over. I completed all three, planned races for this year:

  1. Ironman 70.3 Coeur d’Alene
  2. SeaFair Sprint Triathlon (new PR!)
  3. Lake Meridian Olympic Triathlon (new PR!)

I also finished RAMROD (epic Ride Around Mt Rainier in One Day) and Course d’Equipe. The last bike ride for this season is Gran Fondo Whistler in two weeks.

In the meantime…

The Winter is coming!

Apple is cooking for us iOS 11, and I decided to give it a shot! It actually works nice.

  1. Install latest Xcode beta from here
  2. Install latest Xamarin.iOS (all links are here, hint: version is 10.99, not 11 yet)
  3. Set VS for Mac to Xcode-beta (Preferences -> Projects -> SDK Locations -> Apple -> Location)

If you did everything correct you should be able to see new iOS11 simulator:

iOS 11 simulator

I encountered one issue: when deploying to device I got following errors:

Error: unable to find utility “lipo”, not a developer tool or in PATH
Error: Failed to create the a fat library

Solution was to run the following command:

sudo xcode-select --switch /Applications/Xcode-beta.app/Contents/Developer/

Related Xamarin Forums thread.

Summary

So far everything works pretty well. Occasionally when I run VS for Mac it doesn’t detect simulators, but after restart they are back!

Have you tried iOS 11 yet?


iOS for C# Developer – part 4: Xcode

This post is part of the series: iOS for C# Developer. Previous parts:

C# developers works with Visual Studio. Recently, some are trying to switch to SublimeText. However, I will assume that you, as C# Developer are familiar with VS. To write iOS applications, you need Xcode. I haven’t heard about nobody who created an working app in a different editor (more advanced than Hello World).

In this post I would like to provide a few tips that will help you to get started and be productive.

The first thing that is worth to mention is the fact that Xcode is free. You cannot test you apps on the iPhone or iPad before you pay $99 though. You need to be satisfied with simulator.

Key shortcuts

For the beginning, remember three:

  • ⇧ + ⌘ + O – quick open (equivalent to CTRL+, in VS2013 or CTRL+T in ReSharper)
  • ⌄ + click – jump to documentation (when clicked on class/interface)
  • ⌘ + L – jump to specified line of code

This three will definitely help you to get productive at the beginning. You can find more in this cheat-sheet.

Debugging

In order to debug application in VS, you have to run it in debug mode. In Xcode, when you run the app on emulator it is always in debug mode. Thus, you just need to set the break-point (by mouse click on the line number, like in VS, or by ⌘ + \ shortcut) and run the app (⌘ + R). Once break-point is reached you can step over (F6), step into (F7) or step out (F8).

Very useful during debugging is dumping objects into console with NSLog function:

NSLog(jsonString);

This outputs, e.g., JSON string to the console, which allows you to inspect it.

To get more flavor of debugging, check Apple documentation and Brian Moakley’s blog post.

Playgrounds

The latest version of Xcode (v6), which introduce support for Swift language, adds also very handy feature: Playgrounds. It allows you write and test simple Swift programs, without using iOS emulator. Playgrounds are useful especially, when you are testing some complex logic. They are very powerful. You can use them even to test drawing (still without running iOS emulator). To see it in action, check this demo.

Summary

Xcode is very nice IDE. Works very smooth, and breaks very rarely. I wish there was more code snippets (like it is in Visual Studio, especially with ReSharper or WebEssentials). Xcode could also adapt rich debugging experience from VS (e.g., inspecting object variable after moving mouse pointer in top of them). I find it hard to work with this IDE on small screen (like MacBook display). To get comfortable, I need at least 22-24″. But that is just my opinion. What do you think?