Building Cloud Search as a Service with AI

AI Search

It’s been almost a year since I joined Azure Search team. A lot has changed since then. I joined right after team doubled by merge with Text Analytics team with a mission to add intelligence to search. A few months later entire Cognitive Services (Azure Machine Learning APIs) platform team joined us. Then we hired additional developers to build scalable platform for both Cognitive Services and Azure Search. After that we also got a team of data scientists who are building the actual machine learning models. Now, as the Applied AI team, we are in the center of AI and Cloud at Microsoft.

Azure Search is a search-as-a-service cloud solution that gives developers APIs and tools for adding a rich search experience in web and mobile applications. You get for free things like autocomplete, suggestions, synonyms, results highlighting, facets, filters, sorting and paging. Functionality is exposed through REST API or .NET SDK. The biggest pain, which is infrastructure and availability are managed by us.

While having all of that, we also need a great developer experience. Everybody needs to be able to understand how to build that Search AI pipeline without spending hours on reading docs. This is another thing we are working on. Email me or tweet message me if you are interested in that kind of stuff.

Where are we going?

Cognitive Search

We want to build the best Search as a Service platform that enables developers to add Bing-like Google-like search experience to their websites. No need for hiring search experts who know what inverted index is. No challenges with shard allocation and how to implement master election properly. No need for distributed systems expertise to scale this for large amount of data. Last, but not least: no need for setting up, owning and managing the infrastructure. Everything is being taken care of by the platform. By the Cloud.

Our team is also working on market-leading Machine Learning APIs. We are going to utilize these ML models and enable you to search through not only text, but also through your images, audio and videos.

There is a lot of challenges in that journey. From processing large amounts of data, through doing it in reasonable time (performance/parallelization), to providing efficient user experience throughout the process.

Where are we now?

We already have fast, reliable and production-ready system for full-text search. You can provision it in no-time, scale by adding more replicas or partitions, and monitor using metrics we provide. You can query it with .NET SDK or using REST API. We even have Open Source UI generation tool that gets you started with the latter: AzSearch.js.

To learn more about current capabilities of Azure Search check this awesome presentation by Bryan Soltis:

There are two ways to populate your search index: by simply inserting documents (records) into it, or by using indexer – a mechanism that enables you to sync your search index with your data source (SQL or NoSQL Database, blob storage, etc.).

We have already started adding AI to our search pipeline, by enabling you to run text analytics and OCR on your data. If you are using indexer, you can create a skillset, which can detect people, entities, organizations, locations, key phrases, and language on the textual data. On top of that you can use OCR that can recognize text from your images, and enable you to search through that text. You can also run mentioned text analytics on recognized text. We call this approach Cognitive Search. Here is a quick video by Brian and Corom from our team, with a sneak peak of what’s possible:

Last year we created a prototype of Cognitive Search, using JFK files that went public. You can check out our JFK files website, github repo and below video from Connect(); conference in 2017, where Corom explaines how he built a pipeline to achieve what is possible now with just checking the checkbox:

We announced Cognitive Search at the //build conference earlier this year. Together with NBA we built a website that allows you to search through player’s photos. You can search for players, their shoes or correlations between them:

Similar approach can be used for variety of different scenarios. From filtering your family photos, through analyzing medical records data, to deciding which crypto-currency to buy. Now, all these PDFs and doc documents you have on your hard drive can be used to make an informed business decision.

There are a lot of companies using Azure Search in production. It’s super exciting for me that Real Madrid is using Azure Search. It’s my favorite football club since I was a kid.

How’s the team?

My favorite thing about our team are the people. Every single person is bringing something else to the table, and there is something you can learn from each one of them. From distributed systems expertise, through API design, to building efficient monitoring infrastructure that enables to maintain production cloud service. One of our team members is Henrik Frystyk Nielsen who is best known for his pioneering work on the World Wide Web and subsequent work on computer network protocols. Currently he works on encapsulating Machine Learning models into containers. Our manager, Pablo Castro started not only Azure Search, but also OData protocol and LINQ to Entities. Our Project Manager Lance Olson was one of the founders of the .NET! You can check out what people say about our team on blind! Search for “Azure Search” 😉 There is also a blog post written by Pablo a few years ago: Startup at Microsoft. A lot has changed since then. We went through a few rounds of “funding”, and our team grew. However, we still believe in core values expressed there. For example: every engineer from the team still talks to customers on daily basis either through social media or directly over email or Skype.

BTW: We are hiring!


Seeing AI Photo Gallery

Microsoft Seeing AI

Seeing AI is a mobile app that narrates the world around you. It enables people with low vision to recognize faces, objects, text, bills, colors, and much more! Seeing AI was first announced at //build conference in 2016.

Over last few months I was working with Seeing AI team overnight on in-app Photo Gallery that allows you not to browse through photos that you have taken earlier. Before you could only take photo, analyze it, and save (without description). Now, you can save it with description that you can later retrieve. You can also analyze photos taken with your phone camera.

One of the challenges was to decide what details about the photo should we present. We have a lot of different channels (short text, document, person, scene, etc.). For now we decided to show scene description, place and date when photo was taken. Let us know if you have suggestions to improve this!

SeeingAI - photo gallery

Another problematic part was deciding how to distinguish between analyzed, and not analyzed photos. Initially we had a toggle to switch between recognized photos and all photos. Ultimately we decided to have only 1 view with all photos. Thoughts?

For non recognized photos we needed to provide mechanism to analyze them. Initially user had to open photo, and explicitly tap ‘analyze’. We changed this approach to automatically analyze not recognized photos when going to full screen view, and enable users to reanalyze them. The ‘reanalyze’ button might be useful in situations where we update app with new AI models. These models may do better job in recognizing photos.

Seeing AI is built with Xamarin iOS native. I was surprised that we had to build entire gallery from scratch. There was no plugin or open source sample we could use. Hint: opportunity for you!

You can download Seeing AI from App Store.


QCon conferences – real experts experience exchange

QCon Beijing - speakers

Earlier this year I attended QCon Beijing and QCon Sao Paulo conferences. I really like QCon, because there is no marketing, just real experts exchange of experience. After my first QCon (Shanghai in 2016) I was very excited to come back this year!

QCon Beijing

Cognitive Search

I delivered talk about Building web apps with Cloud and AI. I showed how to build intelligent web apps with Azure Search and Cognitive Services (Azure Machine Learning APIs). We call this approach Cognitive Search. In my demo I showcased how you can determine which crypto currencies to buy using sentiment analysis on tweets. I streamed tweets to Event Hub, which triggers Azure Function that calls Text Analytics API to calculate sentiment of tweet. I store tweets and its sentiments in SQL Database. I also created Azure Search index to be able to effectively search through tweets. This index is being syncronized with SQL DB through integrated change tracking, and Azure Search indexer that runs on schedule.

Cognitive Search architecture

I built UI using AzSearch.js (UI generation tool for Azure Search indexes), ASP.NET Core and TypeScript (BTW: there is a lot of cool stuff in TypeScript these days!).

Crypto Search UI

In addition to search interface I also created aggregation chart comparing sentiments between different cryptos:

crypto charts

Source code is on github: crypto-search.

Video from my talk:

My talk was very well received. Attendees were assessing talks with green (great talk), yellow (ok talk) and red (bad talk) cards. I got 117 green, 9 yellows and 0 reds.

Conference

I had a great opportunity to meet a lot of engineers and architects from leading tech companies from around the World.

Mads Torgersen (the architect of C# language) shared future plans for C#. It was surprising for me how few people attended his talk. Sisie Xia and Chris Coleman from LinkedIn delivered a great talk about facing challenge of growth, and how they tackle it using their in-house tool Redliner. Julius Volz shared insights about Prometheus monitoring system.

Majority of talks were in Chinese. I went to Peng Xing‘s talk about how they develop Progressive Web Apps at Baidu. I was able to gather 60% of content, but got the essence by talking to him directly. I spent most of my time talking to people in the hallways. It was very eye opening to meet engineers from Chinese cloud giants. China cloud market is dominated by Alibaba Cloud (AliCloud), Baidu and Tencent. Azure and AWS have small market share. Google does not exists in China at all. Alibaba (largest online retailer in China) has even ambitions to overtake AWS in near future. It is worth to notice that Alibaba is Chinese equivalent of Amazon. At the same time people consider Baidu being Chinese Google, and Tencent (who owns WeChat) to be like Facebook. I had an opportunity to chat with Lu from Alibaba Cloud and Wang Yao (Head of IaaS at Baidu). After talking to them, and other engineers I would describe both companies’ stack in 3 Words: MacBook, Java and Go. This might be ignorant generalization, but almost every other engineer from big 3 (AliCloud, Baidu, Tencent) that I talked to was either writing code in Java or golang (using Mac of course). I also learned about Alibaba’s Search as a Service: OpenSearch. Something to keep eye on when they will be expanding to USA and Europe market.

The most popular track was of course Blockchain. Room was overflowed for the entire day:

QCon Beijing - blockchain track

China Tech

Every time when I visit China I am impressed by their progress. Highways superior to US interstates, fast trains between all major cities, and now – mobile payments adoption everywhere. Today, in China, most people use WeChat or AliPay. Sometimes cashier can get mad at you if you want to pay cash or credit card, because you cause inconvenience. Scanning QR code is 10x faster! You can even tip a waiter with your mobile phone!

Tipping in China

Last year in Seattle we had three bike-sharing companies, and everybody here thinks that we are at the edge of innovation. By the end of 2017 Beijing had 60 bike-sharing companies. Many of them started in 2016. During my visit I learned that China have engineers dominated government. Maybe this explains their progress?

If you are going to China, it is useful to have these 3 apps:
1. WeChat – Chinese facebook, for communication with other Chinese people, and exchanging contacts
2. DiDi – Chinese Uber
3. AliPay – for mobile payments (currently WeChat payments requires Chinese ID and account in Chinese Bank)

It was also great to meet China division of Cognitive Services team!

Cognitive Services - China team

Trip to China inspired me to read a book comparing Chinese and Western culture in the World of innovation and progress:

From the Great Wall to Wall Street

There is a lot of things that these two Worlds can learn from each other.

QCon Sao Paulo

Cognitive Search

Two days before my talk in Brazil, we officially announced Cognitive Search built in into Azure Search. You do not have to create Cognitive Service anymore. You do not have to write code that orchestrate processing data, and calling API. We do it for you. All what you have to do is to check the checkbox. More details here.

Cognitive Search

I extended my demo of crypto analysis based on tweets by adding sentiment analysis on news articles. A while ago we put news on Azure Search index. We filtered out news related to cryptos, and put it on Azure Blob storage. We also improved our JFK files demo, and now you can deploy it by yourself by following instructions in this github repo.

Usually during my talk I ask if somebody ever deployed ElasticSearch and how long did it take. In China one guy said it was ~2 weeks. In Brazil there was one guy who said: 6 months(!) 🙂 That’s why you don’t want this to be your problem. Azure Search takes care of deployment, availability and upgrades for you.

My talk was pretty well received in Brazil as well. I got 148 green, 15 yellow and 0 red cards.

Conference

QCon Sao Paulo had very diverse mix of experts from all around the World. Starting with Aaron Stannard (co-founder of AKKA.NET), Nicholas Matsakis (from Rust core team), and Rodrigo Kumpera (architect and top contributor of mono project), through Michelle Casbon (now Engineer in Google Cloud Platform focused on machine learning and big data tools), Ben Lesh (RxJS Lead at Google), and Martin Spier (performance engineer at Netflix) to Soups Ranjan (Director of Data Science at Coinbase*), Amanda Casari (Data Scientist at SAP Concur), and Piper Niehaus (Elm lang passionate).

The most interesting thing I learned at QCon Sao Paulo was how different companies struggle with monitoring, telemetry and system malfunction detection. They all have very sophisticated automation, but it is still not enough in today’s World complexity. As systems we build have more, and more complex architecture, we need to build even better monitoring software to maintain them.

In Azure Search we are using OData standard. However, recently GraphQL is gaining popularity. Dan McGhan shared this article about comparing GraphQL and OData: REST API Industry Debate: OData vs GraphQL vs ORDS. Interesting read!

Brazil

May is almost winter in Brazil, but it’s also the best time to visit. Not too hot, not too cold. Perfect weather for enjoying your time!

I like that every bar and restaurant in Brazil have TV on soccer channel 🙂 As a long standing fan of Brazil national football team (since Ronaldo Luiz Nazario de Lima times) I enjoyed it a lot!

Sao Paulo, the largest city in Brazil (21 million people) and financial center of the country, is also the largest tech-hub in south America. Most of tech-companies are there (Microsoft, Google, Amazon).

If you ever go to Brazil, remember to visit Sugarloaf to watch the sunset and after the sunset view 🙂

View from Sugarloaf after night

Summary

Speaking at QCons and connecting with engineers from different backgrounds is very valuable experience. Being able to learn about other cultures is a plus as well. Sharing with other work that you do everyday can also give you different perspective, and notice things that you would never think about.

If you want to learn more about Azure Search check out our getting started docs. To create intelligent search pipelines check out my Cognitive Search blog post. For more details we have quickstart and more comprehensive API overview.

Questions? Find me on twitter!

*Soups did not tell me what is the next coin coming to CoinBase 🙁


Wroc# – developer conference worth attending

Wroc# - audience

Earlier this year I had a pleasure to speak at Wroc# conference in Poland. It was very well organized event that was almost free for attendees. The only cost was required donation to charity: PLN 150 (~$50).

There was only 1 track with awesome speakers lineup! I finally had an opportunity to meet Roy Osherove in person. I learned about unit testing from his book The Art of Unit testing. I’m currently reading his another book: Elastic Leadership – a lot of useful tips not only for team leaders! Among other speakers there were Scott Helme (Security Researcher) that uncovered some things about web security I have never heard about! Zan Kavtaskin gave great overview about building apps on Azure Cloud, Glenn Henriksen showed how on server-less computing works in Real-World, and Sander Hoogendoorn together with Kim van Wilgen shared their perspective on over-engineering development processes.

The conference venue was great. I would say it was the best setup I’ve even seen! There was one big room, divided into 3 parts: stage, chairs for audience and Mix&Mingle zone (AKA M&M). You could talk (in the Mix&Mingle zone), and still be able to follow presentations. Speakers’ room was on the upper floor, but it was more like a balcony, from where you could listen to talks and overlook entire venue.

I delivered talk about building mobile apps with Xamarin. I shared what we have learned while building Azure Mobile App, which started as hackathon project, and later turned into official Microsoft product. The app got announced on the stage of //build conference last year. Along the way we learned how to properly architect Xamarin project for multiple platforms, where to do not take shortcuts, does and don’ts for CI/CD and testing.

There was a guy who put a nice summary of my talk:

Wroc# - building mobile apps with Xamarin

At the end of the conference there was speaker’s panel where we were answering and discussing questions from the audience. We had good discussion about different aspects of software development from estimating project cost to writing unit tests. Almost every speaker had different background, and this made it even more interesting!

Wroc# - speakers panel

If you haven’t been to Poland before: Wroclaw is an amazing city, and many of my foreign friends says it’s their favorite city in Poland. Wroclaw is often refereed as WrocLove 😉

WrocLove

Last, but not least: thank you for everyone who made this conference happen!

Wroc# - organizers


Cognitive Search – Azure Search with AI

Cognitive Search

Today, at Microsoft //build conference we announced Cognitive Search. You may wonder what is Cognitive Search. To put it as simple as possible: it’s Azure Search powered by Cognitive Services (Azure Machine Learning APIs). You remember when you wanted to run some intelligence over your data with Cognitive Services? You had to handle creating, e.g., Text Analytics API, then writing code that would take your data from database, issue request to API (remember to use proper key!), serialize, deserialize data and put result in your database?

Now, with Cognitive Search, you can achieve that by checking one checkbox. You just need to pick a field on which you want to run analytics, and which cognitive services or skills (1 cognitive service usually contain multiple skills) to run. As for now we support 6 skills:

  1. Key phrases
  2. People
  3. Places
  4. Organizations
  5. Language
  6. OCR (Optical Character Recognition)

We output results directly to your search index.

Creating Intelligent Search Index

To take advantage of Cognitive Search you need to create Azure Search service in South-Central US or in West Europe. More regions coming soon!

To create search index powered by cognitive services you need to use ‘import data’ flow. Go to your Azure Search Service and click on ‘Import data’ command:

Cognitive Search - step 1

Then pick your data source (MSSQL, CosmosDB, blob storage etc.). I will choose sample data source that contains real estate data:

Cognitive Search - import data

Now, you need to pick a field on which you want to run analytics. I will choose description. You also need to choose which cognitive services (skills) you want to run, and provide output field names (fields to which we will output cognitive services analysis result):

Cognitive Search - skillset definition

In the next step you need to configure your index. Usually you want to make fields retrievable, searchable, and filterable. You may also consider making them facetable if you want to aggregate results. This is my sample configuration:

Cognitive search - define index

In the last step you just need to configure indexer – a tool that synchronizes your data source with your search index. In my case I will choose to do synchronization only once, as my sample data source will never change.

Cognitive Search - create indexer

After indexer finish you can browse your data, and cognitive services results in search explorer.

Cognitive Search - browse

You can also generate more usable search UI for your data with AzSearch.js.

Generating UI to search data with AzSearch.js

If you don’t like browsing your data with search explorer in Azure Portal that returns raw JSON, you can use AzSearch.js to quickly generate UI over your data.

The easiest way to get started is to use AzSearch.js generator. Before you start, enable CORS on your index:

Cognitive search - CORS

Once you get your query key and index definition JSON paste it into generator together with your search service name, and click ‘Generate’. An html page with simple search interface will be created.

Cognitive Search - AzSearch.js

This site is super easy to customize. Providing html template for results change JSON into nicely formatted search results:

Cognitive search - AzSearch.js pretty

All what I did was to create HTML template:

    const resultTemplate =
        `<div class="col-xs-12 col-sm-5 col-md-3 result_img">
            <img class="img-responsive result_img" src={{thumbnail}} alt="image not found" />
        </div>
        <div class="col-xs-12 col-sm-7 col-md-9">
            <h4>{{displayText}}</h4>
            <div class="resultDescription">
                {{{summary}}}
            </div>
            <div>
                sqft: <b>{{sqft}}</b>
            </div>
            <div>
                beds: <b>{{beds}}</b>
            </div>
            <div>
                baths: <b>{{baths}}</b>
            </div>
            <div>
                key phrases: <b>{{keyPhrases}}</b>
            </div>
        </div>`;

And add it to already present addResults function call:

automagic.addResults("results", { count: true }, resultTemplate);

I also created resultsProcessor to do some custom transformations. I.e., join few fields into one, truncate description to 200 characters, and convert key phrases from array into string separated by commas:

var resultsProcessor = function(results) {
        return results.map(function(result){
            result.displayText = result.number + " " + result.street+ " " +result.city+ ", " +result.region+ " " +result.countryCode;
            var summary = result.description;
            result.summary = summary.length &lt; 200 ? summary : summary.substring(0, 200) + "...";
            result.keyPhrases = result.keyphrases.join(", ");
            return result;
        });
    };
    automagic.store.setResultsProcessor(resultsProcessor);

You can do similar customization with suggestions. You can also add highlights to your results and much more. Everything is described in AzSearch.js README. We also have starter app written with TypeScript and React based on sample real estate data, which takes advantage of more advanced features of AzSearch.js. If you have any questions or suggestions regarding AzSearch.js let me know on Twitter!

Summary

Cognitive Search takes analyzing data with Azure Search to the next level. It takes away the burden of writing your own infrastructure for running AI-based analysis. For more advanced analysis, including OCR on your images, check out our docs. I am super excited to see it in action, and for the next improvements that we are working on. Let us know what do you think!

*This blog post was written in Boeing 787 during my flight from Toronto to São Paulo, when I was on my way to QCon conference.