programming

Continuous Integration and Continuous Delivery for Xamarin.iOS with VSTS (Vistul Studio Team Services)

Visual Studio Team Services has great Continuous Integration and Continues Delivery support for Xamarin.

Recently I was configuring pipeline that would build the project, run unit tests (with xUnit), run UI tests (with Xamarin Test Cloud), and, if all tests pass, deploy new version of the app to Hockey App. During the process of creating VSTS build definition I encountered a few problems that I think are worth to share with you.

To make a basic build definition with building Xamarin.iOS project and deploying to Hockey App with VSTS+Xamarin check James Montemagno’s Continuous Integration for iOS Apps with Visual Studio Team Services first.

The issues I had, that were not described anywhere online:

  • I couldn’t use “Visual Studio Test” task for running xUnit tests, because every build definition on VSTS can be executed by only one build agent, and to build Xamarin project I needed Mac build agent.
  • How to run UI tests (with Xamarin Test Cloud) using the same build that would be later on deployed to Hockey App? Test Cloud requires init code that later on shouldn’t be present in deployed app.

Running xUnit tests with Mac build agent

This requires adding 2 tasks to build definition:

  1. Command Line task – to run tests with mono and xUnit console runner.
  2. Publish Test Results task – to display results in build summary

First, install xUnit console runner NuGet package.

To run tests, add command line task with following settings:

  • Tool: mono
  • Arguments: packages/xunit.runner.console.2.1.0/tools/xunit.console.exe YourApp.UnitTests/bin/Debug/YourApp.UnitTests.dll -xml UnitTestsResults.xml

Remember to replace YourApp with your app name, and set the same xUnit runner version that you have installed.

To publish test results, add “Publish Test Results” task with the following settings:

  • Test Result Format: XUnit
  • Test Result Files: **/UnitTestsResults.xml
  • Always run: true

Running Xamarin UI tests with Test Cloud before deploying to HockeyApp

The solution there is rather simple: build in Debug mode that has TEST_CLOUD preprocessor directive defined, run tests, and then build again in Release mode before deploying to Hockey App.

Delete files before build

For a while I couldn’t figure out why my tests are not passing when run from VSTS while they pass on my machine. It turned out, VSTS was using old builds for tests. I am not sure about MacInCloud, but if you are using your own Build Agent running on your Mac the directories that contains binary files are not being cleaned up before next build automatically. To avoid using old builds, you can add, at the beginning of your pipeline, “Delete Files” task and delete everything from bin/* and obj/*.

Summary

The final build definition looks like that:

Azure Status VSTS build definition

Let me know if I missed some details! I would be happy to add them! Also – if you know better way of doing that – let me know as well!


QCon Shanghai

QCon

Last month I had a pleasure to visit China and attend QCon conference in Shanghai.

I gave a talk about building large scale web apps with TypeScript where I updated my talk from last year to TypeScript 2.0, I showed TypeScript boilerplate project created by Remo Jansen, and using TypeScript with Aurelia Framework.

QCon

//video-coming-soon

QCon

I had a great turnout. Room overflow with over 150 people. I also got a great feedback. People could vote green(awesome), yellow(ok), red(bad), and I received 128 green, only 10 yellow and 1 red, which gives over 92% extremely satisfied attendees.

QCon

QCon has the best of the best from the industry. In Shanghai I met Jaime Levy (famous UX strategist from LA), Kinshuk Mishra (Director of Engineering at Spotify), Pete Soderling (successful entrepreneur from New York and Silicon Valley, founder of Hakka Labs & DataEngConf), Sid Anand (former Cloud Data Architect at Netflix who implemented Netflix migration to the cloud that Adrian Cockcroft later on told the World about, former Tech Lead at LinkedIn where he implemented typeahead search, currently Data Architect at Agari – company that provides fraud detection services), Wesley Reisz (Solutions Architect at HP and University Professor), and Igor Marvić (Software Engineer at Spotify).

China is very interesting place for developers, especially because of its scale. They have over 700 million Internet users! One developer from alibaba.com – largest online store in China – told me that during Singles’ Day (Guanggun Jie), which is equivalent of Black Friday in USA, they had over 400 million customers online at the same time. This is more than entire population of the United States (~325 million). They also sold more than $1 billion in merchandise in first 8 minutes! This is SCALE!

While I was in China I was using WeChat (their Facebook equivalent). People don’t use Facebook and twitter, as they are blocked by China Firewall. You can workaround it using VPN, but sometimes it works very slow, or not at all. Thus, it is much easier to just use WeChat. When I am in USA I use Facebook, messenger and twitter everyday. However, I noticed that on my iPhone 5S, WeChat works much better than all mentioned apps. It doesn’t crash, it’s fast, and it has many cool features. Want to add somebody to friends? Just scan they QRCode. You can also shake your phone, and WeChat will try to find closest person who is also shaking their phone, and show on your phone their profile, and on their phone your profile.

Besides the conference, which was very well organized, I was very impressed by how modern the city of Shanghai is. The skyline looks almost like New York City! Shanghai Tower (2,073 ft / 632 m) is the second tallest building in the WorldShanghai World Financial Center (1,614 ft / 492 m) is number 9 on the same list, which contains only 1 American building – One World Trade Center (number 6 – 1,776 ft / 541 m).

Shanghai - skyline

It was a great journey, and all Chinese people were very nice for me. I look forward to come back to China in near future!


The taste of Netherlands: Tech Days NL in Amsterdam

TechDaysNL 2016

Earlier this month I had a pleasure to attend Tech Days NL conference in Amsterdam. It was very well organized conference.

I had opportunity to meet with many passionate developers. I met Bart de Smet. He is a C# Ninja who works for Microsoft at Bing Team, but he also C# compiler contributor. If you are C# passionate, like I am, I really recommend you to check out his talks. I also met Gil Cleeren (author of a few awesome Pluralsight courses about Xamarin, and organizer of Techorama conference), Roy Corneliasen (Xamarin MVP who is building mobile apps while working for Xpirit), and Mike Martin (Azure MVP). Check out their sessions as well! I’m looking forward to meet them again at MVP summit in Redmond this November!

Third time this year I spoke about the Azure Portal Architecture. I had to shorten my VSLive session from 1:15h to 45 mins. So if the VSLive session was too long, and too much of commitment for you, check out my Tech Days session 😉 I gave a high-level overview of our architecture, technologies we are using, our deployment approach, and lessons learned over last 2 years: performance tips & tricks, how to avoid regressions, and how to handle them when they happen.


UI Acceptance Testing Accessibility

Accessibility keywords

In the previous post Unit Testing Accessibility I showed how to run accessibility check on HTML node with aXe. This approach can be used to test components of your website.

You can take your accessibility testing to the next level by adding accessibility check for entire pages.

Do you remember Martin Fowler’s testing pyramid?

testign pyramid

While high-level UI tests can detect the same issues that unit tests can, usually UI tests are slower to run. Sometimes it takes 30 seconds to 1 minute to run 1 UI test, while unit test can be run in less than 100 milliseconds. Ideally you should have the most common scenario covered by UI test, and all possible customizations (on the component level) covered by unit tests. You can also add UI tests for some complex combinations of your components, and when you are fixing a bug in situation that unit test cannot cover encountered scenario. Because as you know, while fixing a bug, you should add unit test covering buggy scenario.

In general, unit tests and UI tests should be complementary. UI test should indicate an issue, while unit test should help you to find a source of the problem.

In accessibility testing, high-level checks are very useful in detecting accessibility issues caused by some “small change”. Once you detect accessibility violation, you should:

  1. narrow it down to particular unit of your website
  2. add unit test to cover broken scenario
  3. make sure it fails
  4. fix the issue (by writing code)
  5. make sure that unit test pass
  6. make sure that end to end UI tests pass

Check out Marcy Sutton’s article: Accessibility Testing with aXe and WebdriverJS. She created sample github repo that is demonstrating how to set everything up.


Unit Testing Accessibility

In Web Accessibility Hacker Way I mentioned that “only 20% of accessibility requirements can be verified by tools”. Nevertheless, it is worth to cover this 20%. Especially, when it is not very hard. You know that having automated test that guard against regressions always pays off in a long run.

As of today the best automatic verification tool for accessibility is aXe.

aXe

There is aXe Chrome plugin and aXe Firefox plugin that enables you to run accessibility audit manually:

aXe - results

Running automated tool manually is useful, but it is better to run it automatically as unit test, and incorporate it into your Continuous Integration to run it automatically after every commit.

Running accessibility audit with aXe

You can install aXe with npm:

npm i axe-core

The aXe has a function a11yCheck that performs accessibility audit on specified HTML Element. You may run it against widgets or partial views on your web app. That function takes 2 parameters:

  1. HTMLElement to be audited
  2. callback function that is invoked with results parameter
axe.a11yCheck($("#myElement")[0], function (results) {
    expect(results.violations.length).toBe(0);
    results.violations.length && console.error(results.violations);
});

It is useful to print errors to console, as the results.violations is an array of nested objects with different properties. Many of them are helpful to diagnose the issue.

aXe - console errors

*a11y is abbreviation for accessibility (similar like i18n for internationalization), 11 is number of letters between ‘a’ and ‘y’

Running aXe with Jasmine 2.x

In order to run aXe with Jasmine, you need to take into account that a11yCheck is asynchronous. Thus, you need to pass and invoke done function:

describe("a11y check", function() {
  it("has no accessbility violations (check console for errors)", function(done) {
    axe.a11yCheck($("#myElement")[0], function (results) {
        expect(results.violations.length).toBe(0);
        if (results.violations.length>0) {
            console.error(results.violations);
        }
        done();
    });
  });
});

Running aXe with QUnit 2.x

It is similar in QUnit. You also need to invoke done function, but first you need to get it by calling assert.async():

QUnit.test("a11y check", function(assert) {
    var done = assert.async();

    axe.a11yCheck($("#myElement")[0], function (results) {
        assert.strictEqual(results.violations.length, 0, "There should be no A11y violations (check console for errors)");
        if (results.violations.length>0) {
            console.error(results.violations);
        }
        done();
    });
});

Sample

I created a sample with button and input tag:

<div id="fixture">
  <button>My button</button> 
  <input type="text" />
</div>

This sample is not accessible because input tag does not have a label, and a11yCheck should report violations. The sample code, with Jasmine and QUnit tests, is available on github: axe-unittests.

Summary

While automated accessibility unit tests are great, you probably still want to use aXe plugin for Chrome to investigate reported violations. It’s more convenient, and it has neat user interface, while in unit tests you need to dig in into console errors.

When you start adding accessibility checks for different parts of your system, you may encounter many violations at first. It is better to still add tests, that ignore known violations, and then incrementally fix the issues. This approach prevents regressions, while delaying adding test until all violations are fixed may cause introducing new ones while fixing others.