# Using Cognitive Services: Custom Vision Service with Azure IoT Edge

https://jaredrhodes.com/blog/using-cognitive-services-custom-vision-service-with-azure-iot-edge/

This is a guide on how to use Cognitive Services: Custom Vision Service with Azure IoT Edge without having the Edge module host a web endpoint but instead use the built in Module to Module communication. This post will break down the steps into three major sections:

- Creating the Custom Vision Model
- Creating the Edge Module in Python
- Deploy the Module

## Creating the Custom Vision Model

To use the Custom Vision Service for image classification, you must first build a classifier model. In this guide, you'll learn how to build a classifier through the Custom Vision website.

### Prerequisites

- A valid Azure subscription. [Create an account](https://azure.microsoft.com/free/) for free.
- A set of images with which to train your classifier. See below for tips on choosing images.

### Create Custom Vision resources in the Azure portal

To use Custom Vision Service, you will need to create Custom Vision Training and Prediction resources in the [Azure portal](https://portal.azure.com/?microsoft_azure_marketplace_ItemHideKey=microsoft_azure_cognitiveservices_customvision#create/Microsoft.CognitiveServicesCustomVision). This will create both a Training and Prediction resource.

### Create a new project

In your web browser, navigate to the [Custom Vision web page](https://customvision.ai) and select **Sign in**. Sign in with the same account you used to sign into the Azure portal.

![Image of the sign-in page](https://docs.microsoft.com/en-us/azure/cognitive-services/custom-vision-service/media/browser-home.png)

1. To create your first project, select **New Project**. The **Create new project** dialog box will appear.![The new project dialog box has fields for name, description, and domains.](https://docs.microsoft.com/en-us/azure/cognitive-services/custom-vision-service/media/getting-started-build-a-classifier/new-project.png){: loading="lazy" }
2. Enter a name and a description for the project. Then select a Resource Group. If your signed-in account is associated with an Azure account, the Resource Group dropdown will display all of your Azure Resource Groups that include a Custom Vision Service Resource.
3. Select **Classification** under **Project Types**. Then, under **Classification Types**, choose either **Multilabel** or **Multiclass**, depending on your use case. Multilabel classification applies any number of your tags to an image (zero or more), while multiclass classification sorts images into single categories (every image you submit will be sorted into the most likely tag). You will be able to change the classification type later if you wish.
4. Next, select one of the available domains. Each domain optimizes the classifier for specific types of images, as described in the following table. You will be able to change the domain later if you wish.
| Domain | Purpose |
    |---|---|
    | **Generic** | Optimized for a broad range of image classification tasks. If none of the other domains are appropriate, or you are unsure of which domain to choose, select the Generic domain. |
    | **Food** | Optimized for photographs of dishes as you would see them on a restaurant menu. If you want to classify photographs of individual fruits or vegetables, use the Food domain. |
    | **Landmarks** | Optimized for recognizable landmarks, both natural and artificial. This domain works best when the landmark is clearly visible in the photograph. This domain works even if the landmark is slightly obstructed by people in front of it. |
    | **Retail** | Optimized for images that are found in a shopping catalog or shopping website. If you want high precision classifying between dresses, pants, and shirts, use this domain. |
    | **Compact domains** | Optimized for the constraints of real-time classification on mobile devices. The models generated by compact domains can be exported to run locally. |

5. Finally, select **Create project**.

### Choose training images

As a minimum, we recommend you use at least 30 images per tag in the initial training set. You'll also want to collect a few extra images to test your model once it is trained.

In order to train your model effectively, use images with visual variety. Select images with that vary by:

- camera angle
- lighting
- background
- visual style
- individual/grouped subject(s)
- size
- type

Additionally, make sure all of your training images meet the following criteria:

- .jpg, .png, or .bmp format
- no greater than 6MB in size (4MB for prediction images)
- no less than 256 pixels on the shortest edge; any images shorter than this will be automatically scaled up by the Custom Vision Service

### Upload and tag images

In this section you will upload and manually tag images to help train the classifier.

1. To add images, click the **Add images** button and then select **Browse local files**. Select **Open** to move to tagging. Your tag selection will be applied to the entire group of images you've selected to upload, so it is easier to upload images in separate groups according to their desired tags. You can also change the tags for individual images after they have been uploaded.![The add images control is shown in the upper left, and as a button at bottom center.](https://docs.microsoft.com/en-us/azure/cognitive-services/custom-vision-service/media/getting-started-build-a-classifier/add-images01.png){: loading="lazy" }
2. To create a tag, enter text in the **My Tags** field and press Enter. If the tag already exists, it will appear in a dropdown menu. In a multilabel project, you can add more than one tag to your images, but in a multiclass project you can add only one. To finish uploading the images, use the **Upload [number] files** button.![Image of the tag and upload page](https://docs.microsoft.com/en-us/azure/cognitive-services/custom-vision-service/media/getting-started-build-a-classifier/add-images03.png){: loading="lazy" }
3. Select **Done** once the images have been uploaded.![The progress bar shows all tasks completed.](https://docs.microsoft.com/en-us/azure/cognitive-services/custom-vision-service/media/getting-started-build-a-classifier/add-images04.png){: loading="lazy" }

To upload another set of images, return to the top of this section and repeat the steps.

### Train the classifier

To train the classifier, select the **Train** button. The classifier uses all of the current images to create a model that identifies the visual qualities of each tag.

![The train button in the top right of the web page's header toolbar](https://docs.microsoft.com/en-us/azure/cognitive-services/custom-vision-service/media/getting-started-build-a-classifier/train01.png){: loading="lazy" }

The training process should only take a few minutes. During this time, information about the training process is displayed in the **Performance** tab.

![The browser window with a training dialog in the main section](https://docs.microsoft.com/en-us/azure/cognitive-services/custom-vision-service/media/getting-started-build-a-classifier/train02.png){: loading="lazy" }

Custom Vision Service supports the following exports:

- **Tensorflow** for **Android**.
- **CoreML** for **iOS11**.
- **ONNX** for **Windows ML**.
- A Windows or Linux **container**. The container includes a Tensorflow model and service code to use the Custom Vision Service API.

### Convert to a compact domain

To convert the domain of an existing classifier, use the following steps:

1. From the [Custom vision page](https://customvision.ai), select the **Home** icon to view a list of your projects. You can also use the <https://customvision.ai/projects> to see your projects.![Image of the home icon and projects list](https://docs.microsoft.com/en-us/azure/cognitive-services/custom-vision-service/media/export-your-model/projects-list.png){: loading="lazy" }
2. Select a project, and then select the **Gear** icon in the upper right of the page.![Image of the gear icon](https://docs.microsoft.com/en-us/azure/cognitive-services/custom-vision-service/media/export-your-model/gear-icon.png){: loading="lazy" }
3. In the **Domains** section, select a **compact** domain. Select **Save Changes** to save the changes.![Image of domains selection](https://docs.microsoft.com/en-us/azure/cognitive-services/custom-vision-service/media/export-your-model/domains.png){: loading="lazy" }
4. From the top of the page, select **Train** to retrain using the new domain.

### Export your model

To export the model after retraining, use the following steps:

1. Go to the **Performance** tab and select **Export**.![Image of the export icon](https://docs.microsoft.com/en-us/azure/cognitive-services/custom-vision-service/media/export-your-model/export.png){: loading="lazy" }
*Tip*

    *If the **Export** entry is not available, then the selected iteration does not use a compact domain. Use the **Iterations** section of this page to select an iteration that uses a compact domain, and then select **Export**.*

2. Select the export format, and then select **Export** to download the model.

## Creating the Edge Module in Python

You can use Azure IoT Edge modules to deploy code that implements your business logic directly to your IoT Edge devices. This tutorial walks you through creating an IoT Edge module that will be edited to use the Custom Vision model exported. In this tutorial, you learn how to:

- Use Visual Studio Code to create an IoT Edge Python module.
- Use Visual Studio Code and Docker to create a Docker image and publish it to your registry.

If you don't have an [Azure subscription](https://docs.microsoft.com/en-us/azure/guides/developer/azure-developer-guide#understanding-accounts-subscriptions-and-billing), create a [free account](https://azure.microsoft.com/free/?ref=microsoft.com&utm_source=microsoft.com&utm_medium=docs&utm_campaign=visualstudio) before you begin.

### Prerequisites

Before beginning this tutorial, you should have gone through the previous tutorial to set up your development environment for Linux container development: [Develop IoT Edge modules for Linux devices](https://docs.microsoft.com/en-us/azure/iot-edge/tutorial-develop-for-linux). By completing either of those tutorials, you should have the following prerequisites in place:

- A free or standard-tier [IoT Hub](https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-create-through-portal) in Azure.
- A [Linux device running Azure IoT Edge](https://docs.microsoft.com/en-us/azure/iot-edge/quickstart-linux)
- A container registry, like [Azure Container Registry](https://docs.microsoft.com/azure/container-registry/).
- [Visual Studio Code](https://code.visualstudio.com/) configured with the [Azure IoT Tools](https://marketplace.visualstudio.com/items?itemName=vsciot-vscode.azure-iot-tools).
- [Docker CE](https://docs.docker.com/install/) configured to run Linux containers.

To develop an IoT Edge module in Python, install the following additional prerequisites on your development machine:

- [Python extension](https://marketplace.visualstudio.com/items?itemName=ms-python.python) for Visual Studio Code.
- [Python](https://www.python.org/downloads/).
- [Pip](https://pip.pypa.io/en/stable/installing/#installation) for installing Python packages (typically included with your Python installation).

### Create a module project

The following steps create an IoT Edge Python module by using Visual Studio Code and the Azure IoT Tools.

#### Create a new project

Use the Python package **cookiecutter** to create a Python solution template that you can build on top of.

1. In Visual Studio Code, select **View** &gt; **Terminal** to open the VS Code integrated terminal.
2. In the terminal, enter the following command to install (or update) **cookiecutter**, which you use to create the IoT Edge solution template:

    ```
    pip install --upgrade --user cookiecutter
    ```

3. Select **View** &gt; **Command Palette** to open the VS Code command palette.
4. In the command palette, enter and run the command **Azure: Sign in** and follow the instructions to sign in your Azure account. If you're already signed in, you can skip this step.

In the command palette, enter and run the command **Azure IoT Edge: New IoT Edge solution**. Follow the prompts and provide the following information to create your solution:

| Field | Value |
|---|---|
| Select folder | Choose the location on your development machine for VS Code to create the solution files. |
| Provide a solution name | Enter a descriptive name for your solution or accept the default **EdgeSolution**. |
| Select module template | Choose **Python Module**. |
| Provide a module name | Name your module **PythonModule**. |
| Provide Docker image repository for the module | An image repository includes the name of your container registry and the name of your container image. Your container image is prepopulated from the name you provided in the last step. Replace **localhost:5000** with the login server value from your Azure container registry. You can retrieve the login server from the Overview page of your container registry in the Azure portal. The final image repository looks like &lt;registry name&gt;.azurecr.io/pythonmodule. |

![Provide Docker image repository](https://docs.microsoft.com/en-us/azure/iot-edge/media/tutorial-python-module/repository.png)

#### Add your registry credentials

The environment file stores the credentials for your container repository and shares them with the IoT Edge runtime. The runtime needs these credentials to pull your private images onto the IoT Edge device.

1. In the VS Code explorer, open the **.env** file.
2. Update the fields with the **username** and **password** values that you copied from your Azure container registry.
3. Save the .env file.

#### Select your target architecture

Currently, Visual Studio Code can develop C modules for Linux AMD64 and Linux ARM32v7 devices. You need to select which architecture you're targeting with each solution, because the container is built and run differently for each architecture type. The default is Linux AMD64.

1. Open the command palette and search for **Azure IoT Edge: Set Default Target Platform for Edge Solution**, or select the shortcut icon in the side bar at the bottom of the window.
2. In the command palette, select the target architecture from the list of options. For this tutorial, we're using an Ubuntu virtual machine as the IoT Edge device, so will keep the default **amd64**.

## Deploy the Module

### Build and push your module

Now that you have an IoT Edge solution with the PythonModule created, you need to build the solution as a container image and push it to your container registry.

1. Open the VS Code integrated terminal by selecting **View** &gt; **Terminal**.
2. Sign in to Docker by entering the following command in the terminal. Sign in with the username, password, and login server from your Azure container registry. You can retrieve these values from the **Access keys** section of your registry in the Azure portal.

    ```
    docker login -u <ACR username> -p <ACR password> <ACR login server>
    ```

    You may receive a security warning recommending the use of `--password-stdin`. While that best practice is recommended for production scenarios, it's outside the scope of this tutorial. For more information, see the [docker login](https://docs.docker.com/engine/reference/commandline/login/#provide-a-password-using-stdin) reference. In the VS Code explorer, right-click the **deployment.template.json** file and select **Build and Push IoT Edge solution**.

The build and push command starts three operations. First, it creates a new folder in the solution called **config** that holds the full deployment manifest, built out of information in the deployment template and other solution files. Second, it runs `docker build` to build the container image based on the appropriate dockerfile for your target architecture. Then, it runs `docker push` to push the image repository to your container registry.

### Deploy modules to device

Use the Visual Studio Code explorer and the Azure IoT Tools extension to deploy the module project to your IoT Edge device. You already have a deployment manifest prepared for your scenario, the **deployment.json** file in the config folder. All you need to do now is select a device to receive the deployment.

Make sure that your IoT Edge device is up and running.

1. In the Visual Studio Code explorer, expand the **Azure IoT Hub Devices** section to see your list of IoT devices.
2. Right-click the name of your IoT Edge device, then select **Create Deployment for Single Device**.
3. Select the **deployment.json** file in the **config** folder and then click **Select Edge Deployment Manifest**. Do not use the deployment.template.json file.
4. Click the refresh button. You should see the new **PythonModule** running along with the **TempSensor** module and the **$edgeAgent** and **$edgeHub**.