Using Cognitive Services: Custom Vision Service with Azure IoT Edge

This is a guide on how to use Cognitive Services: Custom Vision Service with Azure IoT Edge without having the Edge module host a web endpoint but instead use the built in Module to Module communication. This post will break down the steps into four major sections:

  • Creating the Custom Vision Model
  • Creating the Edge Module in Python
  • Adding the model and custom code for Custom Vision
  • Deploy the Module

Creating the Custom Vision Model

To use the Custom Vision Service for image classification, you must first build a classifier model. In this guide, you’ll learn how to build a classifier through the Custom Vision website.


  • A valid Azure subscription. Create an account for free.
  • A set of images with which to train your classifier. See below for tips on choosing images.

Create Custom Vision resources in the Azure portal

To use Custom Vision Service, you will need to create Custom Vision Training and Prediction resources in the Azure portal. This will create both a Training and Prediction resource.

Create a new project

In your web browser, navigate to the Custom Vision web page and select Sign in. Sign in with the same account you used to sign into the Azure portal.

Image of the sign-in page

  1. To create your first project, select New Project. The Create new project dialog box will appear.The new project dialog box has fields for name, description, and domains.
  2. Enter a name and a description for the project. Then select a Resource Group. If your signed-in account is associated with an Azure account, the Resource Group dropdown will display all of your Azure Resource Groups that include a Custom Vision Service Resource.
  3. Select Classification under Project Types. Then, under Classification Types, choose either Multilabel or Multiclass, depending on your use case. Multilabel classification applies any number of your tags to an image (zero or more), while multiclass classification sorts images into single categories (every image you submit will be sorted into the most likely tag). You will be able to change the classification type later if you wish.
  4. Next, select one of the available domains. Each domain optimizes the classifier for specific types of images, as described in the following table. You will be able to change the domain later if you wish.
    Domain Purpose
    Generic Optimized for a broad range of image classification tasks. If none of the other domains are appropriate, or you are unsure of which domain to choose, select the Generic domain.
    Food Optimized for photographs of dishes as you would see them on a restaurant menu. If you want to classify photographs of individual fruits or vegetables, use the Food domain.
    Landmarks Optimized for recognizable landmarks, both natural and artificial. This domain works best when the landmark is clearly visible in the photograph. This domain works even if the landmark is slightly obstructed by people in front of it.
    Retail Optimized for images that are found in a shopping catalog or shopping website. If you want high precision classifying between dresses, pants, and shirts, use this domain.
    Compact domains Optimized for the constraints of real-time classification on mobile devices. The models generated by compact domains can be exported to run locally.
  5. Finally, select Create project.

Choose training images

As a minimum, we recommend you use at least 30 images per tag in the initial training set. You’ll also want to collect a few extra images to test your model once it is trained.

In order to train your model effectively, use images with visual variety. Select images with that vary by:

  • camera angle
  • lighting
  • background
  • visual style
  • individual/grouped subject(s)
  • size
  • type

Additionally, make sure all of your training images meet the following criteria:

  • .jpg, .png, or .bmp format
  • no greater than 6MB in size (4MB for prediction images)
  • no less than 256 pixels on the shortest edge; any images shorter than this will be automatically scaled up by the Custom Vision Service

Upload and tag images

In this section you will upload and manually tag images to help train the classifier.

  1. To add images, click the Add images button and then select Browse local files. Select Open to move to tagging. Your tag selection will be applied to the entire group of images you’ve selected to upload, so it is easier to upload images in separate groups according to their desired tags. You can also change the tags for individual images after they have been uploaded.The add images control is shown in the upper left, and as a button at bottom center.
  2. To create a tag, enter text in the My Tags field and press Enter. If the tag already exists, it will appear in a dropdown menu. In a multilabel project, you can add more than one tag to your images, but in a multiclass project you can add only one. To finish uploading the images, use the Upload [number] files button.Image of the tag and upload page
  3. Select Done once the images have been uploaded.The progress bar shows all tasks completed.

To upload another set of images, return to the top of this section and repeat the steps.

Train the classifier

To train the classifier, select the Train button. The classifier uses all of the current images to create a model that identifies the visual qualities of each tag.

The train button in the top right of the web page's header toolbar

The training process should only take a few minutes. During this time, information about the training process is displayed in the Performance tab.

The browser window with a training dialog in the main section

Custom Vision Service supports the following exports:

  • Tensorflow for Android.
  • CoreML for iOS11.
  • ONNX for Windows ML.
  • A Windows or Linux container. The container includes a Tensorflow model and service code to use the Custom Vision Service API.

Convert to a compact domain

To convert the domain of an existing classifier, use the following steps:

  1. From the Custom vision page, select the Home icon to view a list of your projects. You can also use the to see your projects.Image of the home icon and projects list
  2. Select a project, and then select the Gear icon in the upper right of the page.Image of the gear icon
  3. In the Domains section, select a compact domain. Select Save Changes to save the changes.Image of domains selection
  4. From the top of the page, select Train to retrain using the new domain.

Export your model

To export the model after retraining, use the following steps:

  1. Go to the Performance tab and select Export.Image of the export icon


    If the Export entry is not available, then the selected iteration does not use a compact domain. Use the Iterations section of this page to select an iteration that uses a compact domain, and then select Export.

  2. Select the export format, and then select Export to download the model.

Creating the Edge Module in Python

You can use Azure IoT Edge modules to deploy code that implements your business logic directly to your IoT Edge devices. This tutorial walks you through creating an IoT Edge module that will be edited to use the Custom Vision model exported. In this tutorial, you learn how to:

  • Use Visual Studio Code to create an IoT Edge Python module.
  • Use Visual Studio Code and Docker to create a Docker image and publish it to your registry.

If you don’t have an Azure subscription, create a free account before you begin.


Before beginning this tutorial, you should have gone through the previous tutorial to set up your development environment for Linux container development: Develop IoT Edge modules for Linux devices. By completing either of those tutorials, you should have the following prerequisites in place:

To develop an IoT Edge module in Python, install the following additional prerequisites on your development machine:

  • Python extension for Visual Studio Code.
  • Python.
  • Pip for installing Python packages (typically included with your Python installation).

Create a module project

The following steps create an IoT Edge Python module by using Visual Studio Code and the Azure IoT Tools.

Create a new project

Use the Python package cookiecutter to create a Python solution template that you can build on top of.

  1. In Visual Studio Code, select View > Terminal to open the VS Code integrated terminal.
  2. In the terminal, enter the following command to install (or update) cookiecutter, which you use to create the IoT Edge solution template:
    pip install --upgrade --user cookiecutter
  3. Select View > Command Palette to open the VS Code command palette.
  4. In the command palette, enter and run the command Azure: Sign in and follow the instructions to sign in your Azure account. If you’re already signed in, you can skip this step.

In the command palette, enter and run the command Azure IoT Edge: New IoT Edge solution. Follow the prompts and provide the following information to create your solution:

Field Value
Select folder Choose the location on your development machine for VS Code to create the solution files.
Provide a solution name Enter a descriptive name for your solution or accept the default EdgeSolution.
Select module template Choose Python Module.
Provide a module name Name your module PythonModule.
Provide Docker image repository for the module An image repository includes the name of your container registry and the name of your container image. Your container image is prepopulated from the name you provided in the last step. Replace localhost:5000 with the login server value from your Azure container registry. You can retrieve the login server from the Overview page of your container registry in the Azure portal.

The final image repository looks like <registry name>

Provide Docker image repository

Add your registry credentials

The environment file stores the credentials for your container repository and shares them with the IoT Edge runtime. The runtime needs these credentials to pull your private images onto the IoT Edge device.

  1. In the VS Code explorer, open the .env file.
  2. Update the fields with the username and password values that you copied from your Azure container registry.
  3. Save the .env file.

Select your target architecture

Currently, Visual Studio Code can develop C modules for Linux AMD64 and Linux ARM32v7 devices. You need to select which architecture you’re targeting with each solution, because the container is built and run differently for each architecture type. The default is Linux AMD64.

  1. Open the command palette and search for Azure IoT Edge: Set Default Target Platform for Edge Solution, or select the shortcut icon in the side bar at the bottom of the window.
  2. In the command palette, select the target architecture from the list of options. For this tutorial, we’re using an Ubuntu virtual machine as the IoT Edge device, so will keep the default amd64.

Adding the model and custom code for Custom Vision


Deploy the Module

Build and push your module

In the previous section, you created an IoT Edge solution and added code to the PythonModule that will filter out messages where the reported machine temperature is within the acceptable limits. Now you need to build the solution as a container image and push it to your container registry.

  1. Open the VS Code integrated terminal by selecting View > Terminal.
  2. Sign in to Docker by entering the following command in the terminal. Sign in with the username, password, and login server from your Azure container registry. You can retrieve these values from the Access keys section of your registry in the Azure portal.
    docker login -u <ACR username> -p <ACR password> <ACR login server>
    You may receive a security warning recommending the use of --password-stdin. While that best practice is recommended for production scenarios, it's outside the scope of this tutorial. For more information, see the docker login reference. In the VS Code explorer, right-click the deployment.template.json file and select Build and Push IoT Edge solution.

    The build and push command starts three operations. First, it creates a new folder in the solution called config that holds the full deployment manifest, built out of information in the deployment template and other solution files. Second, it runs docker build to build the container image based on the appropriate dockerfile for your target architecture. Then, it runs docker push to push the image repository to your container registry.

Deploy modules to device

Use the Visual Studio Code explorer and the Azure IoT Tools extension to deploy the module project to your IoT Edge device. You already have a deployment manifest prepared for your scenario, the deployment.json file in the config folder. All you need to do now is select a device to receive the deployment.

Make sure that your IoT Edge device is up and running.

  1. In the Visual Studio Code explorer, expand the Azure IoT Hub Devices section to see your list of IoT devices.
  2. Right-click the name of your IoT Edge device, then select Create Deployment for Single Device.
  3. Select the deployment.json file in the config folder and then click Select Edge Deployment Manifest. Do not use the deployment.template.json file.
  4. Click the refresh button. You should see the new PythonModule running along with the TempSensor module and the $edgeAgent and $edgeHub.




New Pluralsight Courses Released!

My new Pluralsight courses Cleaning and Preparing Data in Microsoft Azure and Architecting Xamarin.Forms Applications for Code Reuse were just released! Here are the synopsis:

Cleaning and Preparing Data in Microsoft Azure


This course targets software developers and data scientists looking to understand the initial steps in a machine learning solution. The content will showcase methods and tools available using Microsoft Azure.


No data science project of merit has ever started with great data ready to plug into an algorithm. In this course, Cleaning and Preparing Data in Microsoft Azure, you’ll learn foundational knowledge of the steps required to utilize data in a machine learning project. First, you’ll discover different types of data and languages. Next, you’ll learn about managing large data sets and handling bad data. Finally, you’ll explore how to utilize Azure Notebooks. When you’re finished with this course, you’ll have the skills and knowledge of preparing data needed for use in Microsoft Azure. Software required: Microsoft Azure.

Architecting Xamarin.Forms Applications for Code Reuse


A well-architected application is flexible to changing business requirements. This course will teach you how to architect Xamarin.Forms applications in a way that promotes reusable patterns.


As business requirements change, so do solution assumptions. In this course, Architecting Xamarin.Forms Applications for Code Reuse, you’ll learn different architectural patterns in Xamarin.Forms. First, you’ll explore project structure and organization. Next, you’ll discover patterns and standards to promote code sharing. Finally, you’ll learn how to utilize dependency injection in Xamarin.Forms. When you’re finished with this course, you’ll have the skills and knowledge of architecting Xamarin.Forms projects needed to optimally promote code reuse.

Azure IoT Hub – OpenSSL – Generate proof of possession

The Azure IoT documentation has guides on setting up certifications for production use. That documentation showcases how to properly setup using certificate authorities to generate proof of possession. For development purposes, you may want to use self signed certificates.

  1. Assuming  the original key and cert were created with the following commands (Azure IoT reports unverified if you upload it):
# Create root key
openssl genrsa -out iotHubRoot.key 2048

# Create root cert
openssl req -new -x509 -key iotHubRoot.key -out iotHubRoot.cer -days 500
  1. Then generate the verification cert (pay attention to fill in common name with verification code):
# Create verification key and csr
openssl genrsa -out verification.key 2048
openssl req -new -key verification.key -out verification.csr

#It will prompt for cert fields. 
#IMPORTANT: The Common Name needs to be your Verification Code (generate and copy that from portal)

# Create verification pem
openssl x509 -req -in -verification.csr -CA iotHubRoot.cer -CAkey iotHubRoot.key -CAcreateserial -out verification.pem -days 500 -sha256
  1. Upload pem file to portal to verify certificate

New Pluralsight Course Released!

My new Pluralsight course Sourcing Data in Microsoft Azure was just released! Here is the synopsis:


This course targets software developers looking to source data from inside and outside of the cloud. The content will also showcase methods and tools available using Microsoft Azure.


The cloud has nearly infinite compute power for processing. In this course, Sourcing Data in Microsoft Azure, you’ll learn foundational knowledge of data types, data policy, and finding data. First, you’ll learn how to register data sources with Azure Data Catalog. Next, you’ll discover how to extract, transform, and load data with Azure Data Factory. Finally, you’ll explore how to set up data processing with Azure HD Insight. When you’re finished with this course, you’ll have the skills and knowledge of the tools and processes needed to source data in Microsoft Azure. Software required: Microsoft Azure portal.

New Pluralsight Course Released!

My new Pluralsight course Deploying and Managing Models in Microsoft Azure was just released! Here is the synopsis:


In this course, you’ll learn about how data science practitioners can utilize tools for managing the models they create. You’ll also see those tools showcased in Microsoft Azure.


One of the most overlooked processes in data science is managing the life cycle of models. In this course, Deploying and Managing Models in Microsoft Azure, you’ll gain foundational knowledge of Azure Machine Learning. First, you’ll discover how to create and utilize Azure Machine Learning. Next, you’ll find out how to integrate with Azure DevOps. Finally, you’ll explore how to utilize them together to automate the deployment and management of models. When you’re finished with this course, you’ll have the skills and knowledge of model life cycle management needed to manage a machine learning project. Software required: Microsoft Azure.

Authoring for Pluralsight – Azure Machine Learning

Off to start another set of courses for Pluralsight:

  • Sourcing Data in Microsoft Azure
  • Deploying and Managing Models in Microsoft Azure
  • Cleaning and Preparing Data in Microsoft Azure

If you would like to check out any of my other courses, visit my author’s profile.

Sourcing Data in Microsoft Azure

This course is for people looking to move into the data sciences. They can have an existing background in development or IT.

This course will show how to find data in Microsoft Azure, how to move and change that data, and finally how to build workflows around that data.

This course assumes the developer has an understanding of basic computer terminology and the azure portal.

Deploying and Managing Models in Microsoft Azure

This course is for people looking to move into the data sciences. They can have an existing background in development or IT.

This course introduces the audience to the different data preparation steps involved with data projects. This course will show how to clean, transform, and wrangle the data needed for a data project.

This course assumes the developer has an understanding of basic computer terminology and the azure portal.

Cleaning and Preparing Data in Microsoft Azure

This course is for data science practitioners who need to learn more about how to utilize tools for managing the models they create.

The audience will be taken through automation and DevOps to learn more about how to manage their workflows. Everything from versioning, automated deployments, automated hyper-parameter tuning, and more will be discussed.

This course assumes the data scientist has an understanding of machine learning and common terminology and integration in machine learning projects. The course also assumes the data scientist has knowledge of Azure and the Azure portal.

Authoring for Pluralsight – Developing Microsoft Azure Intelligent Edge Solutions

Off to start another course for Pluralsight. This time its Developing Microsoft Azure Intelligent Edge Solutions. If you would like to check out any of my other courses, visit my author’s profile. The new course will cover the following topics:

  • Edge
    • IoT Architecture
    • IoT use cases and solutions
    • Edge Architecture
  • Azure IoT Hub
    • Overview of the IoT Ecosystem in Azure
    • IoT Hub message routing
    • Stream processing overview
  • Hot, Warm, and Cold paths
    • Use cases for hot, warm, and cold paths
    • Hot path with event hubs and log app
    • Warm path with Cosmos DB
    • Cold path with Azure Blob Storage
  • Real Time and Batch Processing
    • Overview and Demos of Stream Analytics Service
    • Overview and Demos of Time Series Insights

iotedge: error while loading shared libraries: cannot open shared object file: No such file or directory – Raspberry Pi

After installing Azure IoT Edge using the guide for Linux ARM32, the following error was presented: “iotedge: error while loading shared libraries: cannot open shared object file: No such file or directory“. 

The fix was simple enough, just install the building libssl1.02 using the following command:

sudo apt-get install libssl1.0.2

Test by running the iotedge command:



If that works successfully, restart the iotedge service:

service iotedge edge restart

Verify that it is running by checking the service status:

service iotedge edge status


Multiple TensorFlow Graphs from Cognitive Services – Custom Vision Service

For one project, there was a need for multiple models within the same Python application. These models were trained using the Cognitive Services: Custom Vision Service. There are two steps to using an exported model:

  1. Prepare the image
  2. Classify the image

Prepare an image for prediction

from PIL import Image
import numpy as np
import cv2
def convert_to_opencv(image):
# RGB -> BGR conversion is performed as well.
image = image.convert('RGB')
r,g,b = np.array(image).T
opencv_image = np.array([b,g,r]).transpose()
return opencv_image
def crop_center(img,cropx,cropy):
h, w = img.shape[:2]
startx = w//2(cropx//2)
starty = h//2(cropy//2)
return img[starty:starty+cropy, startx:startx+cropx]
def resize_down_to_1600_max_dim(image):
h, w = image.shape[:2]
if (h < 1600 and w < 1600):
return image
new_size = (1600 * w // h, 1600) if (h > w) else (1600, 1600 * h // w)
return cv2.resize(image, new_size, interpolation = cv2.INTER_LINEAR)
def resize_to_256_square(image):
h, w = image.shape[:2]
return cv2.resize(image, (256, 256), interpolation = cv2.INTER_LINEAR)
def update_orientation(image):
exif_orientation_tag = 0x0112
if hasattr(image, '_getexif'):
exif = image._getexif()
if (exif != None and exif_orientation_tag in exif):
orientation = exif.get(exif_orientation_tag, 1)
# orientation is 1 based, shift to zero based and flip/transpose based on 0-based values
orientation -= 1
if orientation >= 4:
image = image.transpose(Image.TRANSPOSE)
if orientation == 2 or orientation == 3 or orientation == 6 or orientation == 7:
image = image.transpose(Image.FLIP_TOP_BOTTOM)
if orientation == 1 or orientation == 2 or orientation == 5 or orientation == 6:
image = image.transpose(Image.FLIP_LEFT_RIGHT)
return image
def prepare_image(image):
# Update orientation based on EXIF tags, if the file has orientation info.
image = update_orientation(image)
# Convert to OpenCV format
image = convert_to_opencv(image)
# If the image has either w or h greater than 1600 we resize it down respecting
# aspect ratio such that the largest dimension is 1600
image = resize_down_to_1600_max_dim(image)
# We next get the largest center square
h, w = image.shape[:2]
min_dim = min(w,h)
max_square_image = crop_center(image, min_dim, min_dim)
# Resize that square down to 256×256
augmented_image = resize_to_256_square(max_square_image)
augmented_image = crop_center(augmented_image, 244, network_input_size)
return augmented_image

Classify the image

To run multiple models in Python was fairly simple. Simply call tf.reset_default_graph() after saving the loaded session into memory.

import tensorflow as tf
import numpy as np
# The category name and probability percentage
class CategoryScore:
def __init__(self, category, probability: float):
self.category = category
self.probability = probability
# The categorizer handles running tensorflow models
class Categorizer:
def __init__(self, model_file_path: str, map: []): = map
self.graph = tf.Graph()
self.graph_def = self.graph.as_graph_def()
with tf.gfile.GFile(model_file_path, 'rb') as f:
tf.import_graph_def(self.graph_def, name='')
output_layer = 'loss:0'
self.input_node = 'Placeholder:0'
self.sess = tf.Session()
self.prob_tensor = self.sess.graph.get_tensor_by_name(output_layer)
def score(self, image):
predictions, =, {self.input_node: [image]})
label_index = 0
scores = []
for p in predictions:
category_score = CategoryScore([label_index],np.float64(np.round(p, 8)))
label_index += 1
return scores

After the CustomVisionCategorizer is create, just call score and it will score with the labels in the map.

Securing SSH in Azure

On a recent project I inherited an Azure IaaS setup that managed Linux VMs by connecting via SSH from public IPs. I figured while we did a vNet migration we might as well secure the SSH pipeline.

Disable SSH Arcfour and CBC Ciphers

Arcfour is compatible with RC4 encryption and has issues with weak keys, which should be avoided. See RFC 4353 for more information here.

The SSH server located on the remote host also allows cipher block chaining (CBC) ciphers to be used to establish a Secure Shell (SSH) connection, leaving encrypted content vulnerable to a plaintext recovery attack. SSH is a cryptographic network protocol that allows encrypted connections between machines to be established. These connections can be used for remote login by an end user, or to encrypt network services. SSH leverages various encryption algorithms to make these connections, including ciphers that employ cipher block chaining.

The plaintext recovery attack can return up to thirty two bits of plaintext with a probability of 2-18 or fourteen bits of plain text with a probability of 2-14. This exposure is caused by the way CBC ciphers verify the message authentication code (MAC) for a block. Each block’s MAC is created by a combination of an unencrypted sequence number and an encrypted section containing the packet length, padding length, payload, and padding. With the length of the message encrypted the receiver of the packet needs to decrypt the first block of the message in order to obtain the length of the message to know how much data to read. As the location of the message length is static among all messages, the first four bytes will always be decrypted by a recipient. An attacker can take advantage of this by submitting an encrypted block, one byte at a time, directly to a waiting recipient. The recipient will automatically decrypt the first four bytes received as it the length is required to process the message’s MAC. Bytes controlled by an attacker can then be submitted until a MAC error is encountered, which will close the connection. Note as this attack will lead to the SSH connection to be closed, iterative attacks of this nature will be difficult to carry out against a target system.

Establishing an SSH connection using CBC mode ciphers can result in the exposure of plaintext messages, which are derived from an encrypted SSH connection. Depending on the data being transmitted, an attacker may be able to recover session identifiers, passwords, and any other data passed between the client and server.

Disable Arcfour ciphers in the SSH configuration. These ciphers are now disabled by default in some OpenSSH installations. All CBC mode ciphers should also be disabled on the target SSH server. In the place of CBC, SSH connections should be created using ciphers that utilize CTR (Counter) mode or GCM (Galois/Counter Mode), which are resistant to the plaintext recovery attack.

Disable SSH Weak MAC Algorithms

The SSH server is configured to allow cipher suites that include weak message authentication code (“MAC”) algorithms. Examples of weak MAC algorithms include MD5 and other known-weak hashes, and/or the use of 96-bit or shorter keys. The SSH protocol uses a MAC to ensure message integrity by hashing the encrypted message, and then sending both the message and the output of the MAC hash function to the recipient. The recipient then generates their hash of the message and related content and compares it to the received hash value. If the values match, there is a reasonable guarantee that the message is received “as is” and has not been tampered with in transit.

If the SSH server is configured to accept weak or otherwise vulnerable MAC algorithms, an attacker may be able to crack them in a reasonable timeframe. This has two potential effects:

  • The attacker may figure out the shared secret between the client and the server thereby allowing them to read sensitive data being exchanged. 
  • The attacker may be able to tamper with the data in-transit by injecting their own packets or modifying existing packet data sent within the SSH stream.

Disable all 96-bit HMAC algorithms, MD5-based HMAC algorithms, and all CBC mode ciphers configured for SSH on the server. The sshd_config file should only contain the following options as far as supported MAC algorithms are concerned:

  • hmac-sha2-512
  • hmac-sha2-256
  • hmac-ripemd160

In addition, all CBC mode ciphers should be replaced with their CTR mode counterparts.


To test, run the following command:

nmap -sS -sV -p 22 –script ssh2-enum-algos [TARGET IP]