Tag: C++

gRPC C++ and Self Signed Certificates

Playing around with gRPC with a C++ server caused an issue that took longer to solve than it should. Once the linker and other issues were solved, the following error started to follow:

7562 ssl_transport_security.cc:1238] Handshake failed with fatal error SSL_ERROR_SSL: error:100000c0:SSL routines:OPENSSL_internal:PEER_DID_NOT_RETURN_A_CERTIFICATE.

After searching, it lead me to this file where the different enumeration values for the SSL handling could be set.


/** Server does not request client certificate. A client can present a self
signed or signed certificates if it wishes to do so and they would be
accepted. */
GRPC_SSL_DONT_REQUEST_CLIENT_CERTIFICATE,
/** Server requests client certificate but does not enforce that the client
presents a certificate.

If the client presents a certificate, the client authentication is left to
the application based on the metadata like certificate etc.

The key cert pair should still be valid for the SSL connection to be
established. */
GRPC_SSL_REQUEST_CLIENT_CERTIFICATE_BUT_DONT_VERIFY,
/** Server requests client certificate but does not enforce that the client
presents a certificate.

If the client presents a certificate, the client authentication is done by
grpc framework (The client needs to either present a signed cert or skip no
certificate for a successful connection).

The key cert pair should still be valid for the SSL connection to be
established. */
GRPC_SSL_REQUEST_CLIENT_CERTIFICATE_AND_VERIFY,
/** Server requests client certificate but enforces that the client presents a
certificate.

If the client presents a certificate, the client authentication is left to
the application based on the metadata like certificate etc.

The key cert pair should still be valid for the SSL connection to be
established. */
GRPC_SSL_REQUEST_AND_REQUIRE_CLIENT_CERTIFICATE_BUT_DONT_VERIFY,
/** Server requests client certificate but enforces that the client presents a
certificate.

The cerificate presented by the client is verified by grpc framework (The
client needs to present signed certs for a successful connection).

The key cert pair should still be valid for the SSL connection to be
established. */
GRPC_SSL_REQUEST_AND_REQUIRE_CLIENT_CERTIFICATE_AND_VERIFY

That lead me to find a more through breakout of the use cases for each enumeration here.

  1. With GRPC_SSL_DONT_REQUEST_CLIENT_CERTIFICATE: Server does not request for a client certificate. So the client can choose to present a self-signed or a signed certificate or not present a certificate at all and all of these should be okay.
    With GRPC_SSL_REQUEST_CLIENT_CERTIFICATE_BUT_DONT_VERIFY: Server requests the client for a certificate but the signature enforcement is not done by grpc server framework but left to the app. The app can use metadata like the certificate hash to verify a certificate (essentially provides the server a
    way to verify self signed certificates, provided they have an out of band mechanism to register the certificate with the app)
  2. By “client authentication done by grpc framework”, I meant certificate signature verification is done using the ssl protocol itself by the grpc server framework (SSL_VERIFY_PEER option is being used in ssl options). The client has to provide a signed certificate which can be verified by the server (using the SSL roots file).
  3. “don’t request”/ “request”/ “require” / “verify”
    – Server has the option to either request or not-request for client cert.
    – Client can choose to either present a certificate or not.
    – Server can choose to verify the client certificate or not
    Each of these three options are independent of each other and contribute to multiple options presented.
    “require” for instance is the case server request for client cert, client has to present a certificate for the ssl handshake to continue but the server will not verify the client certificate using signature but can do so if needed based on certificate metadata.
    “verify” – SSL_VERIFY_PEER option is being used in ssl options and the client signature is verified/trusted by the server using the SSL roots file.
  4. All of the above pretty much expected that the private key and the public key files were all in okay and the only question was whether they were self signed or signed by a mutually trusted CA. If the public key and private keys don’t match up then the connection fails.
  5. It is a typo. It should have been “The client needs to either present a signed cert or not present a
    certificate at all for a successful connection”
  6. grpc_auth_context has various properties of the peer like GRPC_X509_CN_PROPERTY_NAME, GRPC_X509_PEM_CERT_PROPERTY_NAME, GRPC_X509_SAN_PROPERTY_NAME that can be used.

Finally, that lead me to understand that for self-signed certificates in development GRPC_SSL_REQUEST_CLIENT_CERTIFICATE_BUT_DONT_VERIFY was the right enumeration.

Creating .proto definitions from existing types at runtime

There was a need to create .proto definition files from the definitions of a reverse engineered database first project. The approach taken was that of using System.Emit to generate the type definitions and feed those to protobuf-net and use its ability to generate the .proto files.

There are only three classes needed:

  • ContextFinder
  • ClassGenerator
  • Program

The ContextFinder is pretty straight forward. It uses reflection to get all the generic parameters of DbSet<> properties within a DbContext. Then, ClassGenerator is used to copy the properties of the Types we harvested into a new type with the addition of adding ProtoContract and ProtoMember appropriately. Then, the Program class just loads the assembly from the file specified and runs the previously two mentioned classes.


public class ClassGenerator
{
private readonly ModuleBuilder _moduleBuilder;
public ClassGenerator()
{
var an = new AssemblyName("DynamicProtoAssembly");
AssemblyBuilder assemblyBuilder = AssemblyBuilder.DefineDynamicAssembly(an,AssemblyBuilderAccess.Run);
_moduleBuilder = assemblyBuilder.DefineDynamicModule("DynamicProtoModule");
}
public Type CreateType(Type typeToCopy)
{
TypeBuilder tb = _moduleBuilder.DefineType(typeToCopy.Name + "Proto",
TypeAttributes.Public |
TypeAttributes.Class |
TypeAttributes.AutoClass |
TypeAttributes.AnsiClass |
TypeAttributes.BeforeFieldInit |
TypeAttributes.AutoLayout,
null);
ConstructorBuilder constructor = tb.DefineDefaultConstructor(MethodAttributes.Public | MethodAttributes.SpecialName | MethodAttributes.RTSpecialName);
var ci = typeof(ProtoContractAttribute).GetConstructor(new Type[0]);
var builder = new CustomAttributeBuilder(ci,new object[0]);
tb.SetCustomAttribute(builder);
var propertiesToCopy = typeToCopy.GetProperties();
for (int i = 0; i < propertiesToCopy.Length; i++)
{
var propertyInfo = propertiesToCopy[i];
CreateProperty(tb,propertyInfo.Name,propertyInfo.PropertyType,i);
}
return tb.CreateType();
}
private static void CreateProperty(TypeBuilder tb, string propertyName, Type propertyType, int i)
{
FieldBuilder fieldBuilder = tb.DefineField("_" + propertyName, propertyType, FieldAttributes.Private);
PropertyBuilder propertyBuilder = tb.DefineProperty(propertyName, PropertyAttributes.HasDefault, propertyType, null);
MethodBuilder getPropMthdBldr = tb.DefineMethod("get_" + propertyName, MethodAttributes.Public | MethodAttributes.SpecialName | MethodAttributes.HideBySig, propertyType, Type.EmptyTypes);
ILGenerator getIl = getPropMthdBldr.GetILGenerator();
getIl.Emit(OpCodes.Ldarg_0);
getIl.Emit(OpCodes.Ldfld, fieldBuilder);
getIl.Emit(OpCodes.Ret);
MethodBuilder setPropMthdBldr =
tb.DefineMethod("set_" + propertyName,
MethodAttributes.Public |
MethodAttributes.SpecialName |
MethodAttributes.HideBySig,
null, new[] { propertyType });
ILGenerator setIl = setPropMthdBldr.GetILGenerator();
Label modifyProperty = setIl.DefineLabel();
Label exitSet = setIl.DefineLabel();
setIl.MarkLabel(modifyProperty);
setIl.Emit(OpCodes.Ldarg_0);
setIl.Emit(OpCodes.Ldarg_1);
setIl.Emit(OpCodes.Stfld, fieldBuilder);
setIl.Emit(OpCodes.Nop);
setIl.MarkLabel(exitSet);
setIl.Emit(OpCodes.Ret);
propertyBuilder.SetGetMethod(getPropMthdBldr);
propertyBuilder.SetSetMethod(setPropMthdBldr);
var ci = typeof(ProtoMemberAttribute).GetConstructor(new [] { typeof(int) });
var builder = new CustomAttributeBuilder(ci, new object[] { i + 1 });
propertyBuilder.SetCustomAttribute(builder);
}
}

view raw

ClassGenerator.cs

hosted with ❤ by GitHub


public class ContextFinder
{
public IEnumerable<Type> GetAllTypesInContextDbSets(Assembly assembly)
{
return GetContextTypes(assembly)
.Select(x => GetDataSetTypes(x))
.SelectMany(x => x)
.Select(x => x.GetGenericArguments()[0]);
}
private IEnumerable<Type> GetContextTypes(Assembly assembly)
{
return assembly.GetTypes()
.Where(myType => myType.IsClass && !myType.IsAbstract && myType.IsSubclassOf(typeof(DbContext)));
}
private IEnumerable<Type> GetDataSetTypes(Type context)
{
return context.GetProperties()
.Select(x => x.PropertyType)
.Where(x => x.IsGenericType && x.GetGenericTypeDefinition() == typeof(DbSet<>));
}
}

view raw

ContextFinder.cs

hosted with ❤ by GitHub


class Program
{
static void Main(string[] args)
{
if (args.Length < 1)
{
Console.WriteLine("The first argument should be a path to the assembly");
return;
}
Assembly assembly = Assembly.LoadFile(args[0]);
ContextFinder finder = new ContextFinder();
var types = finder.GetAllTypesInContextDbSets(assembly);
ClassGenerator generator = new ClassGenerator();
var protoTypes = types.Select(x => generator.CreateType(x));
foreach (var protoType in protoTypes)
{
Console.WriteLine(GenerateProtoFile(protoType));
}
}
private static string GenerateProtoFile(Type protoType)
{
MethodInfo methodInfo = typeof(Serializer).GetMethod(nameof(Serializer.GetProto),new [] {typeof(ProtoSyntax)});
MethodInfo genericMethod = methodInfo.MakeGenericMethod(protoType);
return (string) genericMethod.Invoke(null, new object[] { ProtoSyntax.Proto3 });
}
}

view raw

Program.cs

hosted with ❤ by GitHub

Using Open CV C++ with Azure IoT Edge

If you are looking for a guide on creating an Open CV module in Python, check out a guide here. This guide will focus on creating an Azure IoT Edge module in C++. To accomplish this we need to take the following steps:

Create the Azure IoT Edge Module

Prerequisites

This article assumes that you use a computer or virtual machine running Windows or Linux as your development machine. And you simulate your IoT Edge device on your development machine.

Needs:

To create a module, you need Docker to build the module image, and a container registry to hold the module image:

Create a new solution template

Take these steps to create an IoT Edge module based on Azure IoT C SDK using Visual Studio Code and the Azure IoT Edge extension. First you create a solution, and then you generate the first module in that solution. Each solution can contain more than one module.

  1. In Visual Studio Code, select View > Integrated Terminal.
  2. Select View > Command Palette.
  3. In the command palette, enter and run the command Azure IoT Edge: New IoT Edge Solution.Run New IoT Edge Solution
  4. Browse to the folder where you want to create the new solution. Choose Select folder.
  5. Enter a name for your solution.
  6. Select C Module as the template for the first module in the solution.
  7. Enter a name for your module. Choose a name that’s unique within your container registry.
  8. Provide the name of the module’s image repository. VS Code autopopulates the module name with localhost:5000. Replace it with your own registry information. If you use a local Docker registry for testing, then localhost is fine. If you use Azure Container Registry, then use the login server from your registry’s settings. The login server looks like .azurecr.io.

VS Code takes the information you provided, creates an IoT Edge solution, and then loads it in a new window.

View IoT Edge solution

There are four items within the solution:

  • A .vscode folder contains debug configurations.
  • A modules folder has subfolders for each module. At this point, you only have one. But you can add more in the command palette with the command Azure IoT Edge: Add IoT Edge Module.
  • An .env file lists your environment variables. If Azure Container Registry is your registry, you’ll have an Azure Container Registry username and password in it.

    Note

    The environment file is only created if you provide an image repository for the module. If you accepted the localhost defaults to test and debug locally, then you don’t need to declare environment variables.

  • A deployment.template.json file lists your new module along with a sample tempSensor module that simulates data you can use for testing. For more information about how deployment manifests work, see Learn how to use deployment manifests to deploy modules and establish routes.

Develop your module

The default C module code that comes with the solution is located at modules >  > main.c. The module and the deployment.template.json file are set up so that you can build the solution, push it to your container registry, and deploy it to a device to start testing without touching any code. The module is built to simply take input from a source (in this case, the tempSensor module that simulates data) and pipe it to IoT Hub.

When you’re ready to customize the C template with your own code, use the Azure IoT Hub SDKs to build modules that address the key needs for IoT solutions such as security, device management, and reliability.

Build and deploy your module for debugging

In each module folder, there are several Docker files for different container types. Use any of these files that end with the extension .debug to build your module for testing. Currently, C modules support debugging only in Linux amd64 containers.

  1. In VS Code, navigate to the deployment.template.json file. Update your module image URL by adding .debug to the end.Add **.debug** to your image name
  2. Replace the Node.js module createOptions in deployment.template.json with below content and save this file: 
    "createOptions": "{\"HostConfig\": {\"Privileged\": true}}"
  3. In the VS Code command palette, enter and run the command Edge: Build IoT Edge solution.
  4. Select the deployment.template.json file for your solution from the command palette.
  5. In Azure IoT Hub Device Explorer, right-click an IoT Edge device ID. Then select Create deployment for IoT Edge device.
  6. Open your solution’s config folder. Then select the deployment.json file. Choose Select Edge Deployment Manifest.

You’ll see the deployment successfully created with a deployment ID in a VS Code-integrated terminal.

Check your container status in the VS Code Docker explorer or by running the docker ps command in the terminal.

Start debugging C module in VS Code

VS Code keeps debugging configuration information in a launch.json file located in a .vscode folder in your workspace. This launch.json file was generated when you created a new IoT Edge solution. It updates each time you add a new module that supports debugging.

  1. Navigate to the VS Code debug view. Select the debug configuration file for your module. The debug option name should be similar to ModuleName Remote Debug (C)Select debug configuration.
  2. Navigate to main.c. Add a breakpoint in this file.
  3. Select Start Debugging or select F5. Select the process to attach to.
  4. In VS Code Debug view, you’ll see the variables in the left panel.

The preceding example shows how to debug C IoT Edge modules on containers. It added exposed ports in your module container createOptions. After you finish debugging your Node.js modules, we recommend you remove these exposed ports for production-ready IoT Edge modules.

Create a working Open CV Build

The working environment is an Ubuntu 18.04 64 bit Desktop OS running Clion using an embedded version of CMake 3.10. Open CV is added via source as a submodule to the project and added as a package in the CMakeLists.txt with the following line:

FIND_PACKAGE (OpenCV REQUIRED)

Once that was added to the CMakeLists.txt, the main.cpp file was changed to the following code:


#include <opencv2/opencv.hpp>
int main( int argc, char** argv )
{
VideoCapture cap;
if(!cap.open(0))
return 0;
Mat frame;
cap >> frame;
//Do something with the frame
}

view raw

main.cpp

hosted with ❤ by GitHub

Deploy the Azure IoT Edge Module

Once you create IoT Edge modules with your business logic, you want to deploy them to your devices to operate at the edge. If you have multiple modules that work together to collect and process data, you can deploy them all at once and declare the routing rules that connect them.

This article shows how to create a JSON deployment manifest, then use that file to push the deployment to an IoT Edge device. For information about creating a deployment that targets multiple devices based on their shared tags, see Deploy and monitor IoT Edge modules at scale

Prerequisites

Configure a deployment manifest

A deployment manifest is a JSON document that describes which modules to deploy, how data flows between the modules, and desired properties of the module twins. For more information about how deployment manifests work and how to create them, see Understand how IoT Edge modules can be used, configured, and reused.

To deploy modules using Visual Studio Code, save the deployment manifest locally as a .JSON file. You will use the file path in the next section when you run the command to apply the configuration to your device.

Here’s a basic deployment manifest with one module as an example:

{
  "modulesContent": {
    "$edgeAgent": {
      "properties.desired": {
        "schemaVersion": "1.0",
        "runtime": {
          "type": "docker",
          "settings": {
            "minDockerVersion": "v1.25",
            "loggingOptions": "",
            "registryCredentials": {}
          }
        },
        "systemModules": {
          "edgeAgent": {
            "type": "docker",
            "settings": {
              "image": "mcr.microsoft.com/azureiotedge-agent:1.0",
              "createOptions": "{}"
            }
          },
          "edgeHub": {
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "mcr.microsoft.com/azureiotedge-hub:1.0",
              "createOptions": "{}"
            }
          }
        },
        "modules": {
          "tempSensor": {
            "version": "1.0",
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "mcr.microsoft.com/azureiotedge-simulated-temperature-sensor:1.0",
              "createOptions": "{}"
            }
          }
        }
      }
    },
    "$edgeHub": {
      "properties.desired": {
        "schemaVersion": "1.0",
        "routes": {
            "route": "FROM /* INTO $upstream"
        },
        "storeAndForwardConfiguration": {
          "timeToLiveSecs": 7200
        }
      }
    },
    "tempSensor": {
      "properties.desired": {}
    }
  }
}

Sign in to access your IoT hub

You can use the Azure IoT extensions for Visual Studio Code to perform operations with your IoT hub. For these operations to work, you need to sign in to your Azure account and select the IoT hub that you are working on.

  1. In Visual Studio Code, open the Explorer view.
  2. At the bottom of the Explorer, expand the Azure IoT Hub Devices section.Expand Azure IoT Hub Devices
  3. Click on the … in the Azure IoT Hub Devices section header. If you don’t see the ellipsis, hover over the header.
  4. Choose Select IoT Hub.
  5. If you are not signed in to your Azure account, follow the prompts to do so.
  6. Select your Azure subscription.
  7. Select your IoT hub.

Deploy to your device

You deploy modules to your device by applying the deployment manifest that you configured with the module information.

  1. In the Visual Studio Code explorer view, expand the Azure IoT Hub Devices section.
  2. Right-click on the device that you want to configure with the deployment manifest.
  3. Select Create Deployment for IoT Edge Device.
  4. Navigate to the deployment manifest JSON file that you want to use, and click Select Edge Deployment Manifest.Select Edge Deployment Manifest

The results of your deployment are printed in the VS Code output. Successful deployments are applied within a few minutes if the target device is running and connected to the internet.

View modules on your device

Once you’ve deployed modules to your device, you can view all of them in the Azure IoT Hub Devices section. Select the arrow next to your IoT Edge device to expand it. All the currently running modules are displayed.

If you recently deployed new modules to a device, hover over the Azure IoT Hub Devices section header and select the refresh icon to update the view.

Right-click the name of a module to view and edit the module twin.

Authoring for Pluralsight

Coming soon I will be authoring a course for Pluralsight titled – “Identify Existing Products, Services and Technologies in Use For Microsoft Azure” . This course targets software developers who are looking to get started with Microsoft Azure services to build modern cloud-enabled solutions and want to further extend their knowledge of those services by learning how to use existing products, services, and technologies offered by Microsoft Azure.

Microsoft Azure is a host for almost any application, but determining how to use it within existing workflows is paramount for success. In this course, Identify Existing Products, Services and Technologies in Use, you will learn how to integrate existing workflows, technologies, and processes with Microsoft Azure.

We explore Microsoft Azure with the following technologies:

  • Languages, Frameworks, and IDEs –
    • IntelliJ IDEA
    • WebStorm
    • Visual Studio Code
    • .NET Core
    • C#
    • Java
    • JavaScript
    • Spring
    • NodeJS
    • Docker
  • Microsoft Azure Products
    • Azure App Services
    • Azure Kubernetes
    • Azure Functions
    • Azure IoT Hub

Hopefully we can take a developer familiar with the languages, frameworks, and ides available and make have them up and running on Microsoft Azure after this short course.

Generate Protocol Buffers on build with CMake

Just to see if it was possible on my current project, I tried to generate C++ code files from their .proto definitions whenever CMake ran. To do this, I added a few lines to the CMakeLists.txt file of the project. The idea is to use execute_process to call protoc and generate the files in the appropriate folder in the solution.

First, file(GLOB …) is used to set all of the .proto files into an iterable variable. Then, variables are setup for the proto_path and cpp_out variables.

After that, the files variable is looped and for each of the files we use execute_process to invoke protoc and generate the .pb.h and .pb.cc files.


file(GLOB PROTOBUF_DEFINITION_FILES "*.proto")
set(PROTOBUF_INPUT_DIRECTORY "${PROJECT_SOURCE_DIR}")
set(PROTOBUF_OUTPUT_DIRECTORY "${PROJECT_SOURCE_DIR}/Models/Proto/")
foreach(file ${PROTOBUF_DEFINITION_FILES})
set(PROTOBUF_ARGUMENTS "protoc –proto_path=\"${PROTOBUF_INPUT_DIRECTORY}\" –cpp_out=\"${PROTOBUF_OUTPUT_DIRECTORY}\" \"${file}\"")
execute_process(COMMAND ${PROTOBUF_OUTPUT_DIRECTORY}
WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}
RESULT_VARIABLE PROTOBUF_RESULT
OUTPUT_VARIABLE PROTOBUF_OUTPUT_VARIABLE)
endforeach()
file(GLOB PROTOBUF_MODELS_INCLUDES "Models/Proto/*.pb.cc" "Models/Proto/*.hpp")

view raw

CMakeLists.txt

hosted with ❤ by GitHub

Finally, we want to add the .pb.h and .pb.cc files to a variable for the final build. To do so, use file(GLOB …) again to search for all appropriate files.