gRPC C++ and Self Signed Certificates

Playing around with gRPC with a C++ server caused an issue that took longer to solve than it should. Once the linker and other issues were solved, the following error started to follow:

7562 ssl_transport_security.cc:1238] Handshake failed with fatal error SSL_ERROR_SSL: error:100000c0:SSL routines:OPENSSL_internal:PEER_DID_NOT_RETURN_A_CERTIFICATE.

After searching, it lead me to this file where the different enumeration values for the SSL handling could be set.


/** Server does not request client certificate. A client can present a self
signed or signed certificates if it wishes to do so and they would be
accepted. */
GRPC_SSL_DONT_REQUEST_CLIENT_CERTIFICATE,
/** Server requests client certificate but does not enforce that the client
presents a certificate.

If the client presents a certificate, the client authentication is left to
the application based on the metadata like certificate etc.

The key cert pair should still be valid for the SSL connection to be
established. */
GRPC_SSL_REQUEST_CLIENT_CERTIFICATE_BUT_DONT_VERIFY,
/** Server requests client certificate but does not enforce that the client
presents a certificate.

If the client presents a certificate, the client authentication is done by
grpc framework (The client needs to either present a signed cert or skip no
certificate for a successful connection).

The key cert pair should still be valid for the SSL connection to be
established. */
GRPC_SSL_REQUEST_CLIENT_CERTIFICATE_AND_VERIFY,
/** Server requests client certificate but enforces that the client presents a
certificate.

If the client presents a certificate, the client authentication is left to
the application based on the metadata like certificate etc.

The key cert pair should still be valid for the SSL connection to be
established. */
GRPC_SSL_REQUEST_AND_REQUIRE_CLIENT_CERTIFICATE_BUT_DONT_VERIFY,
/** Server requests client certificate but enforces that the client presents a
certificate.

The cerificate presented by the client is verified by grpc framework (The
client needs to present signed certs for a successful connection).

The key cert pair should still be valid for the SSL connection to be
established. */
GRPC_SSL_REQUEST_AND_REQUIRE_CLIENT_CERTIFICATE_AND_VERIFY

That lead me to find a more through breakout of the use cases for each enumeration here.

  1. With GRPC_SSL_DONT_REQUEST_CLIENT_CERTIFICATE: Server does not request for a client certificate. So the client can choose to present a self-signed or a signed certificate or not present a certificate at all and all of these should be okay.
    With GRPC_SSL_REQUEST_CLIENT_CERTIFICATE_BUT_DONT_VERIFY: Server requests the client for a certificate but the signature enforcement is not done by grpc server framework but left to the app. The app can use metadata like the certificate hash to verify a certificate (essentially provides the server a
    way to verify self signed certificates, provided they have an out of band mechanism to register the certificate with the app)
  2. By “client authentication done by grpc framework”, I meant certificate signature verification is done using the ssl protocol itself by the grpc server framework (SSL_VERIFY_PEER option is being used in ssl options). The client has to provide a signed certificate which can be verified by the server (using the SSL roots file).
  3. “don’t request”/ “request”/ “require” / “verify”
    – Server has the option to either request or not-request for client cert.
    – Client can choose to either present a certificate or not.
    – Server can choose to verify the client certificate or not
    Each of these three options are independent of each other and contribute to multiple options presented.
    “require” for instance is the case server request for client cert, client has to present a certificate for the ssl handshake to continue but the server will not verify the client certificate using signature but can do so if needed based on certificate metadata.
    “verify” – SSL_VERIFY_PEER option is being used in ssl options and the client signature is verified/trusted by the server using the SSL roots file.
  4. All of the above pretty much expected that the private key and the public key files were all in okay and the only question was whether they were self signed or signed by a mutually trusted CA. If the public key and private keys don’t match up then the connection fails.
  5. It is a typo. It should have been “The client needs to either present a signed cert or not present a
    certificate at all for a successful connection”
  6. grpc_auth_context has various properties of the peer like GRPC_X509_CN_PROPERTY_NAME, GRPC_X509_PEM_CERT_PROPERTY_NAME, GRPC_X509_SAN_PROPERTY_NAME that can be used.

Finally, that lead me to understand that for self-signed certificates in development GRPC_SSL_REQUEST_CLIENT_CERTIFICATE_BUT_DONT_VERIFY was the right enumeration.

Azure IoT Hub – OpenSSL – Generate proof of possession

The Azure IoT documentation has guides on setting up certifications for production use. That documentation showcases how to properly setup using certificate authorities to generate proof of possession. For development purposes, you may want to use self signed certificates.

  1. Assuming  the original key and cert were created with the following commands (Azure IoT reports unverified if you upload it):
# Create root key
openssl genrsa -out iotHubRoot.key 2048

# Create root cert
openssl req -new -x509 -key iotHubRoot.key -out iotHubRoot.cer -days 500
  1. Then generate the verification cert (pay attention to fill in common name with verification code):
# Create verification key and csr
openssl genrsa -out verification.key 2048
openssl req -new -key verification.key -out verification.csr

#It will prompt for cert fields. 
#IMPORTANT: The Common Name needs to be your Verification Code (generate and copy that from portal)

# Create verification pem
openssl x509 -req -in -verification.csr -CA iotHubRoot.cer -CAkey iotHubRoot.key -CAcreateserial -out verification.pem -days 500 -sha256
  1. Upload pem file to portal to verify certificate

New Pluralsight Course Released!

My new Pluralsight course Sourcing Data in Microsoft Azure was just released! Here is the synopsis:

Abstract

This course targets software developers looking to source data from inside and outside of the cloud. The content will also showcase methods and tools available using Microsoft Azure.

Description

The cloud has nearly infinite compute power for processing. In this course, Sourcing Data in Microsoft Azure, you’ll learn foundational knowledge of data types, data policy, and finding data. First, you’ll learn how to register data sources with Azure Data Catalog. Next, you’ll discover how to extract, transform, and load data with Azure Data Factory. Finally, you’ll explore how to set up data processing with Azure HD Insight. When you’re finished with this course, you’ll have the skills and knowledge of the tools and processes needed to source data in Microsoft Azure. Software required: Microsoft Azure portal.

New Pluralsight Course Released!

My new Pluralsight course Deploying and Managing Models in Microsoft Azure was just released! Here is the synopsis:

Abstract

In this course, you’ll learn about how data science practitioners can utilize tools for managing the models they create. You’ll also see those tools showcased in Microsoft Azure.

Description

One of the most overlooked processes in data science is managing the life cycle of models. In this course, Deploying and Managing Models in Microsoft Azure, you’ll gain foundational knowledge of Azure Machine Learning. First, you’ll discover how to create and utilize Azure Machine Learning. Next, you’ll find out how to integrate with Azure DevOps. Finally, you’ll explore how to utilize them together to automate the deployment and management of models. When you’re finished with this course, you’ll have the skills and knowledge of model life cycle management needed to manage a machine learning project. Software required: Microsoft Azure.

Authoring for Pluralsight – Azure Machine Learning

Off to start another set of courses for Pluralsight:

  • Sourcing Data in Microsoft Azure
  • Deploying and Managing Models in Microsoft Azure
  • Cleaning and Preparing Data in Microsoft Azure

If you would like to check out any of my other courses, visit my author’s profile.

Sourcing Data in Microsoft Azure

This course is for people looking to move into the data sciences. They can have an existing background in development or IT.

This course will show how to find data in Microsoft Azure, how to move and change that data, and finally how to build workflows around that data.

This course assumes the developer has an understanding of basic computer terminology and the azure portal.

Deploying and Managing Models in Microsoft Azure

This course is for people looking to move into the data sciences. They can have an existing background in development or IT.

This course introduces the audience to the different data preparation steps involved with data projects. This course will show how to clean, transform, and wrangle the data needed for a data project.

This course assumes the developer has an understanding of basic computer terminology and the azure portal.

Cleaning and Preparing Data in Microsoft Azure

This course is for data science practitioners who need to learn more about how to utilize tools for managing the models they create.

The audience will be taken through automation and DevOps to learn more about how to manage their workflows. Everything from versioning, automated deployments, automated hyper-parameter tuning, and more will be discussed.

This course assumes the data scientist has an understanding of machine learning and common terminology and integration in machine learning projects. The course also assumes the data scientist has knowledge of Azure and the Azure portal.

New Pluralsight Courses

I’ve been busy and not able to update that I have new courses available on Pluralsight:

Developing Microsoft Azure Intelligent Edge Solutions

This course targets software developers that are looking to build edge solutions that can process data and make intelligent decisions. This course will showcase how to develop those solutions using Microsoft Azure.

Over time, what was once simply Internet of Things solutions has evolved into Edge solutions. In this course, Developing an Intelligent Edge in Microsoft Azure, you will learn foundational knowledge of edge computing, how it interacts with data and messaging systems, and how to utilize both with Microsoft Azure. First, you will learn the concepts of edge and internet of things computing. Next, you will discover how to process streaming data on hot, warm, and cold paths. Finally, you will explore how real-time and batch processing can be utilized in an edge solution. When you are finished with this course, you will have the skills and knowledge of edge and internet of things in Azure needed to architect your next edge solution. Software required: Microsoft Azure, .NET.

Building Your First Data Science Project in Microsoft Azure

This course targets software developers looking to build data science solutions that can utilize the power of the cloud. The content will also showcase how to create those solutions using Microsoft Azure.

The past five years have shown a boom in the data science field with advancements in hardware and cloud computing. In this course, Building Your First Data Science Project in Microsoft Azure, you will learn about data science and how to get started utilizing it in Microsoft Azure. First, you will learn the data science and the tools surrounding it. Next, you will discover how to create a development environment in Microsoft Azure. Finally, you will explore how to maintain and utilize that development environment. When you are finished with this course, you will have the skills and knowledge of data science to build your first data science project in Microsoft Azure. Software required: Microsoft Azure.

 

Creating .proto definitions from existing types at runtime

There was a need to create .proto definition files from the definitions of a reverse engineered database first project. The approach taken was that of using System.Emit to generate the type definitions and feed those to protobuf-net and use its ability to generate the .proto files.

There are only three classes needed:

  • ContextFinder
  • ClassGenerator
  • Program

The ContextFinder is pretty straight forward. It uses reflection to get all the generic parameters of DbSet<> properties within a DbContext. Then, ClassGenerator is used to copy the properties of the Types we harvested into a new type with the addition of adding ProtoContract and ProtoMember appropriately. Then, the Program class just loads the assembly from the file specified and runs the previously two mentioned classes.


public class ClassGenerator
{
private readonly ModuleBuilder _moduleBuilder;
public ClassGenerator()
{
var an = new AssemblyName("DynamicProtoAssembly");
AssemblyBuilder assemblyBuilder = AssemblyBuilder.DefineDynamicAssembly(an,AssemblyBuilderAccess.Run);
_moduleBuilder = assemblyBuilder.DefineDynamicModule("DynamicProtoModule");
}
public Type CreateType(Type typeToCopy)
{
TypeBuilder tb = _moduleBuilder.DefineType(typeToCopy.Name + "Proto",
TypeAttributes.Public |
TypeAttributes.Class |
TypeAttributes.AutoClass |
TypeAttributes.AnsiClass |
TypeAttributes.BeforeFieldInit |
TypeAttributes.AutoLayout,
null);
ConstructorBuilder constructor = tb.DefineDefaultConstructor(MethodAttributes.Public | MethodAttributes.SpecialName | MethodAttributes.RTSpecialName);
var ci = typeof(ProtoContractAttribute).GetConstructor(new Type[0]);
var builder = new CustomAttributeBuilder(ci,new object[0]);
tb.SetCustomAttribute(builder);
var propertiesToCopy = typeToCopy.GetProperties();
for (int i = 0; i < propertiesToCopy.Length; i++)
{
var propertyInfo = propertiesToCopy[i];
CreateProperty(tb,propertyInfo.Name,propertyInfo.PropertyType,i);
}
return tb.CreateType();
}
private static void CreateProperty(TypeBuilder tb, string propertyName, Type propertyType, int i)
{
FieldBuilder fieldBuilder = tb.DefineField("_" + propertyName, propertyType, FieldAttributes.Private);
PropertyBuilder propertyBuilder = tb.DefineProperty(propertyName, PropertyAttributes.HasDefault, propertyType, null);
MethodBuilder getPropMthdBldr = tb.DefineMethod("get_" + propertyName, MethodAttributes.Public | MethodAttributes.SpecialName | MethodAttributes.HideBySig, propertyType, Type.EmptyTypes);
ILGenerator getIl = getPropMthdBldr.GetILGenerator();
getIl.Emit(OpCodes.Ldarg_0);
getIl.Emit(OpCodes.Ldfld, fieldBuilder);
getIl.Emit(OpCodes.Ret);
MethodBuilder setPropMthdBldr =
tb.DefineMethod("set_" + propertyName,
MethodAttributes.Public |
MethodAttributes.SpecialName |
MethodAttributes.HideBySig,
null, new[] { propertyType });
ILGenerator setIl = setPropMthdBldr.GetILGenerator();
Label modifyProperty = setIl.DefineLabel();
Label exitSet = setIl.DefineLabel();
setIl.MarkLabel(modifyProperty);
setIl.Emit(OpCodes.Ldarg_0);
setIl.Emit(OpCodes.Ldarg_1);
setIl.Emit(OpCodes.Stfld, fieldBuilder);
setIl.Emit(OpCodes.Nop);
setIl.MarkLabel(exitSet);
setIl.Emit(OpCodes.Ret);
propertyBuilder.SetGetMethod(getPropMthdBldr);
propertyBuilder.SetSetMethod(setPropMthdBldr);
var ci = typeof(ProtoMemberAttribute).GetConstructor(new [] { typeof(int) });
var builder = new CustomAttributeBuilder(ci, new object[] { i + 1 });
propertyBuilder.SetCustomAttribute(builder);
}
}


public class ContextFinder
{
public IEnumerable<Type> GetAllTypesInContextDbSets(Assembly assembly)
{
return GetContextTypes(assembly)
.Select(x => GetDataSetTypes(x))
.SelectMany(x => x)
.Select(x => x.GetGenericArguments()[0]);
}
private IEnumerable<Type> GetContextTypes(Assembly assembly)
{
return assembly.GetTypes()
.Where(myType => myType.IsClass && !myType.IsAbstract && myType.IsSubclassOf(typeof(DbContext)));
}
private IEnumerable<Type> GetDataSetTypes(Type context)
{
return context.GetProperties()
.Select(x => x.PropertyType)
.Where(x => x.IsGenericType && x.GetGenericTypeDefinition() == typeof(DbSet<>));
}
}


class Program
{
static void Main(string[] args)
{
if (args.Length < 1)
{
Console.WriteLine("The first argument should be a path to the assembly");
return;
}
Assembly assembly = Assembly.LoadFile(args[0]);
ContextFinder finder = new ContextFinder();
var types = finder.GetAllTypesInContextDbSets(assembly);
ClassGenerator generator = new ClassGenerator();
var protoTypes = types.Select(x => generator.CreateType(x));
foreach (var protoType in protoTypes)
{
Console.WriteLine(GenerateProtoFile(protoType));
}
}
private static string GenerateProtoFile(Type protoType)
{
MethodInfo methodInfo = typeof(Serializer).GetMethod(nameof(Serializer.GetProto),new [] {typeof(ProtoSyntax)});
MethodInfo genericMethod = methodInfo.MakeGenericMethod(protoType);
return (string) genericMethod.Invoke(null, new object[] { ProtoSyntax.Proto3 });
}
}

view raw

Program.cs

hosted with ❤ by GitHub

numpy/core/_multiarray_umath.cpython-35m-arm-linux-gnueabihf.so: undefined symbol: cblas_sgemm – Raspberry Pi

While working on a Raspberry Pi image that had been used prior by an electrical engineer to setup all of the dependencies for the hardware, there was an error when trying to upgrade to use Tensorflow. Tensorflow was needed to run a model trained with Cognitive Services: Custom Vision Service. The error was when the script imported Numpy. That caused the following error:

numpy/core/_multiarray_umath.cpython-35m-arm-linux-gnueabihf.so: undefined symbol: cblas_sgemm

To remedy this, all of the installations of Numpy had to be uninstalled. The following commands were run:

  • apt-get remove python-numpy
  • apt-get remove python3-numpy
  • pip3 uninstall numpy

After all three of those commands complete, Numpy was reinstalled using the package provided for raspian:

apt-get install python3-numpy

Authoring for Pluralsight – Developing Microsoft Azure Intelligent Edge Solutions

Off to start another course for Pluralsight. This time its Developing Microsoft Azure Intelligent Edge Solutions. If you would like to check out any of my other courses, visit my author’s profile. The new course will cover the following topics:

  • Edge
    • IoT Architecture
    • IoT use cases and solutions
    • Edge Architecture
  • Azure IoT Hub
    • Overview of the IoT Ecosystem in Azure
    • IoT Hub message routing
    • Stream processing overview
  • Hot, Warm, and Cold paths
    • Use cases for hot, warm, and cold paths
    • Hot path with event hubs and log app
    • Warm path with Cosmos DB
    • Cold path with Azure Blob Storage
  • Real Time and Batch Processing
    • Overview and Demos of Stream Analytics Service
    • Overview and Demos of Time Series Insights

Pluralsight Course Published – Designing an Intelligent Edge in Microsoft Azure

Designing an Intelligent Edge in Microsoft Azure was just published on Pluralsight! Check it out. Here is a synopsis of what’s in it:

This course targets software developers that are looking to integrate AI solutions in edge scenarios ranging from an edge data center down to secure microcontrollers. This course will showcase how to design solutions using Microsoft Azure.

Cloud computing has moved more and more out of the cloud and onto the edge. In this course, Designing an Intelligent Edge in Microsoft Azure, you will learn foundational knowledge of edge computing, its intersection with AI, and how to utilize both with Microsoft Azure. First, you will learn the concepts of edge computing. Next, you will discover how to create an edge solution utilizing Azure Stack, Azure Data Box Edge, and Azure IoT Edge. Finally, you will explore how to utilize off the shelf AI and build your own for Azure IoT Edge. When you are finished with this course, you will have the skills and knowledge of AI on the edge needed to architect your next edge solution. Software required: Microsoft Azure, .NET