numpy/core/_multiarray_umath.cpython-35m-arm-linux-gnueabihf.so: undefined symbol: cblas_sgemm – Raspberry Pi

While working on a Raspberry Pi image that had been used prior by an electrical engineer to setup all of the dependencies for the hardware, there was an error when trying to upgrade to use Tensorflow. Tensorflow was needed to run a model trained with Cognitive Services: Custom Vision Service. The error was when the script imported Numpy. That caused the following error:

numpy/core/_multiarray_umath.cpython-35m-arm-linux-gnueabihf.so: undefined symbol: cblas_sgemm

To remedy this, all of the installations of Numpy had to be uninstalled. The following commands were run:

  • apt-get remove python-numpy
  • apt-get remove python3-numpy
  • pip3 uninstall numpy

After all three of those commands complete, Numpy was reinstalled using the package provided for raspian:

apt-get install python3-numpy

Authoring for Pluralsight – Developing Microsoft Azure Intelligent Edge Solutions

Off to start another course for Pluralsight. This time its Developing Microsoft Azure Intelligent Edge Solutions. If you would like to check out any of my other courses, visit my author’s profile. The new course will cover the following topics:

  • Edge
    • IoT Architecture
    • IoT use cases and solutions
    • Edge Architecture
  • Azure IoT Hub
    • Overview of the IoT Ecosystem in Azure
    • IoT Hub message routing
    • Stream processing overview
  • Hot, Warm, and Cold paths
    • Use cases for hot, warm, and cold paths
    • Hot path with event hubs and log app
    • Warm path with Cosmos DB
    • Cold path with Azure Blob Storage
  • Real Time and Batch Processing
    • Overview and Demos of Stream Analytics Service
    • Overview and Demos of Time Series Insights

Pluralsight Course Published – Designing an Intelligent Edge in Microsoft Azure

Designing an Intelligent Edge in Microsoft Azure was just published on Pluralsight! Check it out. Here is a synopsis of what’s in it:

This course targets software developers that are looking to integrate AI solutions in edge scenarios ranging from an edge data center down to secure microcontrollers. This course will showcase how to design solutions using Microsoft Azure.

Cloud computing has moved more and more out of the cloud and onto the edge. In this course, Designing an Intelligent Edge in Microsoft Azure, you will learn foundational knowledge of edge computing, its intersection with AI, and how to utilize both with Microsoft Azure. First, you will learn the concepts of edge computing. Next, you will discover how to create an edge solution utilizing Azure Stack, Azure Data Box Edge, and Azure IoT Edge. Finally, you will explore how to utilize off the shelf AI and build your own for Azure IoT Edge. When you are finished with this course, you will have the skills and knowledge of AI on the edge needed to architect your next edge solution. Software required: Microsoft Azure, .NET

 

iotedge: error while loading shared libraries: libssl.so.1.0.2: cannot open shared object file: No such file or directory – Raspberry Pi

After installing Azure IoT Edge using the guide for Linux ARM32, the following error was presented: “iotedge: error while loading shared libraries: libssl.so.1.0.2: cannot open shared object file: No such file or directory“. 

The fix was simple enough, just install the building libssl1.02 using the following command:

sudo apt-get install libssl1.0.2

Test by running the iotedge command:

iotedge

azureiotedgeCapture.PNG

If that works successfully, restart the iotedge service:

service iotedge edge restart

Verify that it is running by checking the service status:

service iotedge edge status

azureiotedgeCapture

MVP Renewal

Proudly, I will be entering my third year as a Microsoft MVP. This will be under the Microsoft Azure category again. Moving forward, I look forward to doing a large amount of work and training with Azure Edge and Azure ML. Specifically, I look forward to working on the Scry Unlimited and other projects I find throughout the year. To contact me for your project, please visit the contact page.

As a start, on 7/16/2019 I will be presenting AI on the Edge at the Azure in the ATL user group. Following that up I will be speaking at events around the country and hopefully internationally again. In addition to my normal speaking on Mobile, Cloud, and Edge; I will be adding Machine Learning and Artificial Intelligence specifically focusing on the integration with Edge and Mobile computing.

Finally, I am still putting together events in Atlanta. If you would like to participate in any of the following events, just follow their links or message me on Twitter:

Multiple TensorFlow Graphs from Cognitive Services – Custom Vision Service

For one project, there was a need for multiple models within the same Python application. These models were trained using the Cognitive Services: Custom Vision Service. There are two steps to using an exported model:

  1. Prepare the image
  2. Classify the image

Prepare an image for prediction

from PIL import Image
import numpy as np
import cv2
def convert_to_opencv(image):
# RGB -> BGR conversion is performed as well.
image = image.convert('RGB')
r,g,b = np.array(image).T
opencv_image = np.array([b,g,r]).transpose()
return opencv_image
def crop_center(img,cropx,cropy):
h, w = img.shape[:2]
startx = w//2(cropx//2)
starty = h//2(cropy//2)
return img[starty:starty+cropy, startx:startx+cropx]
def resize_down_to_1600_max_dim(image):
h, w = image.shape[:2]
if (h < 1600 and w < 1600):
return image
new_size = (1600 * w // h, 1600) if (h > w) else (1600, 1600 * h // w)
return cv2.resize(image, new_size, interpolation = cv2.INTER_LINEAR)
def resize_to_256_square(image):
h, w = image.shape[:2]
return cv2.resize(image, (256, 256), interpolation = cv2.INTER_LINEAR)
def update_orientation(image):
exif_orientation_tag = 0x0112
if hasattr(image, '_getexif'):
exif = image._getexif()
if (exif != None and exif_orientation_tag in exif):
orientation = exif.get(exif_orientation_tag, 1)
# orientation is 1 based, shift to zero based and flip/transpose based on 0-based values
orientation -= 1
if orientation >= 4:
image = image.transpose(Image.TRANSPOSE)
if orientation == 2 or orientation == 3 or orientation == 6 or orientation == 7:
image = image.transpose(Image.FLIP_TOP_BOTTOM)
if orientation == 1 or orientation == 2 or orientation == 5 or orientation == 6:
image = image.transpose(Image.FLIP_LEFT_RIGHT)
return image
def prepare_image(image):
# Update orientation based on EXIF tags, if the file has orientation info.
image = update_orientation(image)
# Convert to OpenCV format
image = convert_to_opencv(image)
# If the image has either w or h greater than 1600 we resize it down respecting
# aspect ratio such that the largest dimension is 1600
image = resize_down_to_1600_max_dim(image)
# We next get the largest center square
h, w = image.shape[:2]
min_dim = min(w,h)
max_square_image = crop_center(image, min_dim, min_dim)
# Resize that square down to 256×256
augmented_image = resize_to_256_square(max_square_image)
augmented_image = crop_center(augmented_image, 244, network_input_size)
return augmented_image

Classify the image

To run multiple models in Python was fairly simple. Simply call tf.reset_default_graph() after saving the loaded session into memory.

import tensorflow as tf
import numpy as np
# The category name and probability percentage
class CategoryScore:
def __init__(self, category, probability: float):
self.category = category
self.probability = probability
# The categorizer handles running tensorflow models
class Categorizer:
def __init__(self, model_file_path: str, map: []):
self.map = map
self.graph = tf.Graph()
self.graph.as_default()
self.graph_def = self.graph.as_graph_def()
with tf.gfile.GFile(model_file_path, 'rb') as f:
self.graph_def.ParseFromString(f.read())
tf.import_graph_def(self.graph_def, name='')
output_layer = 'loss:0'
self.input_node = 'Placeholder:0'
self.sess = tf.Session()
self.prob_tensor = self.sess.graph.get_tensor_by_name(output_layer)
tf.reset_default_graph()
def score(self, image):
predictions, = self.sess.run(self.prob_tensor, {self.input_node: [image]})
label_index = 0
scores = []
for p in predictions:
category_score = CategoryScore(self.map[label_index],np.float64(np.round(p, 8)))
scores.append(category_score)
label_index += 1
return scores

After the CustomVisionCategorizer is create, just call score and it will score with the labels in the map.

Communicating between Python and .NET Core with Boost Interprocess

To see if I could, I put together a cross communication library for .Net Core and Python applications using Boost.Interprocess, Boost.Python, and Boost.Signals2. The goal was simple, expose the same interface for cross communication to C# and Python. The approach taken was to use the condition example and edit it to expose to the different languages.

Shared Definitions

First I need to create the objects to make the interface. There are four files making up these objects:

  • shm_remove.hpp – just a lifecycle object to clear the shared buffer when it is destructed
  • TraceQueue.hpp – The shared memory object
  • SharedMemoryConsumer.hpp – The subscriber to the shared memory data
  • SharedMemoryProducer.hpp – The publisher for the shared memory data
template<size_t BufferSize>
class SharedMemoryConsumer
{
public:
explicit SharedMemoryConsumer(const std::string& memory_name)
: name_length_(memory_name.size()),
shm_(boost::interprocess::open_or_create,memory_name.c_str(),boost::interprocess::read_write)
{
shm_.truncate(sizeof(TraceQueue<BufferSize>));
region_ = boost::interprocess::mapped_region(shm_,boost::interprocess::read_write);
//Get the address of the mapped region
void * addr = region_.get_address();
//Obtain a pointer to the shared structure
data_ = static_cast<TraceQueue<BufferSize>*>(addr);
if (data_->name_length != name_length_)
{
data_ = new (addr)TraceQueue<BufferSize>;
data_->name_length = name_length_;
}
}
~SharedMemoryConsumer()
{
keep_listening_ = false;
background_listener_.join();
}
void start()
{
keep_listening_ = true;
background_listener_ = std::thread([=](){
do {
boost::interprocess::scoped_lock<boost::interprocess::interprocess_mutex> lock(data_->mutex);
if (!data_->message_in) {
data_->cond_empty.wait(lock);
}
sig_(data_->message);
//Notify the other process that the buffer is empty
data_->message_in = false;
data_->cond_full.notify_one();
} while (!data_->closing_stream && keep_listening_);
});
}
void set_callable(const std::function<void(const InterprocessMessage<BufferSize>& memory_block)>& callback)
{
conn_ = sig_.connect([=](const InterprocessMessage<BufferSize>& message)
{
callback(message);
});
}
void wait()
{
background_listener_.join();
}
private:
int name_length_;
boost::signals2::signal<void(const InterprocessMessage<BufferSize>&)> sig_;
boost::signals2::connection conn_;
boost::interprocess::shared_memory_object shm_;
boost::interprocess::mapped_region region_;
std::thread background_listener_;
bool keep_listening_;
TraceQueue<BufferSize> * data_;
};

template <size_t BufferSize>
class SharedMemoryProducer
{
public:
explicit SharedMemoryProducer(const std::string& memory_name)
: name_length_(memory_name.size())
,remove_guard_(memory_name)
, shm_(boost::interprocess::open_or_create,memory_name.c_str(),boost::interprocess::read_write)
{
shm_.truncate(sizeof(TraceQueue<BufferSize>));
region_ = boost::interprocess::mapped_region(shm_,boost::interprocess::read_write);
//Get the address of the mapped region
void * addr = region_.get_address();
//Construct the shared structure in memory
data_ = static_cast<TraceQueue<BufferSize>*>(addr);
if (data_->name_length != name_length_)
{
data_ = new (addr)TraceQueue<BufferSize>;
data_->name_length = name_length_;
}
}
~SharedMemoryProducer()
{
data_->name_length = 0;
data_->closing_stream = true;
}
void write_data(const std::string& buffer)
{
if (buffer.size() > BufferSize)
{
throw std::invalid_argument("Message size is larger than buffer size");
}
boost::interprocess::scoped_lock<boost::interprocess::interprocess_mutex> lock(data_->mutex);
if(data_->message_in){
data_->cond_full.wait(lock);
}
memcpy(data_->message.message,buffer.data(),buffer.size());
data_->message.message_length = buffer.size();
//Notify to the other process that there is a message
data_->cond_empty.notify_one();
//Mark message buffer as full
data_->message_in = true;
}
private:
int name_length_;
shm_remove remove_guard_;
boost::interprocess::shared_memory_object shm_;
boost::interprocess::mapped_region region_;
TraceQueue<BufferSize> * data_;
};

#include <boost/interprocess/shared_memory_object.hpp>
struct shm_remove
{
public:
explicit shm_remove(const std::string& shm_name)
: shm_name_(shm_name)
{
// boost::interprocess::shared_memory_object::remove(shm_name.c_str());
}
~shm_remove(){ boost::interprocess::shared_memory_object::remove(shm_name_.c_str()); }
private:
std::string shm_name_;
};

view raw
shm_remove.hpp
hosted with ❤ by GitHub

template <size_t MESSAGE_SIZE>
struct InterprocessMessage
{
//Items to fill
char message[MESSAGE_SIZE];
//Message size
size_t message_length;
std::string to_string() const
{
return std::string(message,message_length);
}
};
template <size_t MESSAGE_SIZE>
struct TraceQueue
{
TraceQueue()
: message_in(false)
{}
//Mutex to protect access to the queue
boost::interprocess::interprocess_mutex mutex;
//Condition to wait when the queue is empty
boost::interprocess::interprocess_condition cond_empty;
//Condition to wait when the queue is full
boost::interprocess::interprocess_condition cond_full;
//Message
InterprocessMessage<MESSAGE_SIZE> message;
//Is there any message
bool message_in;
//Notify closing shop
bool closing_stream = false;
//Check to see if memory has been created
int name_length;
};

view raw
TraceQueue.hpp
hosted with ❤ by GitHub

These objects comprise the core interface of the shared memory provider. Now, the memory providers need to be exposed to multiple languages. There are different ways to do this and I decided to do it by hand. I should point out SWIG is my usual approach to this task, however, in this instance it seemed easy enough to do it by hand.

Boost Python

To expose the python code, I needed to create a few classes to expose the interface definitions to Boost.Python. The two classes are:

  • PythonSharedMemoryConsumer.hpp – The python interface for the SharedMemoryConsumer
  • PythonModule.cpp  – The file that exposes the module to python
BOOST_PYTHON_MODULE(ScryUnlimitedCommunicator)
{
Py_Initialize();
class_<SeralizedPythonSharedMemoryConsumer, boost::noncopyable>("SeralizedSharedMemoryConsumer",init<std::string>())
.def("set_callable", &SeralizedPythonSharedMemoryConsumer::set_callable)
.def("start",&SeralizedPythonSharedMemoryConsumer::start)
.def("wait",&SeralizedPythonSharedMemoryConsumer::wait);
class_<ImagePythonSharedMemoryConsumer, boost::noncopyable>("ImageSharedMemoryConsumer",init<std::string>())
.def("set_callable", &ImagePythonSharedMemoryConsumer::set_callable)
.def("start",&ImagePythonSharedMemoryConsumer::start)
.def("wait",&ImagePythonSharedMemoryConsumer::wait);
class_<SeralizedSharedMemoryProducer, boost::noncopyable>("SeralizedSharedMemoryProducer",init<std::string>())
.def("write_data", &SeralizedSharedMemoryProducer::write_data);
class_<ImageSharedMemoryProducer, boost::noncopyable>("ImageSharedMemoryProducer",init<std::string>())
.def("write_data", &ImageSharedMemoryProducer::write_data);
}

view raw
PythonModule.cpp
hosted with ❤ by GitHub

template <size_t BufferSize>
class PythonSharedMemoryConsumer
{
public:
explicit PythonSharedMemoryConsumer(const std::string& memory_name)
: sm_(memory_name) {}
void set_callable(boost::python::object callback)
{
if (!PyCallable_Check(callback.ptr()))
{
PyErr_SetString(PyExc_TypeError, "Callback must be a callable object");
boost::python::throw_error_already_set();
}
sm_.set_callable([=](const InterprocessMessage<BufferSize>& buffer)
{
try
{
callback(boost::python::object(
boost::python::handle<>(
PyBytes_FromStringAndSize(buffer.message,buffer.message_length))));
//callback(buffer.to_string());
}
catch (boost::python::error_already_set)
{
PyErr_Print();
}
});
}
void start()
{
sm_.start();
}
void wait()
{
sm_.wait();
}
private:
ScryUnlimited::SharedMemory::SharedMemoryConsumer<BufferSize> sm_;
};

These two classes combine to expose the files to python and can be used in a python script by just importing the shared library.

.NET Core

With the python portion complete, I needed to expose the shared memory objects to CSharp. This is easy enough to do by hand if you expose the classes to be used by PInvoke. To accomplish this, I only needed three files:

  • NetCoreSharedMemoryProducer.hpp – The .NET Core version of the publisher
  • NetCoreSharedMemoryConsumer.hpp – The .NET Core version of the consumer
  • NetCoreModule.cpp – The source file exposing the interfaces for PInvoke

 

extern "C" ImageNetCoreSharedMemoryConsumer* GetImageConsumer(char * name)
{
return new ImageNetCoreSharedMemoryConsumer(std::string(name));
}
extern "C" void DeleteImageNetCoreSharedMemoryConsumer(ImageNetCoreSharedMemoryConsumer* consumer)
{
delete consumer;
}
extern "C" void ImageConsumerSetCallback(ImageNetCoreSharedMemoryConsumer* consumer,
void(*callback)(const unsigned char *,size_t))
{
consumer->set_callback(callback);
}
extern "C" void StartImageConsumer(ImageNetCoreSharedMemoryConsumer* consumer)
{
consumer->start();
}
extern "C" void ImageWait(ImageNetCoreSharedMemoryConsumer* consumer)
{
consumer->wait();
}
extern "C" SeralizedNetCoreSharedMemoryConsumer * GetSerializedConsumer(const char * name)
{
return new SeralizedNetCoreSharedMemoryConsumer(name);
}
extern "C" void DeleteSeralizedNetCoreSharedMemoryConsumer(SeralizedNetCoreSharedMemoryConsumer* consumer)
{
delete consumer;
}
extern "C" void SeralizedConsumerSetCallback(SeralizedNetCoreSharedMemoryConsumer* consumer,
void(*callback)(const unsigned char *,size_t))
{
consumer->set_callback(callback);
}
extern "C" void StartSeralizedConsumer(SeralizedNetCoreSharedMemoryConsumer* consumer)
{
consumer->start();
}
extern "C" void SeralizedWait(SeralizedNetCoreSharedMemoryConsumer* consumer)
{
consumer->wait();
}
extern "C" ImageNetCoreSharedMemoryProducer * GetImageProducer(const char * name)
{
return new ImageNetCoreSharedMemoryProducer(name);
}
extern "C" void DeleteImageProducer(ImageNetCoreSharedMemoryProducer * producer)
{
delete producer;
}
extern "C" void SendImageData(ImageNetCoreSharedMemoryProducer * producer,void * data_ptr,size_t size)
{
producer->write_data(std::string(static_cast<char*>(data_ptr),size));
}
extern "C" SeralizedNetCoreSharedMemoryProducer * GetSeralizedProducer(const char * name)
{
return new SeralizedNetCoreSharedMemoryProducer(name);
}
extern "C" void DeleteSeralizedProducer(SeralizedNetCoreSharedMemoryProducer * producer)
{
delete producer;
}
extern "C" void SendSeralizedData(SeralizedNetCoreSharedMemoryProducer * producer,void * data_ptr,size_t size)
{
producer->write_data(std::string(static_cast<char*>(data_ptr),size));
}

view raw
NetCoreModule.cpp
hosted with ❤ by GitHub

template <size_t BufferSize>
class NetCoreSharedMemoryConsumer
{
public:
explicit NetCoreSharedMemoryConsumer(const std::string& name)
: consumer_(name){}
void set_callback(void (*callback_method)(const unsigned char *,size_t))
{
consumer_.set_callable([=](const InterprocessMessage<BufferSize>& message)
{
callback_method(reinterpret_cast<const unsigned char *>(message.message),message.message_length);
});
}
void start()
{
consumer_.start();
}
void wait()
{
consumer_.wait();
}
private:
SharedMemoryConsumer<BufferSize> consumer_;
};

template <size_t BufferSize>
class NetCoreSharedMemoryProducer
{
public:
explicit NetCoreSharedMemoryProducer(const std::string& name)
: producer_(name){}
void write_data(const std::string& data)
{
producer_.write_data(data);
}
private:
SharedMemoryProducer<BufferSize> producer_;
};

extern "C" ImageNetCoreSharedMemoryConsumer* GetImageConsumer(char * name)
{
return new ImageNetCoreSharedMemoryConsumer(std::string(name));
}
extern "C" void DeleteImageNetCoreSharedMemoryConsumer(ImageNetCoreSharedMemoryConsumer* consumer)
{
delete consumer;
}
extern "C" void ImageConsumerSetCallback(ImageNetCoreSharedMemoryConsumer* consumer,
void(*callback)(const unsigned char *,size_t))
{
consumer->set_callback(callback);
}
extern "C" void StartImageConsumer(ImageNetCoreSharedMemoryConsumer* consumer)
{
consumer->start();
}
extern "C" void ImageWait(ImageNetCoreSharedMemoryConsumer* consumer)
{
consumer->wait();
}
extern "C" SeralizedNetCoreSharedMemoryConsumer * GetSerializedConsumer(const char * name)
{
return new SeralizedNetCoreSharedMemoryConsumer(name);
}
extern "C" void DeleteSeralizedNetCoreSharedMemoryConsumer(SeralizedNetCoreSharedMemoryConsumer* consumer)
{
delete consumer;
}
extern "C" void SeralizedConsumerSetCallback(SeralizedNetCoreSharedMemoryConsumer* consumer,
void(*callback)(const unsigned char *,size_t))
{
consumer->set_callback(callback);
}
extern "C" void StartSeralizedConsumer(SeralizedNetCoreSharedMemoryConsumer* consumer)
{
consumer->start();
}
extern "C" void SeralizedWait(SeralizedNetCoreSharedMemoryConsumer* consumer)
{
consumer->wait();
}
extern "C" ImageNetCoreSharedMemoryProducer * GetImageProducer(const char * name)
{
return new ImageNetCoreSharedMemoryProducer(name);
}
extern "C" void DeleteImageProducer(ImageNetCoreSharedMemoryProducer * producer)
{
delete producer;
}
extern "C" void SendImageData(ImageNetCoreSharedMemoryProducer * producer,void * data_ptr,size_t size)
{
producer->write_data(std::string(static_cast<char*>(data_ptr),size));
}
extern "C" SeralizedNetCoreSharedMemoryProducer * GetSeralizedProducer(const char * name)
{
return new SeralizedNetCoreSharedMemoryProducer(name);
}
extern "C" void DeleteSeralizedProducer(SeralizedNetCoreSharedMemoryProducer * producer)
{
delete producer;
}
extern "C" void SendSeralizedData(SeralizedNetCoreSharedMemoryProducer * producer,void * data_ptr,size_t size)
{
producer->write_data(std::string(static_cast<char*>(data_ptr),size));
}

view raw
NetCoreModule.cpp
hosted with ❤ by GitHub

template <size_t BufferSize>
class NetCoreSharedMemoryConsumer
{
public:
explicit NetCoreSharedMemoryConsumer(const std::string& name)
: consumer_(name){}
void set_callback(void (*callback_method)(const unsigned char *,size_t))
{
consumer_.set_callable([=](const InterprocessMessage<BufferSize>& message)
{
callback_method(reinterpret_cast<const unsigned char *>(message.message),message.message_length);
});
}
void start()
{
consumer_.start();
}
void wait()
{
consumer_.wait();
}
private:
SharedMemoryConsumer<BufferSize> consumer_;
};

template <size_t BufferSize>
class NetCoreSharedMemoryProducer
{
public:
explicit NetCoreSharedMemoryProducer(const std::string& name)
: producer_(name){}
void write_data(const std::string& data)
{
producer_.write_data(data);
}
private:
SharedMemoryProducer<BufferSize> producer_;
};

Now we need to call that code from C# using PInvoke Interop

class ImageSharedMemoryConsumer
: ISharedMemoryConsumer
{
[DllImport("ScryUnlimitedCommunicator.so")]
private static extern IntPtr GetImageConsumer(string name);
[DllImport("ScryUnlimitedCommunicator.so")]
private static extern void DeleteImageNetCoreSharedMemoryConsumer(IntPtr instance);
[DllImport("ScryUnlimitedCommunicator.so")]
private static extern void ImageConsumerSetCallback(IntPtr instance, IntPtr callback);
[DllImport("ScryUnlimitedCommunicator.so")]
private static extern void StartImageConsumer(IntPtr instance);
[DllImport("ScryUnlimitedCommunicator.so")]
private static extern void SeralizedWait(IntPtr instance);
private readonly IntPtr _nativePointer;
private IntPtr _callbackPtr;
private GCHandle _callbackHandle;
public ImageSharedMemoryConsumer(string name)
{
_nativePointer = GetImageConsumer(name);
}
~ImageSharedMemoryConsumer()
{
DeleteImageNetCoreSharedMemoryConsumer(_nativePointer);
}
public long Size()
{
return 3000000;
}
public void SetCallback(HandleMessage handler)
{
void Callback(IntPtr x, int y)
{
byte[] managedArray = new byte[y];
Marshal.Copy(x, managedArray, 0, y);
handler(managedArray);
}
_callbackPtr = Marshal.GetFunctionPointerForDelegate(new InternalHandleMessage(Callback));
_callbackHandle = GCHandle.Alloc((Action<IntPtr, int>) Callback);
ImageConsumerSetCallback(_nativePointer,_callbackPtr);
}
public void Start()
{
StartImageConsumer(_nativePointer);
}
public void Wait()
{
SeralizedWait(_nativePointer);
}
}

class ImageSharedMemoryProducer
: ISharedMemoryProducer
{
[DllImport("ScryUnlimitedCommunicator.so")]
private static extern IntPtr GetImageProducer(string name);
[DllImport("ScryUnlimitedCommunicator.so")]
private static extern void DeleteImageProducer(IntPtr handle);
[DllImport("ScryUnlimitedCommunicator.so")]
private static extern void SendImageData(IntPtr handle, byte[] data,long length);
private IntPtr _nativeHandle;
public ImageSharedMemoryProducer(string name)
{
_nativeHandle = GetImageProducer(name);
}
~ImageSharedMemoryProducer()
{
DeleteImageProducer(_nativeHandle);
}
public long Size()
{
return 3000000;
}
public void Write(byte[] message)
{
SendImageData(_nativeHandle,message,message.LongLength);
}
}

using System;
public delegate void HandleMessage(byte[] message);
internal delegate void InternalHandleMessage(IntPtr array, int length);
public interface ISharedMemoryConsumer
{
long Size();
void SetCallback(HandleMessage handler);
void Start();
void Wait();
}

public interface ISharedMemoryProducer
{
long Size();
void Write(byte[] message);
}

 

Azure IoT Edge – YOLO, Stream Analytics Service, and Blob Storage

As a continuation of the Izon camera hack //TODO: link to previous article, I wanted to detect if my dog was using the doggy door in the main room. The approach was going to be simple at first, detect the dog in the room and not in the room. When in the room changes (or not in the room), upload 10 seconds worth of images to Azure to see if the dog used the door.

Image Classification

To detect the dog, the first and largest challenge to these types of tasks is getting enough images to train the model. For me, this meant saving images of the dog in a pre-aligned shot. This is easy enough to accomplish; the room the images will be processed in should only have the dog moving in it. Since he is the only moving object, YOLO can be used to detect the position of objects in the room and then the position of these objects can be checked to see if there is any movement. If there is any movement, the images can be saved for later categorization. To accomplish this, there will be four modules:

  • Camera Module – Accesses the camera feeds to save the images
  • Object Detection Module – Uses YOLO to detect object and object positions
  • Motion Detection Module – Uses Stream Analytics Service to detect if object positions are moving.
  • Image Storage Module – Uses Blob Storage so save and delete the images

Module Arch

The Camera module will send the timestamped images to the Object Detection Module and the Image Storage Module. The Object Detection Module will then use YOLO to detect the objects and their positions in the image. Those detection results will be sent to the Motion Detection Module, which will use Streaming Analytics Service to see if there was motion detected over the last ten seconds. If there is no motion detected over the last ten seconds, then the Motion Detection Module will send a delete command to the Image Storage Module to remove the image without motion from the store. The routing Table will look as so:

"routes": {
"cameraToObjectDetection": "FROM /messages/modules/camera/outputs/imageOutput INTO BrokeredEndpoint(\"/modules/objectDetection/inputs/incomingImages\")",
"cameraToImageStorage": "FROM /messages/modules/camera/outputs/imageOutput INTO BrokeredEndpoint(\"/modules/imageStorage/inputs/incomingImages\")",
"objectDetectionToMotionDetection": "FROM /messages/modules/objectDetection/outputs/objectDetectionOutput INTO BrokeredEndpoint(\"/modules/motionDetection/inputs/incomingObjectDetection\")",
"motionDetectionToDeleteImage": "FROM /messages/modules/motionDetection/outputs/motionDetectionOutput INTO BrokeredEndpoint(\"/modules/imageStorage/inputs/deleteImages\")"
}

view raw
routes.json
hosted with ❤ by GitHub

These modules will be broken up into their own articles for readability and searchability. If there is no link to a module article it is because that article is not completed or is not published yet.

Securing SSH in Azure

On a recent project I inherited an Azure IaaS setup that managed Linux VMs by connecting via SSH from public IPs. I figured while we did a vNet migration we might as well secure the SSH pipeline.

Disable SSH Arcfour and CBC Ciphers

Arcfour is compatible with RC4 encryption and has issues with weak keys, which should be avoided. See RFC 4353 for more information here.

The SSH server located on the remote host also allows cipher block chaining (CBC) ciphers to be used to establish a Secure Shell (SSH) connection, leaving encrypted content vulnerable to a plaintext recovery attack. SSH is a cryptographic network protocol that allows encrypted connections between machines to be established. These connections can be used for remote login by an end user, or to encrypt network services. SSH leverages various encryption algorithms to make these connections, including ciphers that employ cipher block chaining.

The plaintext recovery attack can return up to thirty two bits of plaintext with a probability of 2-18 or fourteen bits of plain text with a probability of 2-14. This exposure is caused by the way CBC ciphers verify the message authentication code (MAC) for a block. Each block’s MAC is created by a combination of an unencrypted sequence number and an encrypted section containing the packet length, padding length, payload, and padding. With the length of the message encrypted the receiver of the packet needs to decrypt the first block of the message in order to obtain the length of the message to know how much data to read. As the location of the message length is static among all messages, the first four bytes will always be decrypted by a recipient. An attacker can take advantage of this by submitting an encrypted block, one byte at a time, directly to a waiting recipient. The recipient will automatically decrypt the first four bytes received as it the length is required to process the message’s MAC. Bytes controlled by an attacker can then be submitted until a MAC error is encountered, which will close the connection. Note as this attack will lead to the SSH connection to be closed, iterative attacks of this nature will be difficult to carry out against a target system.

Establishing an SSH connection using CBC mode ciphers can result in the exposure of plaintext messages, which are derived from an encrypted SSH connection. Depending on the data being transmitted, an attacker may be able to recover session identifiers, passwords, and any other data passed between the client and server.

Disable Arcfour ciphers in the SSH configuration. These ciphers are now disabled by default in some OpenSSH installations. All CBC mode ciphers should also be disabled on the target SSH server. In the place of CBC, SSH connections should be created using ciphers that utilize CTR (Counter) mode or GCM (Galois/Counter Mode), which are resistant to the plaintext recovery attack.

Disable SSH Weak MAC Algorithms

The SSH server is configured to allow cipher suites that include weak message authentication code (“MAC”) algorithms. Examples of weak MAC algorithms include MD5 and other known-weak hashes, and/or the use of 96-bit or shorter keys. The SSH protocol uses a MAC to ensure message integrity by hashing the encrypted message, and then sending both the message and the output of the MAC hash function to the recipient. The recipient then generates their hash of the message and related content and compares it to the received hash value. If the values match, there is a reasonable guarantee that the message is received “as is” and has not been tampered with in transit.

If the SSH server is configured to accept weak or otherwise vulnerable MAC algorithms, an attacker may be able to crack them in a reasonable timeframe. This has two potential effects:

  • The attacker may figure out the shared secret between the client and the server thereby allowing them to read sensitive data being exchanged. 
  • The attacker may be able to tamper with the data in-transit by injecting their own packets or modifying existing packet data sent within the SSH stream.

Disable all 96-bit HMAC algorithms, MD5-based HMAC algorithms, and all CBC mode ciphers configured for SSH on the server. The sshd_config file should only contain the following options as far as supported MAC algorithms are concerned:

  • hmac-sha2-512
  • hmac-sha2-512-etm@openssh.com
  • hmac-sha2-256
  • hmac-sha2-256-etm@openssh.com
  • hmac-ripemd160-etm@openssh.com
  • umac-128-etm@openssh.com
  • umac-128-etm@openssh.com
  • hmac-ripemd160
  • umac-128@openssh.com
  • umac-128@openssh.com 

In addition, all CBC mode ciphers should be replaced with their CTR mode counterparts.

Testing

To test, run the following command:

nmap -sS -sV -p 22 –script ssh2-enum-algos [TARGET IP]

Upcoming Pluralsight Course – Designing an Intelligent Edge in Microsoft Azure

Off to start another course for Pluralsight. This time its Designing an Intelligent Edge in Microsoft Azure. If you would like to check out any of my other courses, visit my author’s profile. The new course will cover the following topics:

  • Edge –
    • Scenarios
    • Concerns
    • Architecture
  • Azure AI Pipelines – Overview with edge
  • Edge Pipelines –
    • Azure Stack
    • Azure Databox Edge
    • Azure IoT Edge
  • Cognitive Services – Overview with Edge
  • Azure Databricks – Overview
  • Azure Machine Learning VMs
  • Project Brainwave