To see if I could, I put together a cross communication library for .Net Core and Python applications using Boost.Interprocess, Boost.Python, and Boost.Signals2. The goal was simple, expose the same interface for cross communication to C# and Python. The approach taken was to use the condition example and edit it to expose to the different languages.
Shared Definitions
First I need to create the objects to make the interface. There are four files making up these objects:
- shm_remove.hpp – just a lifecycle object to clear the shared buffer when it is destructed
- TraceQueue.hpp – The shared memory object
- SharedMemoryConsumer.hpp – The subscriber to the shared memory data
- SharedMemoryProducer.hpp – The publisher for the shared memory data
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#include <boost/interprocess/shared_memory_object.hpp> | |
struct shm_remove | |
{ | |
public: | |
explicit shm_remove(const std::string& shm_name) | |
: shm_name_(shm_name) | |
{ | |
// boost::interprocess::shared_memory_object::remove(shm_name.c_str()); | |
} | |
~shm_remove(){ boost::interprocess::shared_memory_object::remove(shm_name_.c_str()); } | |
private: | |
std::string shm_name_; | |
}; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
template <size_t MESSAGE_SIZE> | |
struct InterprocessMessage | |
{ | |
//Items to fill | |
char message[MESSAGE_SIZE]; | |
//Message size | |
size_t message_length; | |
std::string to_string() const | |
{ | |
return std::string(message,message_length); | |
} | |
}; | |
template <size_t MESSAGE_SIZE> | |
struct TraceQueue | |
{ | |
TraceQueue() | |
: message_in(false) | |
{} | |
//Mutex to protect access to the queue | |
boost::interprocess::interprocess_mutex mutex; | |
//Condition to wait when the queue is empty | |
boost::interprocess::interprocess_condition cond_empty; | |
//Condition to wait when the queue is full | |
boost::interprocess::interprocess_condition cond_full; | |
//Message | |
InterprocessMessage<MESSAGE_SIZE> message; | |
//Is there any message | |
bool message_in; | |
//Notify closing shop | |
bool closing_stream = false; | |
//Check to see if memory has been created | |
int name_length; | |
}; |
These objects comprise the core interface of the shared memory provider. Now, the memory providers need to be exposed to multiple languages. There are different ways to do this and I decided to do it by hand. I should point out SWIG is my usual approach to this task, however, in this instance it seemed easy enough to do it by hand.
Boost Python
To expose the python code, I needed to create a few classes to expose the interface definitions to Boost.Python. The two classes are:
- PythonSharedMemoryConsumer.hpp – The python interface for the SharedMemoryConsumer
- PythonModule.cpp – The file that exposes the module to python
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BOOST_PYTHON_MODULE(ScryUnlimitedCommunicator) | |
{ | |
Py_Initialize(); | |
class_<SeralizedPythonSharedMemoryConsumer, boost::noncopyable>("SeralizedSharedMemoryConsumer",init<std::string>()) | |
.def("set_callable", &SeralizedPythonSharedMemoryConsumer::set_callable) | |
.def("start",&SeralizedPythonSharedMemoryConsumer::start) | |
.def("wait",&SeralizedPythonSharedMemoryConsumer::wait); | |
class_<ImagePythonSharedMemoryConsumer, boost::noncopyable>("ImageSharedMemoryConsumer",init<std::string>()) | |
.def("set_callable", &ImagePythonSharedMemoryConsumer::set_callable) | |
.def("start",&ImagePythonSharedMemoryConsumer::start) | |
.def("wait",&ImagePythonSharedMemoryConsumer::wait); | |
class_<SeralizedSharedMemoryProducer, boost::noncopyable>("SeralizedSharedMemoryProducer",init<std::string>()) | |
.def("write_data", &SeralizedSharedMemoryProducer::write_data); | |
class_<ImageSharedMemoryProducer, boost::noncopyable>("ImageSharedMemoryProducer",init<std::string>()) | |
.def("write_data", &ImageSharedMemoryProducer::write_data); | |
} |
These two classes combine to expose the files to python and can be used in a python script by just importing the shared library.
.NET Core
With the python portion complete, I needed to expose the shared memory objects to CSharp. This is easy enough to do by hand if you expose the classes to be used by PInvoke. To accomplish this, I only needed three files:
- NetCoreSharedMemoryProducer.hpp – The .NET Core version of the publisher
- NetCoreSharedMemoryConsumer.hpp – The .NET Core version of the consumer
- NetCoreModule.cpp – The source file exposing the interfaces for PInvoke
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
extern "C" ImageNetCoreSharedMemoryConsumer* GetImageConsumer(char * name) | |
{ | |
return new ImageNetCoreSharedMemoryConsumer(std::string(name)); | |
} | |
extern "C" void DeleteImageNetCoreSharedMemoryConsumer(ImageNetCoreSharedMemoryConsumer* consumer) | |
{ | |
delete consumer; | |
} | |
extern "C" void ImageConsumerSetCallback(ImageNetCoreSharedMemoryConsumer* consumer, | |
void(*callback)(const unsigned char *,size_t)) | |
{ | |
consumer->set_callback(callback); | |
} | |
extern "C" void StartImageConsumer(ImageNetCoreSharedMemoryConsumer* consumer) | |
{ | |
consumer->start(); | |
} | |
extern "C" void ImageWait(ImageNetCoreSharedMemoryConsumer* consumer) | |
{ | |
consumer->wait(); | |
} | |
extern "C" SeralizedNetCoreSharedMemoryConsumer * GetSerializedConsumer(const char * name) | |
{ | |
return new SeralizedNetCoreSharedMemoryConsumer(name); | |
} | |
extern "C" void DeleteSeralizedNetCoreSharedMemoryConsumer(SeralizedNetCoreSharedMemoryConsumer* consumer) | |
{ | |
delete consumer; | |
} | |
extern "C" void SeralizedConsumerSetCallback(SeralizedNetCoreSharedMemoryConsumer* consumer, | |
void(*callback)(const unsigned char *,size_t)) | |
{ | |
consumer->set_callback(callback); | |
} | |
extern "C" void StartSeralizedConsumer(SeralizedNetCoreSharedMemoryConsumer* consumer) | |
{ | |
consumer->start(); | |
} | |
extern "C" void SeralizedWait(SeralizedNetCoreSharedMemoryConsumer* consumer) | |
{ | |
consumer->wait(); | |
} | |
extern "C" ImageNetCoreSharedMemoryProducer * GetImageProducer(const char * name) | |
{ | |
return new ImageNetCoreSharedMemoryProducer(name); | |
} | |
extern "C" void DeleteImageProducer(ImageNetCoreSharedMemoryProducer * producer) | |
{ | |
delete producer; | |
} | |
extern "C" void SendImageData(ImageNetCoreSharedMemoryProducer * producer,void * data_ptr,size_t size) | |
{ | |
producer->write_data(std::string(static_cast<char*>(data_ptr),size)); | |
} | |
extern "C" SeralizedNetCoreSharedMemoryProducer * GetSeralizedProducer(const char * name) | |
{ | |
return new SeralizedNetCoreSharedMemoryProducer(name); | |
} | |
extern "C" void DeleteSeralizedProducer(SeralizedNetCoreSharedMemoryProducer * producer) | |
{ | |
delete producer; | |
} | |
extern "C" void SendSeralizedData(SeralizedNetCoreSharedMemoryProducer * producer,void * data_ptr,size_t size) | |
{ | |
producer->write_data(std::string(static_cast<char*>(data_ptr),size)); | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
extern "C" ImageNetCoreSharedMemoryConsumer* GetImageConsumer(char * name) | |
{ | |
return new ImageNetCoreSharedMemoryConsumer(std::string(name)); | |
} | |
extern "C" void DeleteImageNetCoreSharedMemoryConsumer(ImageNetCoreSharedMemoryConsumer* consumer) | |
{ | |
delete consumer; | |
} | |
extern "C" void ImageConsumerSetCallback(ImageNetCoreSharedMemoryConsumer* consumer, | |
void(*callback)(const unsigned char *,size_t)) | |
{ | |
consumer->set_callback(callback); | |
} | |
extern "C" void StartImageConsumer(ImageNetCoreSharedMemoryConsumer* consumer) | |
{ | |
consumer->start(); | |
} | |
extern "C" void ImageWait(ImageNetCoreSharedMemoryConsumer* consumer) | |
{ | |
consumer->wait(); | |
} | |
extern "C" SeralizedNetCoreSharedMemoryConsumer * GetSerializedConsumer(const char * name) | |
{ | |
return new SeralizedNetCoreSharedMemoryConsumer(name); | |
} | |
extern "C" void DeleteSeralizedNetCoreSharedMemoryConsumer(SeralizedNetCoreSharedMemoryConsumer* consumer) | |
{ | |
delete consumer; | |
} | |
extern "C" void SeralizedConsumerSetCallback(SeralizedNetCoreSharedMemoryConsumer* consumer, | |
void(*callback)(const unsigned char *,size_t)) | |
{ | |
consumer->set_callback(callback); | |
} | |
extern "C" void StartSeralizedConsumer(SeralizedNetCoreSharedMemoryConsumer* consumer) | |
{ | |
consumer->start(); | |
} | |
extern "C" void SeralizedWait(SeralizedNetCoreSharedMemoryConsumer* consumer) | |
{ | |
consumer->wait(); | |
} | |
extern "C" ImageNetCoreSharedMemoryProducer * GetImageProducer(const char * name) | |
{ | |
return new ImageNetCoreSharedMemoryProducer(name); | |
} | |
extern "C" void DeleteImageProducer(ImageNetCoreSharedMemoryProducer * producer) | |
{ | |
delete producer; | |
} | |
extern "C" void SendImageData(ImageNetCoreSharedMemoryProducer * producer,void * data_ptr,size_t size) | |
{ | |
producer->write_data(std::string(static_cast<char*>(data_ptr),size)); | |
} | |
extern "C" SeralizedNetCoreSharedMemoryProducer * GetSeralizedProducer(const char * name) | |
{ | |
return new SeralizedNetCoreSharedMemoryProducer(name); | |
} | |
extern "C" void DeleteSeralizedProducer(SeralizedNetCoreSharedMemoryProducer * producer) | |
{ | |
delete producer; | |
} | |
extern "C" void SendSeralizedData(SeralizedNetCoreSharedMemoryProducer * producer,void * data_ptr,size_t size) | |
{ | |
producer->write_data(std::string(static_cast<char*>(data_ptr),size)); | |
} |
Now we need to call that code from C# using PInvoke Interop
ITSM that using memory-mapped files or just direct localhost/loopback sockets would both be easier to implement and performant??
I don’t have performance numbers between the different communication patterns.
There’s also good ol’ stdio…
https://code.msdn.microsoft.com/windowsdesktop/C-and-Python-interprocess-171378ee