Moving Towards Cross Platform Neural Network Deployment
Suppose you are an application developer who develops cross platform applications. So you write your program in a way that allows it to be run on different desktop platforms (mac,windows, linux), and perhaps also targeting mobile platforms (ios, android). And now you want to incorporate neural network models into your cross platform application. How does one do this?
Now if you are just writing a C++ application, Qt provides a great solution for you. Because you can write one code base, and actually target all of the above platforms for deployment of your application. This is great, it works, it isolates you from platform specific distractions, preventing them from isolating your application to just one specific platform, unable to escape it.
And upcoming Qt 6 provides a great way to isolate the platform specific distractions of GPU specifics internal to these various platforms. Platforms which can all be very different as you dive deep into their internal architectures. For example, apple is dropping OpenGL support, focusing on proprietary Metal api instead. OpenGL is getting long in the tooth, so it's evolving into Vulcan. Windows is pushing Direct 3D 11.
So if you tried to write code yourself to target all of them, you will become embedded in a morass of frustration and wasted time. Because they are all so very different internally.
Qt 6 provides an abstraction layer to hide these platform specific GPU details from an application developer. So you can just write your cross platform application with one code base, and then have Qt do all of that grunt work to get everything to run on all of the different internal platform apis.
So my question is, can we come up with something similar for at least deploying trained neural networks? Maybe even doing training as well if we can take advantage of the GPUs on these different platforms.
Think about what a huge win this would be for everyone. Both for application developers, as well as the people who use these cross platform applications.
I would argue that on some level neural network deployment is a mess right now. How can we clean it up?
PyTorch, TorchScript, and JIT
One interesting approach to the problem described above is going on within the development of the PyTorch system.TorchScript is a way to create serializable and optimizable models from PyTorch code. Any TorchScript program can be saved from a Python process and loaded in a process where there is no Python dependency.
The goal being to eventually convert everything into C++ code.