HDF5 is your big training set helper

As we have discussed before here in our ongoing explanation of deep learning system, one of the key things that helped the current deep learning neural net revolution actually move forward was access to very large training sets. Could be images, could be speech samples, could be text corpus, could be what ever you are trying to learn the statistics of with your neural net.

So what is HDF5?

It's a library api designed to work with very large collections of heterogenous data. It's specificlly built around fast I/O processing. It supports n-dimensional data sets. Each element in a dataset could itself be a complex object if you want it to.  It's cross platform. It's a portable format, no vender lock-in. All data and meta data can be stored in one file. The list goes on.

In some sense it's like a custom file system designed for fast access to data sets that can't fit in memory.

But wait, you say, i already have a file system. And it's pretty straightforward to use. Why do i need another one?

Rebecca Stone has a great tutorial that explains why. She also provides a good 'lets use HDF5 with Python' set of code examples to show how to interface to it. We're going to steal from her tutorial a little bit below.

So, let's contrast the storage time for storing 100,000 images associated with a training set. Now you could certainly just write them all out to disk as PNG images. Or, you could use HDF5 and store them the way it works.
The read time graph is also very similar. Hopefully it is now obvious why HDF5 might be of interest to you. Especially if you are interested in working with Google sized training sets for your neural net.

And of course there is a nice Python interface to HDF5.

We'll be more specifically discussing how to work with HDF5 when training deep learning nets using Keras in future posts.

So what's the tradeoff you might be thinking. Surely there is no free lunch. And yes, you are trading off additional disk space for that fast access when using HDF5.


Comments

Post a Comment

Popular posts from this blog

CycleGAN: a GAN architecture for learning unpaired image to image transformations

Image
 Like the Doublemint Twins touting the joys of Doublemint gum, 2 GANs are surely better than 1 GAN.  Especially if we package them together inside of one meta GAN module.  And this is exactly what the CycleGAN architecture does. Have you ever harbored dark secrets of turning a horse into a zebra?  The CycleGAN was developed to do just that. Learn how to turn a horse into a zebra.  And more. Now right away you can notice a difference between the image to image transformation GAN architectures we've been discussing over the last few posts.  Those last few posts described systems that learn from a database of matched input-output image pairs. And if your goal is to turn an edge representation into a nicely filled in continuous tone image, it's easy to build your database of matched input-output image pairs that your GAN system can then learn off of.  Take a continuous tone image (which will be the output of the database pair entry), then run it through an edge detector algorithm. 
Read more

Pix2Pix: a GAN architecture for image to image transformation

Image
 I thought following up yesterday's TraVelGAN post with a Pix2Pix GAN post would be useful to compare what is going on in the 2 architectures.  Two different approaches to the same problem. I stole this Pix2Pix Overview slide below from an excellent deeplearning.ai GAN course (note that they borrowed it from the original paper) because it gives you a good feel for what is going on inside of the Pix2Pix architecture.    Note how the Generator part is very much like an auto-encoder architecture, but rebuilt using the U-Net architecture features (based on skip-connections) that fastai has been discussing in their courses for several years before it became more widely known to the deep learning community at large  (and which originally came from an obscure medical image segmentation paper) . So the Generator in this Pix2Pix GAN is really pretty sophisticated, consisting of a whole image to image auto-encoder network with U-Net skip connections to generate better image quality at highe
Read more

Smart Fabrics

Image
"What makes smart fabrics revolutionary is that they have the ability to do many things that traditional fabrics cannot, including communicate, transform, conduct energy and even grow."  This is a quote from Rebeccah Pailes-Friedman, who is a professor at Pratt with a research focus on developing wearable technology and smart textiles. Enhancing textiles can be viewed through different lens.  Enhancements can be aesthetic, or performance boosting, or durability boosting. 
Read more