Haiku Tech Center

 Today's presentation is a podcast discussion with Kenneth Stanley on the concept of Novelty Search.  Kenneth's research was discussed in depth in this weeks HTC Seminar.  This is a followup that presents the idea of novelty search in an easy to understand way.

 This is a lecture by Andriy Mnih that explores latent variable models for use in deep learning generative models.  This is a part of DeepMind's Deep Learning Lectures series. Let's check it out.

 This is a podcast discussion with Kenneth Stanley on Open-Endedness.  As we continue our 'hammer it home' theme this week, diving into the research of Kenneth Stanley on diversity search and neuroevolution.

 Today's presentation is mainly a conversation between the Machine Learning Street Talk gang and Lena Voita.  She discusses NLP, and her new online NLP course. The beginning of the video is actually very interesting, but a completely different topic.  It's a followup to yesterday's discussion with Kenneth Stanley.

 Today's HTC seminar presentation is rather long, but worth digesting in it's entirety.  Alternatively titled 'Why Greatness Can't be Planned', it's both a conversation between the Machine Learning Street Talk gang and Kenneth Stanley, as well as a very in-depth exploration of Kenneth's research  in neuroevolution. The concept of 'novelty search with no clearly defined objective' is explored in detail.  Another way of saying this is that following an objective may be an impediment to achieving it.  Kenneth thinks that society's focus on objective and improving benchmarks is both counter productive as well as dangerous. If all you care about is optimization algorithms and deep learning, Kenneth's viewpoints should get you thinking as well. Let's dive in. Here's a link to the NEAT algorithm paper. Here's a link to the POET paper. Here's a link to the enhanced Poet paper. Here's a link to the Pic Breeder paper.   Kenneth

 We're continuing our exploration of DeepMind's 'Deep Learning Lectures' today with Viorica Patraucean's presentation on 'Advanced Models for Computer Vision'.   She will be discussing fun things like semantic segmentation, optical flow, action recognition, and tracking.

 This is a great lecture by Sander Dieleman of DeepMind on Convolutional Neural Networks for Image Recognition.  He first explains how CNN's work, then dives into a very detailed analysis of the CNN image recognition various architectures of the last 10 years.   So we'll explore AlexNet, VGG, ResNet in detail, with an explanation of all of their modular components. Let's check it out.

 Generative design strategies are on my mind alot these days.  These presentations are from a meetup on generative design in the Holland in 2018. The first presentation is by  Patrik Hubner. The first half is really an intro to why you might want to use generative design as well as his input-output creative strategy, and then he dives into a specific example in the second half from his commercial portifolio. This next presentation is by Tim Rodenbroker. This next presentation is by Vera van de Seyp. Includes two generative design AI examples (GAN).

 This is a talk by Douglas Eck of Google's Magenta project on some efforts to train generative deep learning models for creating art and music.  This is from the November 2017 Machine Learning Conference in San Francisco. You will note the ubiquitous auto-encoder architecture rears it's head through out this talk.

 Today's HTC Seminar is a talk by Demis Hassabis of DeepMind on Creativity and AI.  It was presented at the Royal Academy of the Arts in September 2018, and is a part of the Rothschild Foundation Lectures.

 We recently featured Francois Chollet in our HTC Seminar Series presenting his super awesome NeurlPS paper titled 'Abstraction an Reasoning in AI Systems: Modern Perspectives'.   This conversation with Lex Fridman and Francois Chollet on Measures of Intelligence is a great followup.  Topics covered include the definition of intelligence, GPY-3, the semantic web, autonomous intelligence tests of intelligence, and the meaning of life.   Let's dive in.

 So the big news in deep learning AI this last week was the announcement of OpenAI's DALL-E and the associated companion work on the CLIP algorithm.  We already have one post on DALL-E, which is a generative model architecture for creating an image from a textural description. CLIP is a deep learning model with a contrastive objective function that generates a textural description of what is in an image.That is pretty slick in itself.  But the resulting model can be turned into an arbitrary zero-shot classifier for new tasks.  It's like transfer learning, but slightly different. Yannic Kilcher gives us the lowdown on the CLIP algorithm. Let's check it out. Here's a link to the 'Learning Transferable Visual Models from Natural Language Supervision' paper. Here's a link to the PyTorch CLIP code.

 Now you may have noticed that the auto-encoder architecture is having a resurgence recently as an alternative to the GAN architecture for constructing a generative model.   Take a look at yesterday's post for one example.  Today's post continues the trend by taking a look at the Adversarial Latent Autoencoder (basic architecture shown below). The ALAE architecture is a modification of original GAN by decomposing the Generator (G)(\textbf{G}) ( G )  and Discriminator (D)(\textbf{D}) ( D )  networks into two networks such that:  G\textbf{G} G  =  G∘F\textit{G} \circ \textit{F} G ∘ F  and  D\textbf{D} D  =  D∘E\textit{D} \circ \textit{E} D ∘ E . The architecture is shown in Figure 3. It's assumed that the latent space between both the decomposed networks is same and is denoted as  W\mathcal{W} W . Below is a modification of ALAE to get StyleALAE There are two components of StyleALAE: The generator of ALAE is replaced with the generator of StyleGAN as shown in the right side o

 There's a new paper out of Berkley that proposes something called a 'Swapping Autoencoder' for neural net image manipulation.  This research is funded by Adobe, and they are specifically looking at alternatives to GAN's for image manipulation.   You will note the global vs local split in the algorithm and discussion of it, which can also be thought of as 'content' vs 'style'. So this is a paradigm we have seen over and over again in things like 'style transfer'. The technique is fully unsupervised, so no class labeling is required. They are using a patch based method, which is something we have seen in a lot of the recent Berkley work. Here's an overview video someone made that helps explain what is going on. Here's a link to the paper. Here's another short video from the project page showing how they are approaching adding this to a 'Photoshop style' interface for artists to use in their work. Here's a link to some Py

 We were recently introduced to the  Einsum notation in PyTorch in a recent post on coding up Transformers in PyTorch. And i believe i ran into it in one of Alfredo's practicum lectures in the NYU Deep Learning 2020 lectures as well (the transformer one). And it seems pretty cool.  So let's dive in. Here's a link to the blog post he mentioned that gave him Einsum notation inspiration. And of course there's the official PyTorch documentation here . What i love about PyTorch code is that it can be extremely elegant.  That's a fancy way of saying real easy to understand when you look at it. And it appears that the Einsum function is indeed a 'swiss army knife for all kinds of tensor operations'.

 This week's HTC Seminar is a presentation by Yann YeCun at ICLR 2020 on energy based models titled 'The Future is Self-Supervised'. Here's an ICLR  link to the slides and talk.  Which they seem to make very hard to view on youtube directly . I was first exposed to Yann's 'generative model theory of everything' in the NYU Deep Learning 2020 course lectures.  We'll be presenting some of those lectures in our Generative Model Deep Dive post series.  And it's definitely a mind expanding moment when what he is saying really starts to sink in. With that in mind, let's check out the reactions from the gang at Machine Learning Street Talk, as they react to and analyze Yann's ICLR presentation. Observations 1:  The self-supervised learning revolution continues.  And here is one of the big movers of that viewpoint explaining why we should care about it.  If that's not already obvious from all of the various papers presented on it over the last ye

This is an interview with Geoff Hinton in which he discusses his latest research.  He specifically discusses capsule networks, self-supervised learning, and relationships to Transformer architectures.   Let's check it out.

 Let's continue with the final lesson 6 in our HTC Deep Learning with PyTorch Basics course.  And the team at Jovean continue their home run streak from last week with another really great information packed lecture. In this lesson you will lean how a GAN works, how to construct one from scratch in PyTorch, and how to then train your constructed GAN.  This is the clearest explanation of how to actually build a working GAN i've seen so far, so i think you will find it extremely useful. The lesson then continues with an explanation of Transfer Learning, which you will also build from scratch based on a RES-Net 49 model.  This part is heavily based on some fastai material you will have seen before if you took our other HTC deep learning course. Let's get started. You can access the jovian course page associated with this lecture here . What was covered in this lecture generative modeling generative adversarial networks   generator - discriminator building a discriminator netw

In this video, Yannic runs us through a new paper currently under review called 'An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale'.  In the process, he schools us on what is really going on inside of the Transformer architecture.   That alone is worth watching this.  Like most things, it's a lot easier to understand once you see what is really going on.

 The state of the art in deep learning and AI is always an ever moving, ever accelerating target.  So things change, and you need to be aware of them and adapt.  It seems obvious that everyone in 2021 needs to bone up on Transformer architectures for Deep Learning.  So there's your new year's resolution. And to start off an apparently endless new series of posts, let's dive right into it.  Let's get to the point.  We'll watch someone code up a Transformer network in PyTorch from scratch.  And we'll learn about some cool new PyTorch calls you may not be familiar with yet, but will be glad you know about afterwards. The 'Attention is all you Need' paper is here . The blog post on Transformers mentioned in the video is here . Observations 1.  Note the computational complexity of the standard Transformer architecture is not the best thing as the size increases.  Some people recently have offered up architecture variations that alleviate this problem.  So the

Here's some up to the minute information on the latest deep learning generative model architecture from OpenAI. It's a transformer architecture based off of GPT-3 that allows the generation of high quality images from a textural description.   We will first turn to a very well put together and extremely timely video tutorial by Yannic Kilcher .  We also have links to the 2 OpenAI blog posts as well.  And of course some HTC observations. So like Yannic says in the video tutorial, some of his commentary is somewhat speculative since there is not a paper to reference for the specific implementation details.  Although i think he did an excellent job of explaining what is presented in the blog posts. Speaking of the blog posts, here they are. 'DALL-E: Creating Images from Text is here . The reference section in this blog post has some good links to other generative 'text to image' arooraches to check out . 'CLIP: Connecting Text and Images' is here . Paper and co

 Today's HTC Seminar is by Yoshua Bengio on 'Priors for System 2 Knowledge Representaation', and was presented at the ICML conference in July 2020. The talk is on Slides Live and can be seen here . Observations 1.  So what is system 1 and system 2 (not to be confused with software 1.0 and software 2.0 metaphors) ? If you ask the all knowing entity 'the googler', you get the following (at least you do today) : System 1  is the brain's automatic, intuitive, and unconscious thinking mode. It requires little energy or attention, but it is often biased prone. ...  System 2  is a slow, controlled, and analytical method of thinking where reason dominates. Unlike  system 1 , it requires energy and attention to think through all the choices. So Yoshua is taking this concept of cognition described by Daniel Kahneman in his book 'Thinking Fast and Slow', and applying it to deep learning. So system 1 is learning the data manifold, system 2 is more like traditional

This is a fascinating interview with Max Welling that is a deep dive into geometric deep learning.  What that means is using deep learning on non-euclidean data. So graph nets would be one example of this.  But in this particular case we're talking about extending this to gauge or SE(30) equivariance. And in fact Max and his students have published papers where they look at how a deep learning network would work on a quantum computer, and then modify it so that that quantum computer network can run on conventional deep learning hardware. Other topics covered include whether protein folding results from deep mind used a SE (3) transformer, GPT-3,  the whole notion of building prior knowledge of your data into a neural net, bias variance tradeoffs associated with that, probabilistic numeric convolutional networks, chaos theory, etc. Definitely brain expanding, so let's check it out. Here are some papers to check out associated with the above discussion Probabilistic Numeric Convo

 After a short holiday break (and software release crunch time, oh joy), let's continue with lesson 5 of our HTC Deep Learning with PyTorch Basic course.  As usual for this course, we'll be starting off with the 5th lecture in the Jovian.ai course called 'Deep Learning with PyTorch: Zero to GANs'.  And what a lecture it is. The best so far in this series (so we look forward with anticipation to lesson 6 and GANs). In this jam packed to the gills lesson, you will learn how to code a state of the art deep learning model from scratch in PyTorch (how far we have come in 5 short lessons).  We'll also learn very useful techniques like data augmentation, regularization, and adding residual layers.  Plus we'll be using the Adam optimizer rather than stochastic gradient descent. Adam uses techniques like momentum and adaptive learning rates for faster training. And we'll learn about weight decay, gradient clipping, and learning rate scheduling. Let's get started

The gang at Machine Learning Street Talk put together a christmas special community edition. It includes some very interesting discussions of recent papers, as well as some overview information about some talks at a recent Montreal AI event (Christof Koch reminds us that neuroscience is hard).  Let's check it out.