Haiku Tech Center

 This is a talk given by Mathias Niessner at MIT in May 2020.

 This is a discussion between the R&D Director and the principal Software Engineer at the Qt Company that contrasts C++ widgets vs QML controls for building desktop applications.

 This overview tutorial is from the Qt Virtual tech Conference in 2020.  It provides a good overview of what is going on under the hood in Qt 6, and how that was changed from what was going on in Qt 5.14 and 5.15. Here's a followup from fall 2020 to help out understanding how all of this comes together in Qt 6. Redirecting Qt Quick rendering QQuickRenderControl API QQuickRenderTarget Shaders CMake in build system converts .frag files to .qsb files for inclusion in the application

Interesting paper out of Google that looks at using a neural net to optimize image resizing for images that are going to be processed by a neural net classifier system.   The results might see surprising at first, but remember, the resized images are for the purpose of improving the classification accuracy of the system, not necessarily for human viewing.  You can check out the paper here .

 This is a workshop Google put together last week associated with Google IO. How does the Brain/Mind (perhaps even an artificial one) work at an algorithmic level? While deep learning has produced tremendous technological strides in recent decades, there is an unsettling feeling of a lack of “conceptual” understanding of why it works and to what extent it will work in the current form. The goal of the workshop is to bring together theorists and practitioners to develop an understanding of the right algorithmic view of deep learning, characterizing the class of functions that can be learned, coming up with the right learning architecture that may (provably) learn multiple functions, concepts and remember them over time as humans do, theoretical understanding of language, logic, RL, meta learning and lifelong learning.

This is a Turing Lecture by Stephane Mallat presented at the Alan Turing Institute in 2016. Observations 1:  Learning invariance to deformation. 2:  Classification of stationary textures. 3:  Multi-scale separation. 4: The role of channel connections in building invariance in CNN.

 Fresh from Google IO 2021 is this overview of what is new in TensorFlow. Let's check out a presentation on Modern Keras Design Patterns.

 This is a breakdown of the paper titled "FNet: Mixing Tokens with Fourier Transforms".  They take a Transformer architecture, and swap in a Fourier transform for the Attention layer.   So is it really just all about sparseness not attention? Or is it really all about 'mixing' as Yannic says? We show that Transformer encoder architectures can be massively sped up, with limited accuracy costs, by replacing the self-attention sublayers with simple linear transformations that "mix" input tokens. These linear transformations, along with simple nonlinearities in feed-forward layers, are sufficient to model semantic relationships in several text classification tasks. Perhaps most surprisingly, we find that replacing the self-attention sublayer in a Transformer encoder with a standard, unparameterized Fourier Transform achieves 92% of the accuracy of BERT on the GLUE benchmark, but pre-trains and runs up to seven times faster on GPUs and twice as fast on TPUs . You

 Andrej Karpathy and friends discuss the recent breakthroughs in artificial intelligence (AI).

Qt Developer Days is happening virtually this week. Some really great presentations.  And once again i'm very fascinated by what one can do with QML. Qt Quick 3D is a new QML feature in Qt 6.  This video is a live coding session that shows off how to get started using it. Here's the Qt 5.15 documentation on Qt Quick 3D. Here's a blog post on what is new in Qt 6 with Qt Quick 3. Here's the Qt 6.2 documentation on Qt Quick 3D. The dynamic geometry from C++ feature is pretty cool. Here's some reference documentation on the Qt Quick Scene Graph. Here's some graphics overview documentation .

 Continuing our recent theme of inverse problems and using invertible neural networks to solve them.  This is a Heidelberg AI talk from november 2019 by Ullrich Kothe on 'Analyzing Inverse Problems in Natural Science using Invertible Neural Networks'.

 I was going to put together something on the recent DDPM paper on paperswithcode, but Yannic Kilcher put together a nice analysis of the algorithm, so we'll go with that instead. Check it out. You should check out yesterdays post on normalizing flows, since it is directly related to this paper.

 What are normalizing flows? Ari Seff will fill us in below. Now we're ready for a more in-depth tutorial on Normalizing Flows.  Let's dive in. Observations: 1:  Built on 'invertible' transformations. 2:  Inherently have a useful latent representation. 3:  Straight forward to train? Examples 1:  Glow 2:  DDPM

 This is a talk by Alexandros Dimakis given in April 2020 for a seminar at the Institute for Advanced Study seminar on theoretical machine learning. So he organizes the world of modeling high dimensionality distributions into 3 categories. 1:  Sparsity (so think wavelets). 2:  Conditional Independence  (so think markov chains, factor graphs, bayes nets used in things like channel coding). 3:  Deep Generative Models (so think passing random noise through a learnable differentiable function (like GANS, VAEs, etc).

 Great lecture by Stephane Mallat on 'Deep Generative Networks as Inverse Problems'.  He analyses several approaches to generative modeling, deconstructs them, then reformulates the entire problem as an inverse problem using scattering wavelets. Awesome.  

Interesting John Timmer article in ArsTechnica today on a neural implant that lets a paralyzed person type by imagining writing.  You can check out the article here . Previous approaches that require the person to imagine using a standard keyboard seem questionable since the current keyboard design was thought up to slow down people's typing on mechanical typewriters by making it more difficult.   Although some people protest the story of QWERTY we were all told.

 This is turning into Bolei Zhou interpretable deep neural networks week.  Its all really interesting research.  This 2020 iMLCv tutorial presentation is on analyzing interpretability of a GAN generator network. Observations 1: Pushing latent space points towards a feature boundary is a way to do editing where you move the synthesized image closer to that feature category characteristics. 2:  Semantic hierarchy emerges across layers in generator. What is relationship to scale space representation? 3:  Correcting the GAN artifacts as an attribute boundary adjustment is interesting. 4:  Extending the latent space allows for better reconstruction in the inversion problem.  But it seems to work by over-fitting, so latent space manipulation performance drops. In-Domain Inversion seems to solve this. Here's a github link for the associated papers and code.

 I stumbled upon Bolei Zhou's phd thesis defense on line.  They should do this for all phd thesis defenses. I should point out that anything being done in Freeman and Torralba's group at MIT is worth checking out.

Continuing our running theme this week on interpretability in deep learning neural networks, let's check out a tutorial talk by Chris Olah titled Introduction to Circuits in CNNs.  This is from a workshop at CVPR 2020. If you are have read any of the Distil publications on feature visualization, you will find this talk interesting.  Observations 1:  Complex Gabor filters. In reciprocal pairs. 2:  Circle detector made from curve detectors. 3.  Triangle detector made from line detectors.  Really detecting 'inflection points'. 4:  Absence of color vs color contrast detectors. 5:  Boundary detectors. Meta level higher order thing compared to normal edge detector. High-low frequency boundary, etc. 6: Clean room implementation of curve detector.

 Interesting tutorial talk by Ruth Fong at CVPR 2020 on Understanding Deep neural Networks. So another discussion of interpretability yo follow up yesterdays post.  This is from another Bolei Zhou CVPR workshop on interpretable machine learning computer vision.

 Bolei Zhou runs us through an analysis of feature visualization in deep learning neural networks.  This is from a CVPR workshop in 2018,

Self-Supervised Learning is the final frontier in Representation Learning: Getting useful features without any labels.  Facebook AI's new system, DINO, combines advances in Self-Supervised Learning for Computer Vision with the new Vision Transformer (ViT) architecture and achieves impressive results without any labels. Attention maps can be directly interpreted as segmentation maps, and the obtained representations can be used for image retrieval and zero-shot k-nearest neighbor classifiers (KNNs). You can find the paper here . There is a blog post with more info  here . The PyTorch code can be found  here . Yannic Kilcher will run us through his astute analysis of the system.

As you start diving into the theoritical details, deep learning neural networks are perhaps not so mysterious after all.  It helps to have a good understanding of non-linear function approximation, manifolds, and gauge theory.  The material below will help out.  Great article by Michael Bronstein on the geometric foundations of deep learning can be found here .  We featured Michael's ICLR keynote talk here. It's a summary of a more comprehensive 'proto-book' paper that can be found here . We're going to feature Petar Velickovic's talk on this research here.  He's one of the co-authors on the paper above.