akash5100

Global Attention

Wed, 21 Aug 2024 00:00:00 +0000

To understand how global attention works quickly, take a long sentence, tokenize it, and then compare how the original attention mechanism (using Key-Value-Query, or KQV) works on the tokens versus how it operates with global tokens derived from blocks of those tokens.

Step 1: Original Sentence and Tokenization

Sentence:
“The quick brown fox jumps over the lazy dog near the riverbank in the sunny park.”

Tokenization:

Tokens: ["The", "quick", "brown", "fox", "jumps", "over", "the", "lazy", "dog", "near", "the", "riverbank", "in", "the", "sunny", "park"]

Step 2: KQV Attention Mechanism

KQV Calculation

Input Tokens:

Creating simple RAG

Sat, 22 Jun 2024 00:00:00 +0000

Life update, last week I joined Plane.so as AI engineer let’s see if I can make it up to AI researcher!

This is the paper which introduced RAG. The problem with LLMs is that they “hallucinate” and have fixed world knowledge, bleh… we already know that RAG architecture tries to address it. My main motive to create RAG is to understand vector embeddings more in-depth and how we create it, search it and what happens next if we find those retieved documents?

Interpretable Features from Neural Networks

Mon, 03 Jun 2024 00:00:00 +0000

I watched a podcast of Jensen and Ilya, in which Ilya talked about how multimodality enables neural networks to learn more features than just a single modality. For example, large language models like GPT-4, without vision, can recognize that the color pink is close to red, but they can’t explain why, because they haven’t seen a single pixel. Multimodality combines image and text, both trained in a unified way. And GPT-4 with vision can tell exactly which pixel is red and why. To achieve intelligence smarter than human-level intelligence, we will definitely need multimodality, because our world is very visual, and neural networks can learn a lot from it.

CNNs

Tue, 21 May 2024 00:00:00 +0000

This is a quick skim notes for CS231n Introduction to CNN lecture 7, I used slides from this lecture to create notes and this is NOT an attempt to replicate notes by cs231n, its already the best notes on CNNs out there.

CNNs are similar to ordinary neural networks, they have trainable weights and bias, receives input, bunch of trainable layers followed with non-linearity. Each layer is completely differentiable, means they can learn. At the end an output layer predicting classes or so.

Gradient-Checkpointing

Thu, 16 May 2024 00:00:00 +0000

You got a very deep neural network to train, lets say wide 128-layers. Does it fits in memory? Yes (barely). The activations and gradients in forward and backward pass respectively takes a lot of memory. But you want to train more deep NN. Why? because, we build the compute (stack more layer) ~= win. (scaling hypothesis) / Blessing of scaling.

See the original gradient checkpointing implementation.¹

Here is a visualization of vanilla training:

Case Study on Transformer-based architecture

Thu, 09 May 2024 00:00:00 +0000

work in progress. publishing this but will be updated every few days, as I learn new stuffs. learning > blog

I am learning how the transformer-based architectures got evolved from 2017 (“Attention is all you need”) till today (May 2024). So, I thought why not write a case-study like this (by Andrej Karpathy).

Table of content

How can unsupervised learning work?
GPT (June 2018)
BERT (October 2018)
Transformer-XL (September 2018) & XLNet (June 2019)
GPT-2 (February 2019)
Sparse Transformers (April 2019)

How can unsupervised learning work?

Why it works? Generalization?, hypothetically…

Key-Value caching

Mon, 06 May 2024 00:00:00 +0000

Long story short, Key value caching is used to reduce the inference speed of AR models (autoregressive / decoder-only). It does so by caching the already computed attention scores (K and V) of previous 1– t-1 tokens. While the generated token is t.

Inference refers to the process of using a trained model to make predictions or generate output for new, unseen input data. In other words, it’s the process of applying the model to real-world data to get a result or prediction.

I challenged myself to visualize attentions (nothing special)

Sat, 04 May 2024 00:00:00 +0000

nothing very special as I expected, because it was just a heat map of embeddings, next time, something better for sure. I aimed to create visualization like this research paper or by Andrej Karpathy.

I started with creating vocabulary for my model using BPE (Byte Pair algorithm). Hyperparameter for BPE merges I used is 2000-256, it would create vocab of size 2000. How? recall your learning and solve that mystery. Before running BPE on ‘Elvis Presley’ wiki page (because it was very long, no other reason), I split them into words using the Llama3 tokenizer Regex:

Transformers

Sun, 28 Apr 2024 00:00:00 +0000

Back in 2017, the deep learning folks were trying to achieve state-of-the-art performance in sequence to sequence modeling. They used RNN’s like LSTM and GRU. The best performing model connects the encoder and decoder through an attention mechanism. Where the encoder is used to encode some sequence into rich embeddings and decoder decodes that embedding to different sequence. Example, language translation.

Sequence to sequence modeling and limitations of RNNs
Transformer
Model Architecture
Attention
Feed Forward (Linear transformation)
Input Embeddings (Vocab and Positional)
Regularizations used in the paper
Transformer Architectures
Sources

Sequence to sequence modeling and limitations of RNNs

RNN had the vanishing gradient problem, models like LSTM and GRU emerged to tackle this problem but realized that altho the text may look better than rubbish, its still rubbish. It may look like English, but jumbled words and the sentence has no meaning.

Parameter updates

Fri, 12 Apr 2024 00:00:00 +0000

Once the gradient is computed, it is then used to update the parameters. There are several approaches to perform the update.

First order(SGD), momentum, nesterov momentum
Per-parameter adaptive learning rates (Adagrad, RMSProp)
Second order methods
Sources

First order(SGD), momentum, nesterov momentum

Stochastic gradient descent

This is the vanilla update, the simplest form of update that changes the parameters along the negative gradient direction, as the gradient indicates the direction of increase, whereas we usually wish to minimize a loss. Assuming a vector of parameters x and the gradient dx, the simplest form of the update is as follows.

Understanding loss.backward()

Tue, 02 Apr 2024 00:00:00 +0000

Relying on any library is not good; therefore, this time, removing this black box and understand what’s happening inside. Writing a backward pass manually would be helpful. Backpropagation is not something that works magically or automatically if you are hoping to debug. In other words, it is easy to fall into the trap of abstracting away the learning process, believing that you can simply stack arbitrary layers together and backprop will magically make them work on your data. So let’s look at a few explicit examples where this is not the case in quite unintuitive ways.

Batch Normalization

Sun, 10 Mar 2024 00:00:00 +0000

Strange output of coupling
Source

Understanding why training deep neural networks can be fragile is crucial. Issues like dead neurons or saturation of non-linearity, and the vanishing or exploding gradients have caused problem for deep learning for years. However, there’s a beacon of hope that emerged around 2015: Batch Normalization.

Here is a visualization of problem using graph. Say we have 1000 datapoints and each has 500 embeddings (latent factors).

Dead Neurons

Mon, 04 Mar 2024 00:00:00 +0000

Peeking inside a hidden layer
But how can we decide which number to multiply? (Initialization)
Summary

I implemented some techniques used in a research paper for n-gram language modeling. It is a simple MLP-based trigram language model that takes three characters’ tokens as input, passes them into an embedding layer, then a hidden linear layer, and finally predicts the next character’s token.

Here are the hyperparameters I used:

Gated Recurrent Unit (GRU)

Fri, 16 Feb 2024 00:00:00 +0000

Reset Gate
Update Gate
Memory Cell
Final Memory
Summary
Sources

GRU was first proposed in 2014 (RNN was in 1986, LSTM in 1995). GRU raised the question of whether we need to be that flexible like LSTM to learn the sequence. GRU is less flexible than LSTM but it is good enough for sequence learning.

GRU redesigned the LSTM cell by introducing reset gate, update gate, and new memory cell; therefore, the number of gates were reduced from four to three. It was empirically shown in (Chung et al., 2014) that the performance of LSTM improves by using GRU cells. Later in 2017, the GRU was further simplified by merging the reset and update gates into a forget gate (Heck & Salem, 2017). Nowadays, GRU is the most commonly used LSTM structure.

Long Short-Term Memory (LSTMs)

Wed, 07 Feb 2024 00:00:00 +0000

Regularizing LSTM
- 1. Dropout
- 2. Activation Regularization and Temporal Activation Regularization
Gradient Clipping (good to know)
Summary

LSTM were designed to deal with the issue of exploding or vanishing gradients in RNNs. This is challenging because, neural nets generally struggle to learn long-term dependencies.

LSTM introduces gate mechanism and has 2 hidden state, that allows information to flow unmodified over many timesteps. Each LSTM cell has a set of gates (forget, input, cell/memory and out) that carefully regulate the information into and out of the cell.

Stateful Recurrent Neural Network

Tue, 06 Feb 2024 00:00:00 +0000

We can remove the hardcoded part by replacing it with a loop
- Improving our RNN
First thing first, let’s solve the resetting of hidden state
Creating Multilayer RNN
Exploding and disappearing Activations

If we have data like: (from the example of previous blog)

X – Y

'one', '.', 'two' – '.'

'.', 'three', '.' – 'four'

Where 3 words are used as input to predict 1 word from a vocabulary as an output, we can create a neural network architecture that takes three words as input and returns a prediction of the probability for each possible next word in the vocabulary.

Tokenize & Numericalize

Mon, 05 Feb 2024 00:00:00 +0000

Here are the main steps for language modeling:

Tokenization– Converting Text into list of words (creating the vocab)
Numericalization– Converting each word in vocab to number, by replacing them with their indices (simple!)

The next steps are: Language Model Data Creation (X&Y) and Language Model Creation

Jargon: Token
One element of a list created by the tokenization process. It could be a word, part of a word (a subword), or a single character.

Self-Supervised & Transfer Learning in Language Models

Sat, 03 Feb 2024 00:00:00 +0000

A language model is a model that is trained to guess the next word in a text (having read the ones before). This kind of task is called self-supervised learning.

Jargon: self supervised learning
Training a model using labels that are embedded in the independent variable, rather than requiring external labels.

Self supervised learning is not usually used for the model that is trained directly, but instead is used for pretraining a model used for transfer learning. Self supervised learning is used to train a base model, and that base model is used to train different model for specific task like text classification!¹

Embeddings in sequential Neural Network

Thu, 01 Feb 2024 00:00:00 +0000

Alright who cares about the follow up blog, here is a notion notes to save me. This is the notes for tweaking Random forest on a tabular dataset to squeeze some performance, I tried this on a Kaggle compe and it worked out pretty badly. Score of something like 2.xx. Maybe I Will improve it when time comes. zzz

As we already initialized the embeddings of our users and products for example, We take the result of the embedding lookup and concatenate those activations together. This gives us a matrix that we can then pass through linear layers and nonlinearities in the usual way.

Embeddings in Recommendation Systems

Mon, 15 Jan 2024 00:00:00 +0000

Collaborative Filtering
A Tabular Dataset -> Movie Recommendation System?
Embeddings?
Speed up the calculation of scores
Why sharp curves of learning weights = overfitting
Weight Decay or L2 Regularization
Movie Recommendation System with Embeddings (MovieLens Dataset)
Direction and Distance of embeddings
Otakus are kinda poison to our embeddings

I will write a follow up blog about Regression and Random Forest, today completes the half of the first month of 2024, I learned lots of stuffs that I am interested and participated on a Kaggle compe, predicing Energy consumption and production using solar panels, just to try what I learned and now I am one step closer to understanding the “Attention”. I will continue this journey and still be writing this blog for future me like creating notes, maybe in more readable form.

Cross Entropy in Classification

Fri, 22 Dec 2023 00:00:00 +0000

Hyperparameters
I coded the Titanic dataset in a simple MLP
Regression and Summarizing Loss functions
Source

Multi label classification refers to the problem of identifying the categories of objects in images that may not contain exactly one type of object. So each data can have either single or multiple label(s). Example, a image has car, bicycle, person, tree.

Why we cant use softmax and NLL loss?

Logarithms in Deep Learning

Mon, 11 Dec 2023 00:00:00 +0000

Understanding Softmax
Logarithm
Finding a right Learning Rate - A technique
Unfreezing & Transfer Learning
- Unfreezing?
Discriminative learning rate

To learn the foundation very clearly, I coded MLP, from scratch and trained MNIST dataset. (it was a 3 vs 7 model, a binary classifier). For that I used a Linear function in each neuron, and Relu as activation. and for the final layer I used Sigmoid. I wanted to expand this model from just a binary classifier to multi-class classifier (where each instance belongs to one and only one class) I learned about Softmax activation that can be used in the final layer and then creating a loss function for MNIST model.

Algorithm behind universal function approximator

Sat, 04 Nov 2023 00:00:00 +0000

A artificial neural network can learn (almost) anything, and so its called a universal function approximator. To understand how it works, we need to know function.

Function, let’s say f(x) is just a system of inputs and outputs, a number in, a number out.

x -> f(x) -> y

We give a input x, and it outputs y. We can plot all the functions on a graph, where it gives an output for an input. What is important is, if you know a function you can always calculate the output (y) for a given input (x).

Traffic on blocked ports

Sat, 30 Sep 2023 00:00:00 +0000

While there are many blogs and tutorials available on accessing a Virtual Cloud Network in a VM instance, such as this comprehensive guide(by oracle). Some of the VM instances I’m working with are not vanilla; they come with the distributor’s packet blockers and they block traffics on all the ports.

I worked around Iptables a default firewall for linux, I think. It is used to set up, maintain, and inspect the tables of IP packet filter rules in the Linux kernel. Several different tables may be defined. Each table contains a number of built-in chains and may also contain user-defined chains.

I relearned entropy

Fri, 15 Sep 2023 00:00:00 +0000

I learned about black hole, how are they possible? (just started). On the sunday morning, I started reading a paper about BBC Reith Lecture. It all started with Albert Einstein writing a paper in 1939 claiming that stars could not collaspe under gravity because matter could not be compressed beyond a certain point, many scientist thought the same but an American scientist John Wheeler, who in many ways is the hero of the black hole story. In his work in 1950s and 1960s, emphasized that many stars would eventually collapse and pointed out problems that possibility posed for theoretical physics. Durring most of the life of a normal star, over billons of years, it will support itself against its own gravity by thermal pressure caused by nuclear processes which convert hydrogen into helium.

About

Mon, 01 Jan 0001 00:00:00 +0000

About Akash Verma

akash5100

Global Attention

Step 1: Original Sentence and Tokenization

Step 2: KQV Attention Mechanism

KQV Calculation

Creating simple RAG

Interpretable Features from Neural Networks

CNNs

Gradient-Checkpointing

Case Study on Transformer-based architecture

Table of content

How can unsupervised learning work?

Key-Value caching

I challenged myself to visualize attentions (nothing special)

Transformers

Table of contents

Sequence to sequence modeling and limitations of RNNs

Parameter updates

Table of contents

First order(SGD), momentum, nesterov momentum

Stochastic gradient descent

Understanding loss.backward()

Batch Normalization

Table of contents

Dead Neurons

Table of contents

Gated Recurrent Unit (GRU)

Table of contents

Long Short-Term Memory (LSTMs)

Table of contents

Stateful Recurrent Neural Network

Table of contents

Tokenize & Numericalize

Self-Supervised & Transfer Learning in Language Models

Embeddings in sequential Neural Network

Embeddings in Recommendation Systems

Table of contents

Cross Entropy in Classification

Table of contents

Logarithms in Deep Learning

Table of contents

Algorithm behind universal function approximator

Traffic on blocked ports

I relearned entropy

About