To understand how global attention works quickly, take a long sentence, tokenize it, and then compare how the original attention mechanism (using Key-Value-Query, or KQV) works on the tokens versus how it operates with global tokens derived from blocks of those tokens. Step 1: Original Sentence and Tokenization Sentence: “The quick brown fox jumps over the lazy dog near the riverbank in the sunny park.” Tokenization: Tokens: ["The", "quick", "brown", "fox", "jumps", "over", "the", "lazy", "dog", "near", "the", "riverbank", "in", "the", "sunny", "park"] Step 2: KQV Attention Mechanism KQV Calculation Input Tokens: ["The", "quick", "brown", "fox", "jumps", "over", "the", "lazy", "dog", "near", "the", "riverbank", "in", "the", "sunny", "park"] Key, Query, Value Calculation: Each token is transformed into keys (K), queries (Q), and values (V) using linear transformations. For simplicity, assume each token has a vector representation (embedding). Attention Scores: Compute attention scores for each token against every other token: For example, the attention score for “fox” attending to “jumps” could be calculated as: $$ \text{score}(\text{fox}, \text{jumps}) = Q_{\text{fox}} \cdot K_{\text{jumps}}^T $$ Softmax and Attention Weights: Apply softmax to the scores to get attention weights, which determine how much focus each token gives to others. Weighted Sum: The output for each token is a weighted sum of the values: $$ \text{output}_{\text{fox}} = \sum_{j} \text{softmax}(\text{score}(\text{fox}, \text{token}_j)) \cdot V_{\text{token}_j} $$ ...

Life update, last week I joined Plane.so as AI engineer let’s see if I can make it up to AI researcher! This is the paper which introduced RAG. The problem with LLMs is that they “hallucinate” and have fixed world knowledge, bleh… we already know that RAG architecture tries to address it. My main motive to create RAG is to understand vector embeddings more in-depth and how we create it, search it and what happens next if we find those retieved documents? ...

I watched a podcast of Jensen and Ilya, in which Ilya talked about how multimodality enables neural networks to learn more features than just a single modality. For example, large language models like GPT-4, without vision, can recognize that the color pink is close to red, but they can’t explain why, because they haven’t seen a single pixel. Multimodality combines image and text, both trained in a unified way. And GPT-4 with vision can tell exactly which pixel is red and why. To achieve intelligence smarter than human-level intelligence, we will definitely need multimodality, because our world is very visual, and neural networks can learn a lot from it. ...

This is a quick skim notes for CS231n Introduction to CNN lecture 7, I used slides from this lecture to create notes and this is NOT an attempt to replicate notes by cs231n, its already the best notes on CNNs out there. CNNs are similar to ordinary neural networks, they have trainable weights and bias, receives input, bunch of trainable layers followed with non-linearity. Each layer is completely differentiable, means they can learn. At the end an output layer predicting classes or so. ...

You got a very deep neural network to train, lets say wide 128-layers. Does it fits in memory? Yes (barely). The activations and gradients in forward and backward pass respectively takes a lot of memory. But you want to train more deep NN. Why? because, we build the compute (stack more layer) ~= win. (scaling hypothesis) / Blessing of scaling. See the original gradient checkpointing implementation.1 Here is a visualization of vanilla training: ...

work in progress. publishing this but will be updated every few days, as I learn new stuffs. learning > blog I am learning how the transformer-based architectures got evolved from 2017 (“Attention is all you need”) till today (May 2024). So, I thought why not write a case-study like this (by Andrej Karpathy). Table of content How can unsupervised learning work? GPT (June 2018) BERT (October 2018) Transformer-XL (September 2018) & XLNet (June 2019) GPT-2 (February 2019) Sparse Transformers (April 2019) How can unsupervised learning work? Why it works? Generalization?, hypothetically… ...

Long story short, Key value caching is used to reduce the inference speed of AR models (autoregressive / decoder-only). It does so by caching the already computed attention scores (K and V) of previous 1– t-1 tokens. While the generated token is t. Inference refers to the process of using a trained model to make predictions or generate output for new, unseen input data. In other words, it’s the process of applying the model to real-world data to get a result or prediction. ...

nothing very special as I expected, because it was just a heat map of embeddings, next time, something better for sure. I aimed to create visualization like this research paper or by Andrej Karpathy. I started with creating vocabulary for my model using BPE (Byte Pair algorithm). Hyperparameter for BPE merges I used is 2000-256, it would create vocab of size 2000. How? recall your learning and solve that mystery. Before running BPE on ‘Elvis Presley’ wiki page (because it was very long, no other reason), I split them into words using the Llama3 tokenizer Regex: ...

Back in 2017, the deep learning folks were trying to achieve state-of-the-art performance in sequence to sequence modeling. They used RNN’s like LSTM and GRU. The best performing model connects the encoder and decoder through an attention mechanism. Where the encoder is used to encode some sequence into rich embeddings and decoder decodes that embedding to different sequence. Example, language translation. Sequence to sequence modeling and limitations of RNNs RNN had the vanishing gradient problem, models like LSTM and GRU emerged to tackle this problem but realized that altho the text may look better than rubbish, its still rubbish. It may look like English, but jumbled words and the sentence has no meaning. ...

Once the gradient is computed, it is then used to update the parameters. There are several approaches to perform the update. First order(SGD), momentum, nesterov momentum Stochastic gradient descent This is the vanilla update, the simplest form of update that changes the parameters along the negative gradient direction, as the gradient indicates the direction of increase, whereas we usually wish to minimize a loss. Assuming a vector of parameters x and the gradient dx, the simplest form of the update is as follows. ...

Relying on any library is not good; therefore, this time, removing this black box and understand what’s happening inside. Writing a backward pass manually would be helpful. Backpropagation is not something that works magically or automatically if you are hoping to debug. In other words, it is easy to fall into the trap of abstracting away the learning process, believing that you can simply stack arbitrary layers together and backprop will magically make them work on your data. So let’s look at a few explicit examples where this is not the case in quite unintuitive ways. ...

Understanding why training deep neural networks can be fragile is crucial. Issues like dead neurons or saturation of non-linearity, and the vanishing or exploding gradients have caused problem for deep learning for years. However, there’s a beacon of hope that emerged around 2015: Batch Normalization. Here is a visualization of problem using graph. Say we have 1000 datapoints and each has 500 embeddings (latent factors). D = torch.randn(1000, 500) Y = torch.randn(1000, 1) * 0.1 And we have 10 layers each with 500 * 500 neurons. ...

I implemented some techniques used in a research paper for n-gram language modeling. It is a simple MLP-based trigram language model that takes three characters’ tokens as input, passes them into an embedding layer, then a hidden linear layer, and finally predicts the next character’s token. Here are the hyperparameters I used: n = 3 #trigram model emb_sz = 2 #latent factor n_hidden = 100 vocab_sz = len(chars) # it's 27 Here is the architecture: Source: https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf C = torch.randn(vocab_sz, emb_sz) w1 = torch.randn(n*emb_sz, n_hidden) b1 = torch.randn(n_hidden) w2 = torch.randn(n_hidden, vocab_sz) b2 = torch.randn(vocab_sz) # Setting up gradient requirements parameters = [C, w1, b1, w2, b2] for p in parameters: p.requires_grad = True # --------------------------------------------------- # training epoch = 200_000 bs = 32 # batch sz for i in range(epoch): # mini batch idx = torch.randint(0, xtrain.shape[0], size=(bs,)) xbs, ybs = xtrain[idx], ytrain[idx] # batch # forward pass emb = C[xbs] hpreact = emb.view(emb.shape[0], -1) @ w1 + b1 h = torch.tanh(hpreact) logits = h @ w2 + b2 loss = F.cross_entropy(logits, ybs) perplexity = torch.exp(loss) # backward pass for p in parameters: p.grad = None loss.backward() # update lr = 0.01 if i < 100_000 else 0.001 for p in parameters: p.data -= p.grad.data * lr # track lossi.append(loss.log10().item()) if i%10000 == 0: print(f"epoch: {i:7d}/{epoch:7d} | loss: {loss.item():.4f} | perplexity: {perplexity.item():.4f}") epoch: 0/ 200 | loss: 9.9113 | perplexity: 20157.0859 epoch: 10/ 200 | loss: 4.8961 | perplexity: 133.7642 epoch: 20/ 200 | loss: 3.5018 | perplexity: 33.1759 epoch: 30/ 200 | loss: 3.0638 | perplexity: 21.4093 epoch: 40/ 200 | loss: 2.8681 | perplexity: 17.6032 epoch: 50/ 200 | loss: 2.8150 | perplexity: 16.6932 Firstly, I want to explain why we need to initialize the weights of our model with good values. As observed, the initial loss is very high, and then subsequent losses are very low. The weights we initialized are significantly incorrect, as some are very confidently wrong while others can be overly confident. This necessitates spending time and computational resources to correct them before starting unbiased learning. Thus, improving parameter initialization can help us skip this unnecessary step, which otherwise wastes energy and computation. ...

GRU was first proposed in 2014 (RNN was in 1986, LSTM in 1995). GRU raised the question of whether we need to be that flexible like LSTM to learn the sequence. GRU is less flexible than LSTM but it is good enough for sequence learning. GRU redesigned the LSTM cell by introducing reset gate, update gate, and new memory cell; therefore, the number of gates were reduced from four to three. It was empirically shown in (Chung et al., 2014) that the performance of LSTM improves by using GRU cells. Later in 2017, the GRU was further simplified by merging the reset and update gates into a forget gate (Heck & Salem, 2017). Nowadays, GRU is the most commonly used LSTM structure. ...

LSTM were designed to deal with the issue of exploding or vanishing gradients in RNNs. This is challenging because, neural nets generally struggle to learn long-term dependencies. LSTM introduces gate mechanism and has 2 hidden state, that allows information to flow unmodified over many timesteps. Each LSTM cell has a set of gates (forget, input, cell/memory and out) that carefully regulate the information into and out of the cell. The forget gate modulates what information gets discarded from the cell state. The input gate decides what new information gets stored in the cell state. And the output gate decides what information propagates to the next steps. ...