[{"content":"To understand how global attention works quickly, take a long sentence, tokenize it, and then compare how the original attention mechanism (using Key-Value-Query, or KQV) works on the tokens versus how it operates with global tokens derived from blocks of those tokens.\nStep 1: Original Sentence and Tokenization Sentence:\n\u0026ldquo;The quick brown fox jumps over the lazy dog near the riverbank in the sunny park.\u0026rdquo;\nTokenization:\nTokens: [\u0026quot;The\u0026quot;, \u0026quot;quick\u0026quot;, \u0026quot;brown\u0026quot;, \u0026quot;fox\u0026quot;, \u0026quot;jumps\u0026quot;, \u0026quot;over\u0026quot;, \u0026quot;the\u0026quot;, \u0026quot;lazy\u0026quot;, \u0026quot;dog\u0026quot;, \u0026quot;near\u0026quot;, \u0026quot;the\u0026quot;, \u0026quot;riverbank\u0026quot;, \u0026quot;in\u0026quot;, \u0026quot;the\u0026quot;, \u0026quot;sunny\u0026quot;, \u0026quot;park\u0026quot;] Step 2: KQV Attention Mechanism KQV Calculation Input Tokens:\n[\u0026quot;The\u0026quot;, \u0026quot;quick\u0026quot;, \u0026quot;brown\u0026quot;, \u0026quot;fox\u0026quot;, \u0026quot;jumps\u0026quot;, \u0026quot;over\u0026quot;, \u0026quot;the\u0026quot;, \u0026quot;lazy\u0026quot;, \u0026quot;dog\u0026quot;, \u0026quot;near\u0026quot;, \u0026quot;the\u0026quot;, \u0026quot;riverbank\u0026quot;, \u0026quot;in\u0026quot;, \u0026quot;the\u0026quot;, \u0026quot;sunny\u0026quot;, \u0026quot;park\u0026quot;] Key, Query, Value Calculation:\nEach token is transformed into keys (K), queries (Q), and values (V) using linear transformations. For simplicity, assume each token has a vector representation (embedding). Attention Scores:\nCompute attention scores for each token against every other token: For example, the attention score for \u0026ldquo;fox\u0026rdquo; attending to \u0026ldquo;jumps\u0026rdquo; could be calculated as: $$ \\text{score}(\\text{fox}, \\text{jumps}) = Q_{\\text{fox}} \\cdot K_{\\text{jumps}}^T $$ Softmax and Attention Weights:\nApply softmax to the scores to get attention weights, which determine how much focus each token gives to others. Weighted Sum:\nThe output for each token is a weighted sum of the values: $$ \\text{output}_{\\text{fox}} = \\sum_{j} \\text{softmax}(\\text{score}(\\text{fox}, \\text{token}_j)) \\cdot V_{\\text{token}_j} $$ Visualization of Attention (KQV) Each token attends to all other tokens, resulting in a dense attention matrix. For instance, \u0026ldquo;fox\u0026rdquo; might attend strongly to \u0026ldquo;jumps\u0026rdquo; and \u0026ldquo;lazy,\u0026rdquo; while attending less to \u0026ldquo;the\u0026rdquo; or \u0026ldquo;riverbank.\u0026rdquo; Step 3: Breaking into Blocks and Global Tokens Block Creation Let’s break the tokens into blocks of size 5:\nBlocks: Block 1: [\u0026quot;The\u0026quot;, \u0026quot;quick\u0026quot;, \u0026quot;brown\u0026quot;, \u0026quot;fox\u0026quot;, \u0026quot;jumps\u0026quot;] Block 2: [\u0026quot;over\u0026quot;, \u0026quot;the\u0026quot;, \u0026quot;lazy\u0026quot;, \u0026quot;dog\u0026quot;, \u0026quot;near\u0026quot;] Block 3: [\u0026quot;the\u0026quot;, \u0026quot;riverbank\u0026quot;, \u0026quot;in\u0026quot;, \u0026quot;the\u0026quot;, \u0026quot;sunny\u0026quot;, \u0026quot;park\u0026quot;] Global Token Calculation Global Token for Each Block: For Block 1: Average the embeddings of [\u0026quot;The\u0026quot;, \u0026quot;quick\u0026quot;, \u0026quot;brown\u0026quot;, \u0026quot;fox\u0026quot;, \u0026quot;jumps\u0026quot;] For Block 2: Average the embeddings of [\u0026quot;over\u0026quot;, \u0026quot;the\u0026quot;, \u0026quot;lazy\u0026quot;, \u0026quot;dog\u0026quot;, \u0026quot;near\u0026quot;] For Block 3: Average the embeddings of [\u0026quot;the\u0026quot;, \u0026quot;riverbank\u0026quot;, \u0026quot;in\u0026quot;, \u0026quot;the\u0026quot;, \u0026quot;sunny\u0026quot;, \u0026quot;park\u0026quot;] Visualization of Global Attention Now, each block has a global token that summarizes its context:\nGlobal Tokens: Global Token 1: Represents the context of Block 1 (e.g., action of jumping). Global Token 2: Represents the context of Block 2 (e.g., nearby elements). Global Token 3: Represents the context of Block 3 (e.g., the environment). Attention Mechanism with Global Tokens Attention Scores:\nEach token in a block can attend to both its local tokens and the global token of its block. For example, \u0026ldquo;fox\u0026rdquo; in Block 1 can attend to \u0026ldquo;jumps\u0026rdquo; and the Global Token 1. Weighted Sum:\nThe output for \u0026ldquo;fox\u0026rdquo; now considers both its local context and the global context: $$ \\text{output}_{\\text{fox}} = \\text{softmax}(\\text{score}(\\text{fox}, \\text{jumps})) \\cdot V_{\\text{jumps}} + \\text{softmax}(\\text{score}(\\text{fox}, \\text{Global Token 1})) \\cdot V_{\\text{Global Token 1}} $$ Comparison of Attention Mechanisms Original KQV Attention:\nEach token attends to every other token, creating a dense attention matrix. This can capture fine-grained relationships but may be computationally expensive for long sequences. Global Token Attention:\nEach token attends to local tokens and a global token, reducing the complexity. The global token provides a broader context, allowing the model to understand relationships across blocks without needing to attend to every individual token. Conclusion Global attention captures the attention of tokens in sequence as well as the attention of blocks of tokens. this makes the learning better.\nSource LongT5 Paper\n","permalink":"https://akash5100.github.io/posts/2024-08-21-global_attention/","summary":"\u003cp\u003eTo understand how global attention works quickly, take a long sentence, tokenize it, and then compare how the original attention mechanism (using Key-Value-Query, or KQV) works on the tokens versus how it operates with global tokens derived from blocks of those tokens.\u003c/p\u003e\n\u003ch3 id=\"step-1-original-sentence-and-tokenization\"\u003eStep 1: Original Sentence and Tokenization\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eSentence\u003c/strong\u003e:\u003cbr\u003e\n\u003cem\u003e\u0026ldquo;The quick brown fox jumps over the lazy dog near the riverbank in the sunny park.\u0026rdquo;\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTokenization\u003c/strong\u003e:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eTokens: \u003ccode\u003e[\u0026quot;The\u0026quot;, \u0026quot;quick\u0026quot;, \u0026quot;brown\u0026quot;, \u0026quot;fox\u0026quot;, \u0026quot;jumps\u0026quot;, \u0026quot;over\u0026quot;, \u0026quot;the\u0026quot;, \u0026quot;lazy\u0026quot;, \u0026quot;dog\u0026quot;, \u0026quot;near\u0026quot;, \u0026quot;the\u0026quot;, \u0026quot;riverbank\u0026quot;, \u0026quot;in\u0026quot;, \u0026quot;the\u0026quot;, \u0026quot;sunny\u0026quot;, \u0026quot;park\u0026quot;]\u003c/code\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"step-2-kqv-attention-mechanism\"\u003eStep 2: KQV Attention Mechanism\u003c/h3\u003e\n\u003ch4 id=\"kqv-calculation\"\u003eKQV Calculation\u003c/h4\u003e\n\u003col\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eInput Tokens\u003c/strong\u003e:\u003c/p\u003e","title":"Global Attention"},{"content":" Life update, last week I joined Plane.so as AI engineer let\u0026rsquo;s see if I can make it up to AI researcher!\nThis is the paper which introduced RAG. The problem with LLMs is that they \u0026ldquo;hallucinate\u0026rdquo; and have fixed world knowledge, bleh\u0026hellip; we already know that RAG architecture tries to address it. My main motive to create RAG is to understand vector embeddings more in-depth and how we create it, search it and what happens next if we find those retieved documents?\nRAG Model Architecture This paper introduced, this simple architecture.\nGet a encoder model, which encodes anything into vectors Use this encoder to encode Documents (z) Use the same encoder to encode the query (x) find document vectors z having similar to query vector x. concat x and z vectors and feed them to a generator G (generates output) In this paper, the author used BERT base as encoder for query and documents and used BART as generator. Starting to understand how this \u0026ldquo;encoder\u0026rdquo; works.\nVectorDB But first, let\u0026rsquo;s assume we don\u0026rsquo;t know how the embeddings are created :), but we know that embeddings are high-dimensional representation of words/sentences/images ~ our data. For now let\u0026rsquo;s say we have a model D, which embeds your data into high dimensional vectors. i.e.,:\ndata -\u0026gt; D -\u0026gt; vectors # we use this encoder to create embeddings for our documents # we have docs, z z -\u0026gt; D -\u0026gt; Zi # vectors for z z -\u0026gt; D(z) -\u0026gt; Zi We got a query (x) and we want to our LLM to response something related to documents (z) and give output (y).\n# we have query, x x -\u0026gt; Q -\u0026gt; Xi # Q is the query encoder x -\u0026gt; Q(x) -\u0026gt; Xi # The query encoder Q and the document encoder D are based on same model. Now we have query vector and N document vectors.\nSimilarity Search Given a query vector and N candidate vectors, find the most similar one.\nDot product (Inner product) The inner product (or dot product) of two vectors measures the magnitude of their overlap. For vectors q (query vector) and v (candidate vector), the inner product is defined as q.v = sum(qi * vi) (dot product, at i\u0026rsquo;th index). if the result is high, it means vectors are similar in terms of their magnitude and direction. The candidate vector with high values are more similar to the query Suitable for scenarios where the magnitude of the vectors is important Euclidian distance Learned in school right? The distance between two points, in a geometric plane. It measures the straight-line distance between two points in a multidimensional space.\na = [1,2,3] b = [4,5,6] x = sum((a[0]-b[0])**2, (a[1]-b[1])**2, (a[2]-b[2])**2) ed = x**0.5 Cosine Similarity Calculates the cosine of the angle between two vectors, which indicates their directional similarity regardless of their magnitude. Given A and B vectors\nma = sum(x**2 for x in A) # magnitude of a mb = sum(x**2 for x in B) # magnitude of b dp = A.B #dot-product cosine_similarity = dp / ma * mb Cosine similarity is close to 1, vectors are pointing in same direction\nDot product gives a magnitude (MAGNITUDE)\nEuclidian distance gives a straight-line difference (DISTANCE)\ncosine similarity purely focuses on the angle between them and thus their directional similarity (DIRECTION)\nSo for a RAG on wiki pages, what should I use?\nIn a high-dimentional spaces, magnitude can vary I can use cosine similarity, because it It measures the direction and orientation CS normalizes the vectors too! ED is good but it doesn\u0026rsquo;t normalizes vectors, in a high-dimensional embedding spaces, bleh Some details on encoder according to the paper In this paper, the author used BERT base as an encoder and maximum inner product search (MIPS) to search for similar vectors, because it is sub-linear. (other is MCSS - Maximum cosine similarity search)\nAfter retrieving content when generating the BART model: we combine input vector X with Z by simply concatenating them.\nThis is why they choose BART: BART was pre-trained using a denoising objective and a variety of different noising functions. It has achieved state-of-the-art (SOTA) results on a diverse set of generation tasks and outperforms comparable-sized T5 models.\nThis again raises question: If BERT outperformed GPT-2 of same parameter size and BART outperformed BERT, T5 and GPT-2 of same size, can a encoder-decoder model similar to BART of GPT-4 parameter size will outperform GPT-4?\nTraining\nUpdating the document encoder D while training showed no significant improvement + its was slow, so they (authors) decided to freeze the document encoder and only finetuned the query encoder Q and the generator G, which is the BART model.\nFinal small detail before we create our own embedder: They introduced 2 architectures:\nRAG sequence RAG token 1. RAG-Sequence The difference is simple, the sequence model retrieves bunch of docs and generate output from generation individually from the generator G. Say we retrieved n documents, then we will generate n outputs, and finally we reconsider all the n outputs to generate a single output. For this reconsidering all outputs from the previous generations, we use BEAM SEARCH. damn, I need to study what is beam seach (self remainder).\n2. RAG-Token This one is simple, after we retrieved n document we use every documents to generate a single output and each token generated can be based on any of the n document.\nBut this architecture is used now a day (i think) because perplexity uses it :)\nBack to encoder Before this paper, another paper tried to create RAG, they used something called Dense Passage Retrieval (DRP). And DRP is based on BERT. So, they somehow used BERT (LLM) to create encoder which can embed data? what?\nI found out that there is a leaderboard for data embedders called MTEB (Massive Text Embedding Benchmark), the models in this leaderboard are named as Qwen2-instruct, Mistral, Meta-llama, these are LLMs, so we can use LLMs to create embeddings, or rather (just assumtions ahead) instead of predicting the next tokens we can somehow use the hidden layers of the LLMs as Embeddings? or maybe make them give the embeddings in an unsupervised way?\nIt turns out, for creating embeddings from scratch: the skip-gram model (and word2vec in general) is one way to create word embeddings from scratch. This process is often called \u0026ldquo;training word embeddings\u0026rdquo; or \u0026ldquo;learning word representations.\u0026rdquo; Its unsupervised training as expected.\nBag-of-words Model\nIn bag of words, as we know given a context predict the target. BoW dont consider the context in which the words appear, it just considers the frequency.\nSkip-Gram model\nSkip gram is opposite of BoW, given a target predict the context, example for a given sentence: \u0026ldquo;Hello how are you bob?\u0026rdquo;, if the target is \u0026ldquo;bob\u0026rdquo; we might have dataset like (bob, hello), (bob, you) etc.\nWord2Vec uses this two approaches for generating word embeddings. Its word level, for our use case, which is creating embeddings for documents like articles/blog, we might need to go beyond word-level, something like sentence, passage or even document-level embeddings.\nModern techniques do use pre-trained LLMs such as BERT and fine-tune them for generating embeddings or they use average of hidden states to create embeddings.\nFor our problem, I found this paper S-BERT / Sentence BERT, sounds promising for our understanding, because it used BERT to create encoder.\nSentence-BERT todo\nRoadmap to MinRAG: From what I understand, I need a document, query encoder and a generator\nUse S-BERT (small) as encoder (probably) Use GPT-2 (small) as generator setup encoder some how encode few wiki pages and store it? encode query use MCSS and get top-k vectors concat and generate ","permalink":"https://akash5100.github.io/posts/2024-06-22-creating_minrag/","summary":"\u003cblockquote\u003e\n\u003cp\u003eLife update, last week I joined \u003ca href=\"https://plane.so\"\u003ePlane.so\u003c/a\u003e as AI engineer let\u0026rsquo;s see if I can make it up to AI researcher!\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eThis is the \u003ca href=\"https://arxiv.org/abs/2005.11401\"\u003epaper\u003c/a\u003e which introduced RAG. The problem with LLMs is that they \u0026ldquo;hallucinate\u0026rdquo; and have fixed world knowledge, bleh\u0026hellip; we already know that RAG architecture tries to address it. My main motive to create RAG is to understand vector embeddings more in-depth and how we create it, search it and what happens next if we find those retieved documents?\u003c/p\u003e","title":"Creating simple RAG"},{"content":"I watched a podcast of Jensen and Ilya, in which Ilya talked about how multimodality enables neural networks to learn more features than just a single modality. For example, large language models like GPT-4, without vision, can recognize that the color pink is close to red, but they can\u0026rsquo;t explain why, because they haven\u0026rsquo;t seen a single pixel. Multimodality combines image and text, both trained in a unified way. And GPT-4 with vision can tell exactly which pixel is red and why. To achieve intelligence smarter than human-level intelligence, we will definitely need multimodality, because our world is very visual, and neural networks can learn a lot from it.\nI can relate his thoughts from this video \u0026ldquo;Why Humans are intelligent?\u0026rdquo;:\n1) Communication, they can pass their knowledge to future generation whereas cats and dogs can\u0026#39;t. 2) They can interact with the surrounding world/environment. 3) eye sight. So, I shifted my gears from architectural improvements (I will come back to it, but when time comes) to multimodality.\nThe first thing that came to mind was:\n\u0026gt; revise CNNs \u0026gt; created a blog post on it. \u0026gt; There must be something good in fast.ai course that I left unfinished after NLP. \u0026gt; found it, UNet! What is U-Net? U-Net came out before ResNet and was originally focused on medical applications but it has now revolutionized all kinds of generative vision models. This is what I learned. The basic idea is to start with pretrained model (like alexnet or VGG?) and instead of predicting labels, cut the head and add custom head that \u0026ldquo;reconstructs the input image\u0026rdquo;.\nHow can we do that? The convolution layer is used to reduce the special dimension and increase the features (depth)?\nThe basic idea is, once you have a feature map (activations) say 7x7, what if you replace each element of the 7x7 with 2x2 matrix? the result would be 14x14. This process is called nearest neighbour interpolation (NNI). In which every pixel is replaced by a grid of same pixel. Another approach is called transpose convolution. In which, first we pad each of the input pixel with 0s and then use a kernel (3x3 common to use) with stride 1. This results in the same effect. [Check it with this formula (L-K)/S + 1]. Here is a good visulization.\nThe difference between Nearest neighbour interpolation and Transpose convolutions. Nearest neighbour interpolation is fast and no learnable parameters, whereas Transpose kernel has learnable parameters (kernel) but slow.\nI was trying to solve a competetion by comma.ai, in which we have compress a numpy vectors losslessly. And I found out that the SOTA open source self-driving model uses Nearest neighbour interpolation. Whereas, generative ai, where we use transpose convs in generative ai (I assume, haven\u0026rsquo;t looked into GANs, VAEs, diffusion etc. YET).\nThe architecture of U-Net. UNet is an encoder decoder structure, the Encoder encodes the input image to features by using conv-pool-conv-pool. Later the Decoder decodes the activation map using transpose convs (up): up-conv-up-conv. The game changer is the residual connections.\nResidual connections in a nutshell: Instead of learning the full output, the network learns the \u0026ldquo;residuals\u0026rdquo;, or the difference, between the input and the desired output. This is achieved by adding the input to the output, effectively \u0026ldquo;jumping over\u0026rdquo; one or more layers. This allows the network to focus on learning the residual, or the error, rather than the full output.\nIt\u0026rsquo;s easier for the network to learn the residual between the input and output, rather than learning the full output from scratch. (Kaiming He et al. 2016).\nNext, new paper on interpretability of LLMs by anthropic.com\n\u0026gt; my thoughts. They used sparse autoencoder, a kind of dictionary learning to extract the features of their production level model Claude Sonnet. \u0026gt; what the heck is dictionary learning and Sparse autoencoder? \u0026gt; I researched and found a paper/notes \u0026gt; https://web.stanford.edu/class/cs294a/sparseAutoencoder.pdf Dictionary learning\nA technique that involves learning basic functions (features) that can be used to represent input data efficiently.\nDeep learning approach is called autoencoders\nAn autoencoder neural network is an unsupervised learning that applies backpropagation, setting the target equal to the input. In other words, the autoencoder tries to learn a function aims to reconstruct the original input. This function can be then used to visualize features of the input data.\nAs a concrete example, suppose the inputs x are the pixel intensity values from a 10×10 image (100 pixels) so for our first layer the input features is n = 100, and there are 50 hidden units in layer L2. And the output would be 100 same as the input. Since there is only 50 hidden units, the network is forced to learn a compressed representation of the input.\n10x10 -\u0026gt; 1,100 -\u0026gt; 100x50 -\u0026gt; 50x100 -\u0026gt; 1,100 -\u0026gt; 10x10 Let\u0026rsquo;s say, if the input is a completely random noise, then it would be hard and very difficult for this compression task. But if there is a structure in the data, Then this algorithm will be able to discover some of those correlation. In fact, this simple autoencoder often ends up learning a low-dimensional representation very similar to PCA\u0026rsquo;s.\nIf we impose a contraint other than \u0026lsquo;compression\u0026rsquo;, say opposite, greater (perhaps even greater than the number of inputs pixels) we can still discover interesting patterns. This is called the sparsity constraint on the hidden units. This will cause most of the neurons in the large activations to say zero or near zero (in-active) only firing few neurons for special data \u0026ndash; Sparse Autoencoder!!!\nConsider a hidden layer in a neural network where we have a sparse autoencoder. In a typical sparse autoencoder, we want most of the neurons to be inactive (close to zero) and only a few to be active for any given input.\nImagine you have an activation vector A from a hidden layer with 10 neurons for a particular input:\n# say our activations: the result of 1,100 * 100x50 = 1x50 is: a = torch.randn((1,50)) For this vector, suppose we want only 10% of the neurons to be active on average (making 90% close to zero or inactive and 10% close to 1, active). To enforce this sparisity, we need a way to measure how far the input data distrubution is from our output distribution.\nKL divergence does that.\nKL divergence is a measure of how one distribution differs from a another distribution.\nif a is the activations from a single forward pass, x is the weights of autoencoder hidden layer, we calculate the average activation over the training set. where M is the length of dataset. So p^ (rho_hat) is the average activation on the training set. we want the constrain this \"rho_hat\" to something like 0.05. Sparse.\nwe use KL divergence: s2 is the number of neuron in the hidden layer, p (rho) is the sparisity parameter (0.05 what we decided), in other words we would like the average activations of each hidden neurons to have to be close to p \"sparsity parameter\". This is the KL divergence, which tells us how distance is the distribution of activations (rho_hat) from what we want (rho). We can use KL divergence as our criterion (loss function), infact this is very close to cross-entropy loss.\nthe difference between cross-entropy loss and KL divergence is, the cross-entropy loss calculates the distance between predictions and labels. Whereas, KL divergence calculates the difference between two distance distributions.\nTo incorporate the KL-divergence term into the derivative calculation, we only need a small change, in addition to gradient, we also accumulate the average of activations on all training set, before backprop. and now in the backprop we can add the sparsity penalty as well as the weight decay.\n# assuming we have avg_z # average of all activation, basically acts.mean(0) rho_t # sparsity parameter, same size avg_z but \u0026#34;full_like\u0026#34;! # restoration loss r_loss = criterion(_x, x) # where _x is the reconstructed x # kl divergence kl_d = rho_t * torch.log(rho_t/avg_z) + (1 - rho_t) * torch.log((1-rho_t) / (1-avg_z)) # sparsity penalty sp = beta + torch.sum(kl_d) # where \u0026#39;beta\u0026#39; is the strength of penalty loss = r_loss + sp Here is my Sparse autoencoder implementation.\nthere are 2 versions, 2nd is the sparse, 3rd is compressed\nBack to Antropic\u0026rsquo;s paper Antropic team, grabbed a layer\u0026rsquo;s activation of Claude 3 Sonnet from somewhere in \u0026ldquo;middle\u0026rdquo; 1 and trained their sparse autoencoder in it.\nimage source: https://transformer-circuits.pub/2023/monosemantic-features/index.html # implementation encoder = Linear(input_dim, hidden_dim) decoder = Linear(hidden_dim, input_dim) # x is the input encoded = torch.relu(encoder(x)) decoded = decoder(encoded) # loss r_loss = torch.mean((x - x_hat) ** 2) # MSE sp = torch.sum(torch.abs(encoded), 1).mean() loss = r_loss + _lambda + sp Notice this:\nAntropic used L1 penalty directly on acts\nStanford notes used KL divergence to penalize deviations\nAntropic directly used magnitude of activations\nStanford ensured to use averaged activations\nFeatures just skim over these: long story short, LLMs activations if touched every RL saftey breaks. LLMs follow sycophancy, which means the tendency of models to provide responses that match user beliefs or desires rather than truthful ones. The models are not truthful. Next steps? multimodality is the key to intelligence speech recognition and synthesis notes and my notion listed here References every other source is provided directly in the hyperlink. Mapping the Mind of a Large Language Model\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://akash5100.github.io/posts/2024-06-03-interpretable_features_from_nns/","summary":"\u003cp\u003eI watched a \u003ca href=\"https://www.youtube.com/results?search_query=Jensen+And+Ilya\"\u003epodcast of Jensen and Ilya\u003c/a\u003e, in which Ilya talked about how multimodality enables neural networks to learn more features than just a single modality. For example, large language models like GPT-4, without vision, can recognize that the color pink is close to red, but they can\u0026rsquo;t explain why, because they haven\u0026rsquo;t seen a single pixel. Multimodality combines image and text, both trained in a unified way. And GPT-4 with vision can tell exactly which pixel is red and why. To achieve intelligence smarter than human-level intelligence, we will definitely need multimodality, because our world is very visual, and neural networks can learn a lot from it.\u003c/p\u003e","title":"Interpretable Features from Neural Networks"},{"content":"This is a quick skim notes for CS231n Introduction to CNN lecture 7, I used slides from this lecture to create notes and this is NOT an attempt to replicate notes by cs231n, its already the best notes on CNNs out there.\nCNNs are similar to ordinary neural networks, they have trainable weights and bias, receives input, bunch of trainable layers followed with non-linearity. Each layer is completely differentiable, means they can learn. At the end an output layer predicting classes or so.\nSo what changes? ConvNet architecture make the explicit assumption that the inputs are image which allows us to encode certain properties into the architecture. These then make the forward function more efficient to implement and vastly reduce the amount of parameter in the network.\nWhy can\u0026rsquo;t regular NNs don\u0026rsquo;t scale to images?\nConsider an image from CIFAR-10 dataset, size 32x32x3, input neurons would be 32x32x3 = 3072 weights. Good? Now consider an image 200x200x3 image, 120,000 weights in the first layer, moreover we would almost certainly want to have several such neurons, so the parameters would add up quickly. Clearly, this full connectivity is wasteful and the huge number of parameters would quickly lead to overfitting.\n3D volumes of neurons\nCNN\u0026rsquo;s take the advantage of the fact that the input are images and use the architecture in a more sensible way. In particular, unlike the regular neural networks the layer of convolution net have neurons arranged in three dimensions: Height, width and depth. The depth here is not the depth of the whole neural net. But instead, it\u0026rsquo;s the depth of trainable weights. A good visualization:\nTop: A regular 3-layer Neural Network. Bottom: A ConvNet arranges its neurons in three dimensions (height, width, depth) as visualized in one of the layers. Every layer of ConvNet transforms the 3D output volume of neuron activations. In this example the red block is the image, so its dimensions would be, height (28), width (28) and depth (3, because 3 channels ~RGB) Convolve Filters, also called kernels, are small (typically 3x3 or 5x5) used to slide over the image spatially, computing dot products. The filters also have depth, eg. 3x3x96 here 96 is depth of the kernel. The depth of kernel should be equal to the depth of the input. If the input is 32x32x3 image, where 3 is ~RGB. then kernel would have depth 3 (3x3x3 or 5x5x3).\nin the above case, spatial dimension is the 32x32 (height and width of input).\nKernels slide over the image, to convolve/feature extract. The dimension of the activation map (right) depends on input (left) size \u0026 depth and the kernel size \u0026 depth. Notice the reduction in size of the output image. let's say, there are 2 input channel (shown in the image 'consider a second green filter', blue and green) the kernel will also be slide through all location of the 2nd channel. Creates another output activations. Each kernel create a output activation for each channel. Now, if there are 6, 5x5 kernel. we will get 6 seperate activation maps. 6x28x28 in this case. Stack this up, plus non-linearlity (ReLU) on the activation maps. an overview of input to the conv layers and its output. each convs are followed by non-linearity (for example, ReLU), just stack them and we have a architecture? An overview of the ConvNet architecture\nThe activations of an example ConvNet architecture. The initial volume stores the raw image pixels (left) and the last volume stores the class scores (right). The middle shows visulizations of activations, since it is difficult to visualize 3D, here the representation is in 2D. How do we know, the size of the output after convolution?\n(N-F)/S + 1, where N is the number if input dimension, F is the kernel dimension. S is stride.\nOh yes, what is Stride?\nKernels slide through the input layer, Stride refers to number of pixels the filter mover over the input image during convolution (see some gifs)\nPadding\nIn practice, it is common to pad the input image with 0s around the edges.\nGood effect, we can control the spatial size of the output. Example, to retain the original size of input image, we can use (F-1)/2 padding. where F is the filter dim. You see. math time, calculate how many trainable params\nkernels are trainable some common settings for Convolutions we can also have 1x1 Kernels\n1x1 kernels are used for dimensionality reduction without altering spatial dimensions, 64 -\u003e 32 (halved but the spatial dimension remains same) Summary 1: Convolution Layer Accepts input of shape W, H, D Requires hyperparameters Number of Kernels\u0026ndash; N Kernel dim\u0026ndash; F Stride\u0026ndash; S Padding\u0026ndash; P The brain/neuron view of CONV layer\nRecall how the activation map depth is equal to the number of kernels.\na neuron view of convolution the output of a patch from img and element wise multiplication of kernel is the size of kernel itself, then we calculate the sum of it. ~Dot product. This creates a single activation in the activation layer Create layers spatially, they are not interconnected. It similar to channels stacked. Pooling Layer\nThis throws away some spatial information, but dont worry, not all information is equally valuable. Hypothetically, Pooling layer discards some less important details. Max pooling is generally better than average pooling because it preserves the most prominent features and introduces non-linearity, making it more effective in capturing key patterns and reducing overfitting. Case Study LeNet-5 Architecture: [Conv-pool-Conv-pool-conv-FC] AlexNet Increased the kernel depth, from 6 to 96. Applied kernel of size 11x11 at stride of 4. AlexNet was the first paper to use ReLU. This made ReLU popular. There are ReLU after each convolution layers and FC layers. We don't use the NORM layers used in AlexNet now, because it doesn't actually gives any improvement. Architecture of AlexNet: C = Conv, P = Pool, O = Output layer\n[C-P-C-P-C-C-C-P-FC-FC-O-Softmax]\nsome details:\nLR is reduced by 10, when Val accuracy plateaus ZFNet Similar architecture to AlexNet but increased number of kernels (N) VGGNet A rough diagram that I made, shows there are 13 Conv Layers used in VGG net (there are also pooling) ~by Andrej Karpathy Memory footprint of VGGNet shows,\nMost memory is in early CONV layer (3M) Most Parameters are in late FC layers (10M) 93MB/image in forward pass alone Average Pooling Layer replaces FC layer in the end\nThe example is from the FC layer of VGG net, the last POOL2 output is of shape: 7x7x512, where 7x7 are spatial dim and 512 is depth/volume. each 7x7 is averaged pool to 1. reducing the size quite a lot. GoogLeNet (introduced inception) This paper introduced Inception module, remove last FC layer with AveragePool layer to reduce parameters. ResNet In 2015, ResNet won 1st place in many competition at the same time. GoogleNet was 22 layers, introduced resnet was 152 layers.\nSo increasing layers == win?\nWell, yes thats what scaling hypothesis says. But increasing layers is challenging. Problems with vanilla nets, increasing the number of layers converges in training (left img, dashed lines) but the validation error of 56 layer \u003e 20 layer net. Which makes no sense. ResNet took, 2-3 weeks of training on 8 GPU, but at runtime, its faster than VGGNet, even though it has 8x more layers.\nHighway network is also a good paper by Jürgen Schmidhuber. Introduced just before ResNet. The Skip connection in Highway network had trainable weights. where as ResNet Skip connections is literally vanilla addition of input to output. AlphaGo (policy network?) Playing Go (harder than chess) using CNNs? The 48 rules of playing Go embedding into the image\u0026rsquo;s channel?\nSummary 2: Trends ConvNets Stack Conv, pool, FC layers Smaller filters and deep architecture get rid of Pool/FC layers (just convs) Treads toward stride-convs only layer, where you try to reduce image dimension (not depth) using convs instead of using pooling layer. ResNet/GoogleNet challenge this paradigm Sources Andrej Karpathy lecture on CNNs ","permalink":"https://akash5100.github.io/posts/2024-05-21-convolutional_neural_networks/","summary":"\u003cp\u003eThis is a quick skim notes for \u003ca href=\"https://www.youtube.com/watch?v=LxfUGhug-iQ\u0026amp;list=PLkt2uSq6rBVctENoVBg1TpCC7OQi31AlC\u0026amp;index=8\"\u003eCS231n Introduction to CNN lecture 7\u003c/a\u003e, I used slides from this lecture to create notes and this is NOT an attempt to replicate \u003ca href=\"https://cs231n.github.io/convolutional-networks/\"\u003enotes by cs231n\u003c/a\u003e, its already the best notes on CNNs out there.\u003c/p\u003e\n\u003cp\u003eCNNs are similar to ordinary neural networks, they have trainable weights and bias, receives input, bunch of trainable layers followed with non-linearity. Each layer is completely differentiable, means they can learn. At the end an output layer predicting classes or so.\u003c/p\u003e","title":"CNNs"},{"content":"You got a very deep neural network to train, lets say wide 128-layers. Does it fits in memory? Yes (barely). The activations and gradients in forward and backward pass respectively takes a lot of memory. But you want to train more deep NN. Why? because, we build the compute (stack more layer) ~= win. (scaling hypothesis) / Blessing of scaling.\nSee the original gradient checkpointing implementation.1\nHere is a visualization of vanilla training:\ntop left is \"input\", and the bottom right most is the \"loss\". Upper layer is the forward pass i.e, attentions and lower layer is the backward pass, i.e, gradients The purple shaded circles indicate which of the nodes need to be held in memory at any given time. However, if the cost of computation \u0026lt; cost of memory, we have limited memory and we are willing to recalculate those nodes (attentions and gradients) then we can save a lot of memory. We can simply recompute them when we need in the backward pass. We can always recompute the activations by running the same input data through a forward pass. See below:\nWe recompute the activation when we need during the backward pass. There is a lot of compute waste. For N-layers there would be N extra forward pass. Gradient checkpointing1 is something in between this two methods, where we recompute the forward pass but not too often.\nThere are checkpoints nodes in the memory during the forward pass, while the remaining nodes are recomputed at most once. After being recomputed, the non-checkpoint nodes are kept in memory until they are no longer required1.\nIn the first forward pass, we set checkpoints. In the backward pass we recalculate from the last checkpoint. This method trades off compute and memory Optimal checkpointing selection\nMarking every \\(\\sqrt{n}\\) -th node as a checkpoint optimizes memory usage, scaling with the square root of the number of layers.\nSources Gradient Checkpointing\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://akash5100.github.io/posts/2024-05-16-deep_network_training_hack_gradient_checkpointing/","summary":"\u003cp\u003eYou got a very deep neural network to train, lets say wide 128-layers. Does it fits in memory? Yes (barely). The activations and gradients in forward and backward pass respectively takes a lot of memory. But you want to train more deep NN. Why? because, we build the compute (stack more layer) ~= win. (\u003ca href=\"https://gwern.net/scaling-hypothesis#scaling-hypothesis\"\u003escaling hypothesis\u003c/a\u003e) / Blessing of scaling.\u003c/p\u003e\n\u003cp\u003eSee the original gradient checkpointing implementation.\u003csup id=\"fnref:1\"\u003e\u003ca href=\"#fn:1\" class=\"footnote-ref\" role=\"doc-noteref\"\u003e1\u003c/a\u003e\u003c/sup\u003e\u003c/p\u003e\n\u003cp\u003eHere is a visualization of vanilla training:\u003c/p\u003e","title":"Gradient-Checkpointing"},{"content":" work in progress. publishing this but will be updated every few days, as I learn new stuffs. learning \u0026gt; blog\nI am learning how the transformer-based architectures got evolved from 2017 (\u0026ldquo;Attention is all you need\u0026rdquo;) till today (May 2024). So, I thought why not write a case-study like this (by Andrej Karpathy).\nTable of content How can unsupervised learning work? GPT (June 2018) BERT (October 2018) Transformer-XL (September 2018) \u0026amp; XLNet (June 2019) GPT-2 (February 2019) Sparse Transformers (April 2019) How can unsupervised learning work? Why it works? Generalization?, hypothetically\u0026hellip;\nGPT (June 2018) TODO https://openai.com/index/language-unsupervised/\nBERT (October 2018) Google introduced BERT, which stands for Bidirectional Encoder Representations from Transformers. It is another language model based on Transformer architecture but unlike recent language models BERT is designed To pretrain bidirectional representation from unlabeled text. As a result, the pre-trained BERT model can be fine tuned with just one additional output layer to create state of the art models for a wide range of tasks such as question-answering, system language inference. Without any substantial task-specific architecture modification.\nLanguage model pre-training has been shown to be effective in the recent years. There are two existing strategies for applying pre-trained language representation to down-stream tasks:\nfeature-based: Uses task-specific architecture, example adding a layer at the end of the trained model and train them for downstream task. fine-tuning based: basically transfer learning? (like used in GPT). The trained weights gets updated as well finetuned on a task specific dataset. The two approaches share the same objective function during pre-training, where they use unidirectional language models to learn general language representations. But the authors of paper argues:\nCurrent technique limit the potential of pre-trained representations. Especially for fine tuning the issue stems from the unidirectional nature of standard language model like openai\u0026rsquo;s GPT. Which only allows token to attend from past. This restriction hampers performance and task requiring bidirectional context, such as question answering.\nArchitecture. The architecture was based on original Transformer based on Vaswani et al. (2017). BERT-base parameters equals to GPT-base for comparision purposes. But BERT uses bidirection self-attention. While the GPT uses constrained self-attention where every token can only attend to context to its left, BERT can see in both direction. This means BERT uses the Encoder-only part of the Transformer.\nPreTraining BERT Implementing this (encoder-only model) however was challenging. Unfortunately, standard conditional language models can only be trained left-to-right or right-to-left, since bidirectional would allow each word to indirectly attent itself (they called it \u0026ldquo;see itself\u0026rdquo;) and with this instead of learning important features, the model would cheat and wont learn anything at all. In order to fix this, BERT is trained using two unsupervised task.\nTASK #1: Masked Sequence: they masked some percentange of the input tokens at random, and then predict those tokens. This procedure is called \u0026ldquo;masked LM (LML)\u0026rdquo;. Introduced back in 1952 (Taylor, Cloze task in the literature). TASK #2: Next Sequence Prediction: TODO Input/Output representations BERT can handle, [CLS], [SEP], [MASK] and [PAD]. TODO explain the input and output of BERT.\nI pretrained a BERT model from scratch. see :)\nTransformer-XL (September 2018) and XLNet (June 2019) TL;DR: In the next section, you\u0026rsquo;ll learn that GPT-2 has a summarization problem - it struggles with long-term dependencies, leading to non coherent with words like color/log/hat/car etc. While I couldn\u0026rsquo;t find a specific paper addressing this issue, it\u0026rsquo;s related to the limited attention scope within the context window. TransformerXL tackles this problem by increasing long-term dependencies without expanding the context window. Additionally, it introduces a new positional encoding technique called Relative Positional Encoding, which replaces the absolute positional encoding used in the original Transformer. This approach has been adopted in later models like OpenAI\u0026rsquo;s \u0026ldquo;Fill In the Middle\u0026rdquo; (2022).\nIn the vanilla Transformer, attention doesn\u0026rsquo;t flow between context windows. Consider a context length (sequence length) of 64, where 64 tokens constitute a single segment during training. In this scenario, the Transformer is unable to attend to long-term dependencies, as it\u0026rsquo;s limited by the context length of 64. The next segment will have no information about the previous sentence. As a result,\nIllustration of the vanilla transformer with a segment length 4 ~from the Transformer-XL paper The fixed context length means the model cannot capture longer-term dependencies beyond the predefined context length. Moreover, fixed-length segments are created by selecting a consecutive chunk of symbols without respecting sentence or semantic boundaries. This leads to a lack of necessary contextual information, making it difficult for the model to accurately predict the first few symbols. Consequently, optimization is inefficient, and performance suffers. They refer to this issue as context fragmentation.\nAnd during inference, the vanilla model consumes a segment of the same length as in training, but only makes one prediction at the last position and in the next step, the segment is shifted to the right by only one position. We lost the context detail again for the next prediction.\nTransformerXL used caching of Key and Value attention computation to speed up inference. I think originally introduced in \u0026ldquo;Edouard Grave, et al. 2016 Improving neural language models with a continuous cache\u0026rdquo;.\nIllustration of the Transformer-XL with a segment length 4 ~from the Transformer-XL paper Introducing Segment-Level recurrence with state reuse.\nTo address the limitation of using fixed-length context, they propose to introduce a recurrence mechanism to the Transformer architecture. During training, the hidden state sequence computed for the previous segment is fixed and cached to reuse as an extended context when the model processes the next new segment is fixed and cached to be reused as an extended context when the model process the next segment. If we assume two consecutive segments of length L, then\nS_t = [x_t,1 , ... , x_t,L] and\nS_t+1 = [x_t+1,1, ... , x_t+1,L].\nSay the hidden state/calculated attention \u0026ldquo;memory\u0026rdquo; of the first segment is given by: h_t^n where n is the n-th layer hidden state. Then we can use this to calculate the hidden state of next segment.\nh^~ = [ SG(h_t^n) ◦ h_t+1^n-1 ]\nWhere SG is \u0026ldquo;Stop gradient\u0026rdquo;, similar to torch.no_grad, because we dont want to calculate the gradient for previous context, we store the cache as a constant with no backward graph.\n◦ indicates concat within the context dimension, say the input is of shape, B,T,C and after concatination of two segments, it would be B,2*T,C. (This notation is from my old blogs, where B is the batch size, T is the sequence length/context length, C is the embedding size).\nWe use h^~ to calculate Key and Value representations. Not the Query of course.\nK_t+1^n = h^~ * W_k^t\nV_t+1^n = h^~ * W_v^t\nQ_t+1^n = h_t+1^n-1 * W_q^t\nWhy not cache Q? That\u0026rsquo;s a mystery. or find out here.\nRelative positional encodings.\nWith the above achievement (reusing the previous context attention) we just created another problem. Notice, the positional encoding of current segment\u0026rsquo;s first token is equal to the previous segment\u0026rsquo;s first token.\nmakes model confuse? ~from https://vimeo.com/384795188 To fix this the authors introduced the concept of \u0026ldquo;relative positional encoding\u0026rdquo; to capture the relative position information between the element in the input sequence. This is different from The absolute positional encoding used in the original transformer model, which represents the absolute position of each element in the sequence.\nThe idea behind \u0026ldquo;relative positional encoding\u0026rdquo; is to represent the positional information as a relative distance between elements rather than their absolute positions.\nThis is particularly useful for processing sequence as it allow module to generalize better. The sequence of different length.\n```md Original transformer: E = VE + PE (Vocab Embeddings + Positional Embeddings) TransformerXL, incorporates directly VE and PE into attention: Attn = F(VE,PE) \u0026gt; similar sinusoidal functions are used to generate positional embeddings \u0026gt; additional relative_pos and relative_key encodings are also generated, along with Q, K and V - Q, K, V, rel_key and rel_pos - relative_pos: Learnable relative positional embeddings. (Parameter) - relative_key: Derived from relative_pos using a linear transformation. (Linear) \u0026gt; 2 attn scores are calculated: content-based \u0026amp; position-based - content-based: Q @ K.T - position-based: rel_k = rel_pos -\u0026gt; rel_key Q @ rel_k.T \u0026gt; attn = content-based + position-based (continue attention scale, tril, softmax, V ...) ``` GPT-2 (February 2019) TODO-- easy already read the paper\nSparse Transformers (April 2019) The Transformer architecture, introduced in the paper \u0026ldquo;Attention is All You Need\u0026rdquo; by Vaswani et al. in 2017, is a widely used neural network model for natural language processing tasks. However, its computational complexity and memory requirements grow quadratically with the sequence length, which can become a limitation for long sequences.\nGiven sequence of length n. number of computations required to process attention weights: n x n = n^2 Memory required to store the intermediate results and attention weights grows quadratically Sparse Transformer reduced this complexity to n x sqrt(n) Some understanding from this paper about the Transformer architecture. Transformer autoregressive model is used to model joint probability of sequence x = {x1, x2, x3, ... xn } as the product of conditional probability p(xn+1 | x).\nIt has the ability to model arbitary dependencies in a constant number of layers. As self-attention layer has a global receptive field.\nIt can allocate representation capacity to input regions, thus this architecture is maybe more flexible at generating diverse data types than networks with fixed connectivity patterns.\nThey trained 128-layer Transformer on image and found that the network\u0026rsquo;s attention mechanism is able to learn specialized sparse structures\nearly layer learned locally connected patterns, which resemble convolution. layer 19-20 learned to split the attention across row and column attention several attention layers showed global data-dependent access patterns layer 64-128 exhibited high sparsity, with positions activating rarely and only for specific input patterns. Learned attention patterns from a 128-layer network In a fully connected attention mechanism, for each query at i'th position needs to attend all the keys in the sequence. This dense Attention Matrix has sparse attention patterns across most datapoints, suggesting that some form of sparsity could be introduced without significantly affecting performance.\nThe paper introduced three key techniques to solve the quadratic complexity issue of standard Transformer when modeling long sequence:\nFactorized self-attention\nFactorization of self-attention is breaking them down into several faster attention, when combined can approximate the dense attention. This is applied on sequence of unprecedented length. Original self-attention involves every position attending, factorized self-attention splits attention into multiple heads, each focuses on a subset of positions. Mathematically, instead of having each head attend to all positions in the input sequence, we restrict it to a subset, for example, if there is n positions in the input sequence, we don\u0026rsquo;t need each head to look at all n position instead, each head can look approximately \\( \\sqrt{n} \\) positions.\nThe size of each attention set A is proportional to the square root of the total number of positions.\nTwo Dimentional Factorized Attention\nproposed for structured data like images or audio. Strided Attention: One head attends to previous \\(l\\) positions, another attends to every \\(l\\)-th position (stride). Strided attention is beneficial for structured data but less effective (fails) for unstructured data like text. Fixed Attention: For unstructured data a fixed pattern is used where each position attends to specific past positions, aiming to ensure relevant information is propogated efficiently. Scaling to 100 of layers: Used pre-activation residual block. from the Sparse Transformer paper Reuse attention matrices (saving memory)\nGradient checkpointing is particularly effective for self-attention layers when long sequences are processed, as memory usage is high for these layers relative to the cost of computing them. Using recomputation alone, we are able to train dense attention networks with hundreds of layers on sequence lengths of 16,384, which would be infeasible on modern hardware otherwise. Mixed precision training\nStoring weights in single precision floating-point (32-bit), but compute network activations and gradients in half-precision (16-bit). (Micikevicius et al., 2017). Dynamic loss scaling. When we use half precision to calculate activations and gradients, they are more prone to underflow (very small becoming 0) and overflow (very large becoming infinity). To fix that, when calculating gradient, the loss value is scaled (multiplied) with a large number, before backward pass. We then calculate the gradients. Unscale the gradients by dividing them by the scale value. Update the parameters using unscaled gradients.\nDynamic loss scaling is adaptive, meaning scale value is chosen dynamically. Initialized with a large number like 1024. After each backward pass, we check if the gradients contains any NaNs or Inf. If overflow, reduce the scaling factor by factor of 2. If no overflow, increase the scaling factor optionally. [Pytorch docs]\nEfficient sparse attention kernels\nFused Softmax kernel\nTypically, the softmax operation is computed as a seperate kernel or function that requires writing, loading, computing and again writing. This paper proposes fusing the softmax operation into single kernel that uses registers to avoid reloading inputs. This means that the softmax operation is computed directly in the kernel, without storing and loading. No need to calculate the upper triangular in attention, as it is never used in autoregressive task. ","permalink":"https://akash5100.github.io/posts/2024-05-09-case_study_transformer_based_architecture_development/","summary":"\u003cblockquote\u003e\n\u003cp\u003ework in progress. publishing this but will be updated every few days, as I learn new stuffs. learning \u0026gt; blog\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eI am learning how the transformer-based architectures got evolved from 2017 (\u0026ldquo;Attention is all you need\u0026rdquo;) till today (May 2024). So, I thought why not write a case-study like \u003ca href=\"https://cs231n.github.io/convolutional-networks/#case-studies\"\u003ethis\u003c/a\u003e (by Andrej Karpathy).\u003c/p\u003e\n\u003ch4 id=\"table-of-content\"\u003e\u003cstrong\u003eTable of content\u003c/strong\u003e\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#how-can-unsupervised-learning-work\"\u003eHow can unsupervised learning work?\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#gpt-june-2018\"\u003eGPT (June 2018)\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#bert-october-2018\"\u003eBERT (October 2018)\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#transformer-xl-september-2018-and-xlnet-june-2019\"\u003eTransformer-XL (September 2018) \u0026amp; XLNet (June 2019)\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#gpt-2-february-2019\"\u003eGPT-2 (February 2019)\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#sparse-transformers-april-2019\"\u003eSparse Transformers (April 2019)\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003c!-- - [Reformer](#reformer)\n- [Linformer](#linformer)\n- [Vision Transformer \u0026 Image transformer (Niki parmar)](#vision-transformer-image-transformer-niki-paramr)\n- [RoBERTa (July 2019)](#roberta-july-2019)\n- [DistilBERT (March 2020)](#distilbert-march-2020) --\u003e\n\u003ch4 id=\"how-can-unsupervised-learning-work\"\u003eHow can unsupervised learning work?\u003c/h4\u003e\n\u003cp\u003e\u003ca href=\"https://www.youtube.com/watch?v=AKMuA_TVz3A\"\u003eWhy it works? Generalization?\u003c/a\u003e, hypothetically\u0026hellip;\u003c/p\u003e","title":"Case Study on Transformer-based architecture"},{"content":"Long story short, Key value caching is used to reduce the inference speed of AR models (autoregressive / decoder-only). It does so by caching the already computed attention scores (K and V) of previous 1\u0026ndash; t-1 tokens. While the generated token is t.\nInference refers to the process of using a trained model to make predictions or generate output for new, unseen input data. In other words, it\u0026rsquo;s the process of applying the model to real-world data to get a result or prediction.\nStarting from single token\nlet\u0026rsquo;s say, we start with single token. During inference, when generating output sequences token-by-token, the sequence length increases by 1 at each step. Here\u0026rsquo;s how it works:\nInitial input: A single token (sequence length 1) Generate output token 1: SL becomes 2 (input token + generated token) Generate output token 2: SL becomes 3 (input token + 2 generated tokens) Generate output token 3: SL becomes 4 (input token + 3 generated tokens) \u0026hellip; (continues)\nIf you see, we are again and again calculating the Key and Values for last t-1 tokens. Because attention mechanism needs to consider K, Q, V for all previous tokens (equals to context length) when generating each new token, resulting in a quadratic increase in computation with respect to sequence length.\nKV caching addresses this issue by storing the computed Key (K) and Value (V) matrices for each token generation step. This allows the model to reuse these cached matrices instead of recomputing them from scratch at each step, reducing the computational cost.\nSource of the image at the end With KV caching, the sequence length still increases by 1 at each step, but the model can efficiently reuse previously computed K and V matrices, mitigating the quadratic increase in computation. This makes long-range generation more efficient and scalable.\nSource of the image at the end Inference from a prompt (starting from many token)\nQuestion: if the sequence length the transformer is trained on is 16, and the inference input prompt is of length 18 or 20 or anything, the input to start sampling is always last 16 tokens. ([-T:]). Does KV-caching still helps?\nIf the transformer is trained on sequences of length 16 and the inference input prompt is longer (e.g., 18), the model will typically use a sliding window approach, processing the input prompt in chunks of 16 tokens at a time. This is known as \u0026quot; truncation\u0026quot; or \u0026ldquo;windowing\u0026rdquo;.\nIn this case, the input to generate the inference would be the last 16 tokens of the prompt (using slicing notation, input_prompt[-16:]).\nKV caching still helps in this scenario. Even though the input prompt is longer than the training sequence length, the model is still generating output tokens one at a time, and the attention mechanism needs to consider the previous 16 tokens when generating each new token.\nBy caching the Key (K) and Value (V) matrices for each token generation step, KV caching reduces the computational cost of recomputing these matrices from scratch at each step. This is especially important when generating long output sequences, as the attention mechanism needs to consider an increasing number of previous tokens.\nWhy Query (Q) is not cached\nAs the next generated token is used as query to the next prediction, we need to recompute this every time. We don\u0026rsquo;t know the future prediction, do we? :)\nSources As this blog is an personal notes, this medium blog is an awesome read: kv-caching explained ","permalink":"https://akash5100.github.io/posts/2024-05-06-key-value_caching_for_fast_inference/","summary":"\u003cp\u003eLong story short, Key value caching is used to reduce the inference speed of AR models (autoregressive / decoder-only). It does so by caching the already computed attention scores (K and V) of previous \u003ccode\u003e1\u003c/code\u003e\u0026ndash; \u003ccode\u003et-1\u003c/code\u003e tokens. While the generated token is \u003ccode\u003et\u003c/code\u003e.\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eInference refers to the process of using a trained model to make predictions or generate output for new, unseen input data. In other words, it\u0026rsquo;s the process of applying the model to real-world data to get a result or prediction.\u003c/p\u003e","title":"Key-Value caching"},{"content":" nothing very special as I expected, because it was just a heat map of embeddings, next time, something better for sure. I aimed to create visualization like this research paper or by Andrej Karpathy.\nI started with creating vocabulary for my model using BPE (Byte Pair algorithm). Hyperparameter for BPE merges I used is 2000-256, it would create vocab of size 2000. How? recall your learning and solve that mystery. Before running BPE on \u0026lsquo;Elvis Presley\u0026rsquo; wiki page (because it was very long, no other reason), I split them into words using the Llama3 tokenizer Regex:\nLLAMA3_SPLIT_PATTERN = r\u0026#34;\u0026#34;\u0026#34;(?i:\u0026#39;s|\u0026#39;t|\u0026#39;re|\u0026#39;ve|\u0026#39;m|\u0026#39;ll|\u0026#39;d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+\u0026#34;\u0026#34;\u0026#34; Model. Decided to use the GPT-2 transformer architecture.\nCase Study on GPT-1 and GPT-2.\nGPT-1\u0026rsquo;s architecture is very similar to the original transformer presented in Attention is All You Need paper, but they use learnable positional embedding instead of Sine function. The paper proposed a framework to train language model capable of multitasking with little finetuning (as this was the trend in 2018, example ULM-FIT by Howard. Transfer learning etc). The framework consist of 2 stages, (1) learning high capacity language model on a large corpus of text, unsupervised (i.e, predict next token.) They used BPE for tokenization as it was lossless and can compress tokens. They achieved 18.4 perplexity with GPT-1 on BookCorpus dataset. (2) supervised finetuning stage, where they adapt the model for discriminative task with labeled data, like text classification, entailement, similarity, MCQ. Example would be\u0026ndash; applying linear transformation to the output of GPT-1 model for specific task such as classification.\nGPT-2 on the other hand demonstrated that the language models can perform down-stream tasks in a zero-shot setting– without any parameter or architecture modification. Highlighting the ability of language models to perform a wide range of tasks in a zero-shot setting. Small changes on the architecture of transformer is, (1) adding Normalization layer before the multihead-attention layer. (2) Adding a normalization layer before the final linear layer.\nThe main game changer of GPT-2 is the quality of dataset it was trained on, they created a new web scrape which emphasizes document quality. They scraped only the web pages which has been curated/filtered by humans. Manually filtering a full web scrape would be exceptionally expensive so as a starting point, scraped all outbound links from Reddit, a social media platform, which received at least 3 karma. This can be thought of as a heuristic indicator for whether other users found the link interesting, educational, or just funny.\nGoal openai tried to achieve: Current systems are better characterized as narrow experts rather than competent generalists. We would like to move towards more general systems which can perform many tasks – eventually without the need to manually create and label a training dataset for each one.\nZeroShot behavior of GPT-2\nWhy pre-training language model is effective? Hypothesis is underlying generative model learns to perform many task in order to improve its language modeling capacity. The more structured attentional memory of the transformer assists in transfer compared to LSTMs.\nHow Zero shot prompts worked?\nFor summarization, they added text (article) and at the end TL;DR: GPT-2 Focuses on recent contents from the article or confuse specific details such as how many cars were involved in the crash or whether a logo was on a hat or shirt.\nFor Translation, the prompt is english sentence = french sentence \u0026lt; some english sentence \u0026gt; = , the model the generates translated sentence, altho low accuracy because training data have only 10MB in 40GB.\nQuestion Answering is a bit interesting and funny, figure out yourself. Go here\nFine, here it is. Back to visualization of attentions.\nTrained the GPT-2 model, which has 4 Head, 256 head size -\u0026gt; 256/4 = 64 each block, 16 context size, 5 Transformer Block, 32 batch size and 64 embedding for possitional and vocab. Dropout of 0.3, and used AdamW with wd - 0.01.\nSampling code:\n@torch.no_grad() def generate(self, idx, max_new_tokens): if type(idx) is str: idx = torch.tensor(tkn.encode(idx), dtype=torch.long).unsqueeze(0) # add batch for _ in range(max_new_tokens): i = idx[:, -SL:] logits, _ = self(i) # forward logits = logits[:,-1,:] probs = logits.softmax(-1) next_idx = torch.multinomial(probs, 1) idx = torch.cat((idx, next_idx), dim=-1) yield next_idx prompt = \u0026#34; Hello, this is\u0026#34; gen = model.generate(prompt, max_new_tokens=100) for token in gen: token = [token.item()] print(tkn.decode(token), end=\u0026#39;\u0026#39;, flush=True) Observations.\nGathered all the attentions from Transformer class and painted them. Increasing the context size (Sequence length) from 8 to 16 gave significant improvement. Limited with CPU, so I didn\u0026rsquo;t increase the embedding size, another reason was I also wanted to visualize them. Training model like GPT-2 (1.4B parameters) on a big and quality dataset, was enough to generalize diversity of Natural langauge sementics that it performed zero-shot on variety of task. some headmaps bruh, generating 4 different tokens each. Edit: okay, I trained a small 2 layer model, with good big context length in Amazon SageMaker, here are the results.\nPrompt: \"Swift at the 1989 World Tour, the highest-grossing tour of 2015. In March 2014, Swift began living in New York City\", prediction: \" as\" Prompt: \"New\", preds: \" York\" Prompt: \"Are you\", preds: \" Harris\" Prompt: \"She is one of the most\", preds: \"-streamed\" (it created most-streamed) Source GPT-1 GPT-2 Visualization notebook Generation and Sampling notebook found a good read tonight, 4.4.24 edit: (Edit, 10th May) I found this blog, when I was reading the BERT paper which tried the similar kind of heatmaps across different layers of the transformer. Do I think like a scientist? hmm. (Edit, 15th May): In the Sparse Transformer paper by openai, they trained 128 layered deep network and found that attentions at different depth learns different patterns (see Figure 2.), and if you notice, (maybe) we got all the 4 above. Outro Reasoning in human beings refers to the cognitive process of systematically making sense of information, drawing conclusions, and solving problems through logical thinking, analysis, and inference.\nWhat is possibility of GPT-5 or a future AI models being highly finetuned to answer with human-like reasoning? Furthermore, breaking down the training process into fine-tuning for tasks such as predicting the best possible analysis, same for logic from available information is a plausible direction for advancement. Just a thought.\n","permalink":"https://akash5100.github.io/posts/2024-05-04-i_challenged_myself_to_visualize_attentions/","summary":"\u003cblockquote\u003e\n\u003cp\u003enothing very special as I expected, because it was just a heat map of embeddings, next time, something better for sure. I aimed to create visualization like this \u003ca href=\"https://arxiv.org/pdf/1601.06733\"\u003eresearch paper\u003c/a\u003e or by \u003ca href=\"https://karpathy.github.io/2015/05/21/rnn-effectiveness/\"\u003eAndrej Karpathy\u003c/a\u003e.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eI started with creating vocabulary for my model using BPE (\u003ca href=\"https://en.wikipedia.org/wiki/Byte_pair_encoding\"\u003eByte Pair algorithm\u003c/a\u003e). Hyperparameter for BPE merges I used is \u003ccode\u003e2000-256\u003c/code\u003e, it would create vocab of size \u003ccode\u003e2000\u003c/code\u003e. How? recall your learning and solve that mystery. Before running BPE on \u0026lsquo;Elvis Presley\u0026rsquo; wiki page (because it was very long, no other reason), I split them into words using the Llama3 tokenizer Regex:\u003c/p\u003e","title":"I challenged myself to visualize attentions (nothing special)"},{"content":"Back in 2017, the deep learning folks were trying to achieve state-of-the-art performance in sequence to sequence modeling. They used RNN\u0026rsquo;s like LSTM and GRU. The best performing model connects the encoder and decoder through an attention mechanism. Where the encoder is used to encode some sequence into rich embeddings and decoder decodes that embedding to different sequence. Example, language translation.\nTable of contents Sequence to sequence modeling and limitations of RNNs Transformer Model Architecture Attention Types of attention function Why scaling Multihead attention Auto-regression mask for self-attention Head (Masked-attention) Feed Forward (Linear transformation) Input Embeddings (Vocab and Positional) Regularizations used in the paper Transformer Architectures Encoder-Decoder Encoder only Decoder only Sources Sequence to sequence modeling and limitations of RNNs RNN had the vanishing gradient problem, models like LSTM and GRU emerged to tackle this problem but realized that altho the text may look better than rubbish, its still rubbish. It may look like English, but jumbled words and the sentence has no meaning.\nhttps://arxiv.org/pdf/1409.0473 (TLDR, first paper on using attention on RNN) https://arxiv.org/pdf/1601.06733 \u0026ndash; This is a good paper on visualizing attention mechanism and using LSTM with Attention They then tried to use attention mechanism with LSTM, Aligning the positions to steps in computation time, they generate a sequence of hidden states ht , as a function of the previous hidden state ht−1 and the input for position t. This inherently sequential nature precludes parallelization within training examples, which becomes critical at longer sequence lengths, as memory constraints limit batching across examples. Recent work has achieved significant improvements in computational efficiency through factorization tricks and conditional computation, while also improving model performance in case of the latter. The fundamental constraint of sequential computation, however, remains.\nRNN generates a sequence of hidden states ht, as a function of the previous hidden states ht-1 and the input for position t. This in a natural way, prevents the sequence to be trained parallelly, which becomes critical at longer sequence lengths, as memory constaints limit batching across examples. There are researches that had significant improvements in computational efficiency through factorization tricks and conditional computation, while also improving model performance. But the fundamental constraint of sequential computation still remains.\nTLDR, one limitation is their computational complexity, especially with longer sequences, which can lead to slower training and inference times. They might struggle with capturing long-range dependencies in sequences.\nTransformer Transformer was the first model architecture which avoided RNN and instead relying entirly on an attention mechanism to draw global dependencies between input and output. Transformer allows for significantly more parallelization and thus requires less time to train.\nshows how the next word is predicted based on the previous words, Image is stolen from here: https://arxiv.org/pdf/1601.06733, I might create a project to visualize this myself stay tuned :D Model Architecture The paper was for natural language translation and most competitive neural sequence transduction model have an encoder-decoder structure, so the transformer proposed is also an encoder-decoder transformer. Where the encoder took sequence x (1-\u0026gt;n) and generates continuous representation z (1-\u0026gt;n). Given z, the decoder then generates an output sequence one element at a time. At each step the model is auto-regressive. Consuming the previously generated symbols as additinal input when generating the next.\nTransformer Architecture: taken from the all time fav paper, Attention is All You Need. Attention Understanding the attention is the core of transformer. Attention can be defined as mapping a query and a set of key-value pairs to an output, where the query, key, values and output are all vectors.\nScaled Dot-product attention: again, taken from the all time fav paper, Attention is All You Need. Each vector is derived from the input X. In relatable words, to interpret this, think you have\na query matrix as a question or prompt (all latent factors, you know nothing about what kind of query is present), and the key as a set of potential answers or relevant information. By multiplying the query with the transpose of the key, you\u0026rsquo;re essentially comparing the question with each potential answer to determine their similarity/relevance. you get the attention score, (typically scaled and then normalized using softmax) scale? why? because big numbers == more computation. this normalized attention distribution will then guide the model (talking about optimization process aka backprop) on how much weight to assign to each value when computing weighted sum, which represents the attended information or the answer to the query. Types of attention function The two most commonly used attention functions are additive attention, and dot-product attention. Additive attention computes the compatibility function using a feed forward network with a single hidden layer. Both achieves the same but the dot-product attention is much faster and more space-efficient in practice, since it can be implemented using highly optimized matrix multiplication code.\n\\[ \\text{Attention}(Q, K, V) = \\text{softmax}\\left(\\frac{QK^T}{\\sqrt{d_k}}\\right) V \\] After the matrix multiplication, we scale the output by 1/(Dk)**0.5. Where, Dk is the dimension of keys and queries. After that we apply softmax to obtain weights on the values.\nWhy scaling While for the small value of Dk, additive function outperforms the dot-product attention without scaling for larger value of Dk. This is because, when the value of Dk is small, softmax receives bigger input which tends the softmax to create the probability distribution more easily. But, as we scale the dimension of head, i.e head_size (Dk) the input to softmax gets smaller and its hard for softmax to distribute the probability. To counteract this effect, we scale the dot product by 1/(Dk)**0.5.\nas we scale, i.e, the parameters gets bigger, we need to scale harder. This is the reason we choose dimension of Key-Query. The scaling prevents the dot products from becoming too large, which could lead to extremely small gradients in the softmax function, making training difficult Therefore, we use scaled dot-product attention rather than additive attention without scaling.\nMultihead attention Instead of performing a single attention function with d dimension\u0026ndash; d keys, queries and values, it is beneficial to linearly project the queries, keys and values n times with different. We then perform the attention function in parallel, yielding n times head_size output values. These are then concatenated and then have a linear transformation.\nMultihead attention Auto-regression mask for self-attention Head (Masked-attention) Recall this attention mechanism, but what is the mask?\nThe Mask is optional. We create a triangular matrix, 0\u0026rsquo;s on the top right and 1\u0026rsquo;s on the bottom left.\n\u0026gt;\u0026gt;\u0026gt; import torch \u0026gt;\u0026gt;\u0026gt; torch.tril(torch.ones((4,4))) tensor([[1., 0., 0., 0.], [1., 1., 0., 0.], [1., 1., 1., 0.], [1., 1., 1., 1.]]) take another matrix\n\u0026gt;\u0026gt;\u0026gt; torch.randn(4,4) tensor([[ 0.8350, 1.4479, -2.2848, -1.5227], # \u0026lt;- Sequence 1 acts [-0.8851, -0.4905, -1.1630, -1.0715], # \u0026lt;- Sequence 2 \u0026#39;\u0026#39; [ 1.0836, -1.5426, 1.9595, -1.3338], # \u0026lt;- Sequence 3 \u0026#39;\u0026#39; [-0.3320, 0.5324, 0.2499, -0.5427]]) # \u0026lt;- Sequence 4 \u0026#39;\u0026#39; Now if we have matrix multiplication, the 0\u0026rsquo;s effectively cancels the future occuring numbers.\nThe activations of tokens getting multiplied with 0's, doesnt take part in the prediction, thus achieving autoregressive property. We can improve this, instead of zeros in the mask matrix, if we replace them with -inf it will help Softmax. Replacing zeros with -inf in the triangular matrix masks future tokens, ensuring the model only attends to past and present tokens during self-attention by setting future attention scores to effectively zero.\n\u0026gt;\u0026gt;\u0026gt; torch.tensor(float(\u0026#39;-inf\u0026#39;)).exp() tensor(0.) \u0026gt;\u0026gt;\u0026gt; torch.tensor(0).exp() tensor(1.) Feed Forward (Linear transformation) In addition to attention sub-layers, each of the layers in encoder and decoder contains a fully connected feed-forward network, which is applied to each position separately and identically. This consists of two linear transformations with a ReLU activation in between.\nFF(x) = [W1*x + B1] -\u0026gt; [relu] -\u0026gt; [h -\u0026gt; W2*h + B2] Linear transformation on attention as described in \"Attention is All You Need\". Input Embeddings (Vocab and Positional) Purpose. Since the model has no conv or recurrent network, in order to make the use of the order of the sequence, we must inject some information about the relative or absolute position of the token in sequence. Therefore, in addition to the classic vocab embedding (embedding for each token) we also add embeddings for each position in the sequence. The positional embeddings have the same dimension as the embeddings, so that the two can be summed.\nChoice. It is possible to start with a Fixed positional embedding or learned positional encoding, it is mentioned in the research paper that they found the two version produced nearly identical results. Hence, they choose the sine version (fixed) because it may allow the model to extrapolate to sequence lengths longer than the ones encountered during training. Fixed positional encodings are pre-defined functions that map each position in the sequence to a fixed vector representation. Useful resource.\nread TransformerXL paper.\nPositional embedding added with token embeddings. Comparison. both versions produced nearly identical results, suggesting that the choice of positional encoding method did not significantly impact model performance. Ultimately, the authors opted for the sinusoidal positional encodings due to their potential for extrapolating to longer sequence lengths beyond those seen in training data.\nRegularizations used in the paper Residual Dropout: dropout to the output of each sublayer before it is added to the sublayer input and normalized. Dropout is also applied to the sum of embeddings and positional embeddings in both encoder and decoder stack. (p=0.1) Label Smoothing: instead of one hot encoded labels, smoothing the layers. high-level example: 0,1,0 -\u0026gt; 0.1, 0.8, 0.1. Transformer Architectures Encoder-Decoder Used for sequence to sequence task like translation, summarization. Encoder process the input sequence, decoder with masked-self-attention, cross-attention over encoder output and feed forward sub layers. Generates output sequence autogressively.\nHere is the intution, the encoder creates feature rich representation of the input sequence, the decoder then uses the memory (i.e, key-value) of the encoder to generate output autoregressively. But the decoder also has the feature rich representation of inputs (i.e, query).\nThe using of encoder\u0026rsquo;s attention in decoder is called cross-attention. (encoder: key-value \u0026ndash;\u0026gt; decoder: query)\nEncoder only Used for task like text classification, feature extraction, pretrained representations, document embeddings, transfer learnings.\nbasically creating feature rich embeddings for given sequence. The embedding then can be used to different tasks mentioned above.\nDecoder only Used for language modeling, text generation tasks. Decoder components are masked self-attention and feed-forward layers, no encoder, no cross attention. Autoregressive in nature (due to attn-mask).\nSources Attention is All You Need, 2017 useful for visualization of sin positional encoding tensorflow original transformer ","permalink":"https://akash5100.github.io/posts/2024-04-28-transformers/","summary":"\u003cp\u003eBack in 2017, the deep learning folks were trying to achieve state-of-the-art performance in sequence to sequence modeling. They used RNN\u0026rsquo;s like LSTM and GRU. The best performing model connects the encoder and decoder through an attention mechanism. Where the encoder is used to encode some sequence into rich embeddings and decoder decodes that embedding to different sequence. Example, language translation.\u003c/p\u003e\n\u003ch3 id=\"table-of-contents\"\u003eTable of contents\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#sequence-to-sequence-modeling-and-limitations-of-rnns\"\u003eSequence to sequence modeling and limitations of RNNs\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#transformer\"\u003eTransformer\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#model-architecture\"\u003eModel Architecture\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#attention\"\u003eAttention\u003c/a\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#types-of-attention-function\"\u003eTypes of attention function\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#why-scaling\"\u003eWhy scaling\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#multihead-attention\"\u003eMultihead attention\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#auto-regression-mask-for-self-attention-head-masked-attention\"\u003eAuto-regression mask for self-attention Head (Masked-attention)\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#feed-forward-linear-transformation\"\u003eFeed Forward (Linear transformation)\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#input-embeddings-vocab-and-positional\"\u003eInput Embeddings (Vocab and Positional)\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#regularizations-used-in-the-paper\"\u003eRegularizations used in the paper\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#transformer-architectures\"\u003eTransformer Architectures\u003c/a\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#encoder-decoder\"\u003eEncoder-Decoder\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#encoder-only\"\u003eEncoder only\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#decoder-only\"\u003eDecoder only\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#sources\"\u003eSources\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"sequence-to-sequence-modeling-and-limitations-of-rnns\"\u003eSequence to sequence modeling and limitations of RNNs\u003c/h2\u003e\n\u003cp\u003eRNN had the vanishing gradient problem, models like LSTM and GRU emerged to tackle this problem but realized that altho the text may look better than rubbish, its still rubbish. It may look like English, but jumbled words and the sentence has no meaning.\u003c/p\u003e","title":"Transformers"},{"content":"Once the gradient is computed, it is then used to update the parameters. There are several approaches to perform the update.\nTable of contents First order(SGD), momentum, nesterov momentum Stochastic gradient descent Momentum Update Nesterov Momentum (NAG) Per-parameter adaptive learning rates (Adagrad, RMSProp) Adagrad RMSProp (Gef. Hinton) Adam AdamW AdamScheduleFree Second order methods Sources First order(SGD), momentum, nesterov momentum Stochastic gradient descent This is the vanilla update, the simplest form of update that changes the parameters along the negative gradient direction, as the gradient indicates the direction of increase, whereas we usually wish to minimize a loss. Assuming a vector of parameters x and the gradient dx, the simplest form of the update is as follows.\nx += -learning_rate * dx # vanilla SGD update Where learning_rate is the hyperparameter. In practice, we barely use this vanilla update because it is way too slow.\nConsider a slope which is shallow horizontally and steep vertically. So with this approach, the gradients of the parameters would have high gradients vertically and small gradients horizontally, which then means the update would look like this.\nparameter present in shallow horizontally and steep vertically, converging to minima using SGD. Momentum Update As the name suggests, this update applies momentum to the vanilla update.\nHere is the formula. V is equal to mu. times V Lr into gradient X += V. Where v is the momentum that we build along the process of training and mu is a hyperparameter that is used to decay the velocity, just like the friction decays the velocity of a ball rolling on the ground. This hyperparameter mu decays the velocity, and the process gradually converges.\nThis method overshoots but converges quickly compared to standard gradient descent.\nimage credit: https://twitter.com/alecrad Usually, the value of `mu` is set to `0.5` or `0.9`, but sometimes it is annealed over time from `0.5` to `0.9`. Nesterov Momentum (NAG) Also called Nesterov accelerated gradient descent (NAG).\nNesterov momentum update modifies the ordinary momentum update with a technique \u0026ldquo;lookahead\u0026rdquo;.\nNesterov momentum is a slightly different version of momentum that has gained popularity. It enjoys stronger theoretical convergence guarantees for convex functions, and in practice, it also consistently works slightly better than standard momentum.\nThe core idea behind Nesterov momentum is that when the current parameter vector is at some position x, then looking at the momentum update above, we know that the momentum term alone (i.e. ignoring the second term with the gradient) is about to nudge the parameter vector by mu * v. Therefore, if we are about to compute the gradient, we can treat the future approximate position x + mu * v as a \u0026ldquo;lookahead\u0026rdquo; \u0026ndash; this is a point in the vicinity of where we are soon going to end up. Hence, it makes sense to compute the gradient at x + mu * v instead of at the \u0026ldquo;old/stale\u0026rdquo; position x. If we know the momentum of the parameter (calculated previously), we use that and calculate the gradients ahead in that position. That is, in a slightly awkward notation, we would like to do the following: x_ahead = x + mu * v # evaluate dx_ahead (the gradient at x_ahead instead of at x) v = mu * v - learning_rate * dx_ahead x += v However, in practice people prefer to express the update to look as similar to vanilla SGD or to the previous momentum update as possible. This is possible to achieve by manipulating the update above with a variable transform x_ahead = x + mu * v, and then expressing the update in terms of x_ahead instead of x. That is, the parameter vector we are actually storing is always the ahead version. The equations in terms of x_ahead (but renaming it back to x) then become:\nv_prev = v # backup v = mu * v - learning_rate * dx # velocity update stays the same x += -mu * v_prev + (1 + mu) * v # position update changes form Per-parameter adaptive learning rates (Adagrad, RMSProp) Adagrad We scale the gradient with an additional variable, cache.\ncache += dx**2 x += -learning_rate * dx / sqrt(cache + 1e-7) We are keeping the track of gradient for every single parameters. So this is often called per-parameter adaptive learning method. And then we divide the gradient element-wise with the square root of cache. This effectively changes the learning rate per parameter that is scaled dynamically based on their gradient.\nTLDR;\nlarge gradient vertically are added up to cache this increase the cache size we end up dividing large number. We get smaller number in the update (the denominator is large) conclusion: The gradient updates are vertical. So, in this technique: the vertically updates are reduced and update look like this: link to the Adagrad paper: https://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf Problem with ADA grad: What happens as the step (epoch) size increases?\nthe cache became larger and larger denominator becomes extremely big updates are reduced to almost None But we don\u0026rsquo;t want this, in a deep neural network We want our optimization technique to keep shuffling the parameters untill we minimize the loss.\nRMSProp (Gef. Hinton) Instead of keeping the sum of square in cache, we define the cache so that it leaks and loses information so that it never grows too much big.\nWe set the decay hyperparameter\u0026ndash; this way we are collecting the squares of gradient but slowly leaking it.\n# ADAgrad cache += dx**2 x += -learning_rate * dx / sqrt(cache + 1e-7) # RMSProp beta = 0.9 # leaking constant cache = beta*cache + (1-beta)*(dx**2) # Notice removal of += x += -learning_rate * dx / sqrt(cache + 1e-7) # I can see 2 kind of decay, # - rapid decay on build up cache # - slow decay on current grad parameters Adagrad stops too quickly, but RMSProp continues.\nAdam This technique proposed update that looks a bit like RMSProp with Momentum update.\n# recall the Momentum v = mu * v - learning_rate * dx x += v # recall the RMSProp beta = 0.9 cache = cache * beta + (1-beta) * (dx**2) x += -learning_rate * dx / sqrt(cache + 1e-7) # Adam beta1 = 0.9 # decay hyparam for momentum beta1 = 0.995 # decay hyparam for momentum v = beta1 * v + (1-beta1) * dx cache = cache * beta2 + (1-beta2) * (dx**2) x += -learning_rate * v / sqrt(cache + 1e-7) # commonly written as: beta1 = 0.9 beta1 = 0.995 m = beta1 * m + (1-beta1) * dx v = beta2 * v + (1-beta2) * (dx**2) x += -learning_rate * m / sqrt(v + 1e-7) ### we often do m /= (1-beta1)**t v /= (1-beta2)**t # \u0026gt; where t is the iteration in epoch Dividing by (1 - beta1)^t and (1 - beta2)^t corrects these biases, especially at the beginning of training when t is small. This bias correction helps to make the estimates of m and v more accurate, leading to better optimization performance, particularly in the initial stages of training. ~ Karpathy.\nAdamW In AdamW, weight decay is incorporated directly into the update step for the parameters. Updated code for AdamW:\nm = beta1 * m + (1-beta1) * dx v = beta2 * v + (1-beta2) * (dx**2) x += -learning_rate * (m / (sqrt(v) + 1e-7) + weight_decay * x) AdamScheduleFree By the time I was writing this blog, AdamScheduleFree was hot topic of research in meta\nfor each parameter group: retrieve beta1, beta2, learning rate (lr), weight decay, and epsilon (eps) for each parameter p in the group: retrieve or initialize m and v in state dictionary # Update m and v m = beta1 * m + (1 - beta1) * gradient v = beta2 * v + (1 - beta2) * (gradient ** 2) # Compute update update = -lr * m / (sqrt(v) + eps) + weight_decay * parameter # Update parameter parameter += update # Save updated m and v back to state save m and v in state dictionary the state dictionary is used to store the momentum (m) and the exponentially weighted moving average of squared gradients (v) for each parameter. After each update step, these values are updated and stored back into the state dictionary for later use in subsequent optimization steps. The state dictionary is typically managed by the optimizer and is associated with each parameter being optimized.\nSecond order methods A second-order group of methods for optimization in deep learning is based on Newton\u0026rsquo;s method, which treats the following update. It uses a Hessian matrix, which is a matrix of second-order partial derivatives of the function. In particular, multiplying by the inverse Hessian leads to optimization taking more aggressive steps. So, if stochastic gradient descent techniques are used to find local minima one by one and piece by piece, the second-order method directly jumps to the local minimum of the surrounding area. However, the update above is impractical for most deep learning applications because computing such a large Hessian matrix and then inverting it is very costly, both in space and time. For instance, a neural network with 1 million parameters would have a Hessian matrix of size 1 million by 1 million, occupying approximately 3725 gigabytes of RAM. Hence, a large variety of approximation techniques have been developed that seek to approximate the inverse. Among these, the most popular is L-BFGS.\nIn practice, it is currently not common to see Lbfgs or similar second order methods applied to large scale deep learning and convolutional neural networks instead sgd variants based on nested momentum are more standard Because they\u0026rsquo;re simpler and scale more easily.\nSources Inspired from cs231n \u0026ldquo;Lecture 6: Neural Networks Part 3 / Intro to ConvNets\u0026rdquo;. cs231n Notes ","permalink":"https://akash5100.github.io/posts/2024-04-12-optimization_techniques/","summary":"\u003cp\u003eOnce the gradient is computed, it is then used to update the parameters. There are several approaches to perform the update.\u003c/p\u003e\n\u003c!-- TOC --\u003e\n\u003ch3 id=\"table-of-contents\"\u003eTable of contents\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#first-ordersgd-momentum-nesterov-momentum\"\u003eFirst order(SGD), momentum, nesterov momentum\u003c/a\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#stochastic-gradient-descent\"\u003eStochastic gradient descent\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#momentum-update\"\u003eMomentum Update\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#nesterov-momentum-nag\"\u003eNesterov Momentum (NAG)\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#per-parameter-adaptive-learning-rates-adagrad-rmsprop\"\u003ePer-parameter adaptive learning rates (Adagrad, RMSProp)\u003c/a\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#adagrad\"\u003eAdagrad\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#rmsprop-gef-hinton\"\u003eRMSProp (Gef. Hinton)\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#adam\"\u003eAdam\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#adamw\"\u003eAdamW\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#adamschedulefree\"\u003eAdamScheduleFree\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#second-order-methods\"\u003eSecond order methods\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#sources\"\u003eSources\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"first-ordersgd-momentum-nesterov-momentum\"\u003eFirst order(SGD), momentum, nesterov momentum\u003c/h2\u003e\n\u003ch3 id=\"stochastic-gradient-descent\"\u003eStochastic gradient descent\u003c/h3\u003e\n\u003cp\u003eThis is the \u003cstrong\u003evanilla\u003c/strong\u003e update, the simplest form of update that changes the parameters along the negative gradient direction, as the gradient indicates the direction of increase, whereas we usually wish to minimize a loss. Assuming a vector of parameters \u003ccode\u003ex\u003c/code\u003e and the gradient \u003ccode\u003edx\u003c/code\u003e, the simplest form of the update is as follows.\u003c/p\u003e","title":"Parameter updates"},{"content":"Relying on any library is not good; therefore, this time, removing this black box and understand what\u0026rsquo;s happening inside. Writing a backward pass manually would be helpful. Backpropagation is not something that works magically or automatically if you are hoping to debug. In other words, it is easy to fall into the trap of abstracting away the learning process, believing that you can simply stack arbitrary layers together and backprop will magically make them work on your data. So let\u0026rsquo;s look at a few explicit examples where this is not the case in quite unintuitive ways.\nThis blog post is clearly set of personal notes for the notes by Andrej Karpathy\u0026rsquo;s cs231n lectures.123\nTable of contents Simple expressions and interpretation of the gradient Compound Expressions with Chain Rule Intutive understanding of backpropogation Backprop in practice: Staged computation Gradients for vectorized operations Gradient checking Sources Simple expressions and interpretation of the gradient Let's start with simple expressions to develop the notation and conventions for more complex ones. Consider a straightforward multiplication function of two numbers \\( f(x,y) = xy \\). It's a matter of simple calculus to derive the partial derivative for either input: \\[ \\frac{\\partial f}{\\partial x} = y \\quad \\text{and} \\quad \\frac{\\partial f}{\\partial y} = x \\]Interpretation: Derivatives indicate the rate of change of a function with respect to that variable around an infinitesimally small region near a particular point:\n\\[ \\frac{df(x)}{dx} = \\lim_{h \\to 0} \\frac{f(x+h) - f(x)}{h} \\]A technical note is that the division sign on the left-hand side is not a division; it indicates that the operator \\( \\frac{d}{dx} \\) is being applied to the function \\( f \\), returning a different function (the derivative). When \\( h \\) is very small, the function is well-approximated by a straight line, and the derivative is its slope. In other words, the derivative on each variable tells you the sensitivity of the whole expression to its value. For example, if \\( x = 4, y = -3 \\), then \\( f(x,y) = -12 \\) and the derivative on \\( x \\), \\( \\frac{\\partial f}{\\partial x} = -3 \\), indicates that increasing the value of \\( x \\) by a tiny amount would decrease the whole expression by three times that amount. Similarly, since \\( \\frac{\\partial f}{\\partial y} = 4 \\), increasing \\( y \\) by a small amount would increase the output of the function by four times that amount. The Gradient: The gradient \\( \\nabla f \\) is the vector of partial derivatives, so we have \\( \\nabla f = [\\frac{\\partial f}{\\partial x}, \\frac{\\partial f}{\\partial y}] = [y, x] \\). Though technically a vector, we\u0026rsquo;ll often use terms like \u0026ldquo;the gradient on \\( x \\)\u0026rdquo; instead of the technically correct phrase \u0026ldquo;the partial derivative on \\( x \\)\u0026rdquo; for simplicity.\nWe can also derive the derivatives for the addition operation:\n\\[ f(x,y) = x+y \\quad \\rightarrow \\quad \\frac{\\partial f}{\\partial x} = 1 \\quad \\text{and} \\quad \\frac{\\partial f}{\\partial y} = 1 \\]That is, the derivative on both \\( x \\) and \\( y \\) is one, regardless of their values. This makes sense since increasing either \\( x \\) or \\( y \\) would increase the output of \\( f \\), and the rate of increase would be independent of their actual values.\nLet\u0026rsquo;s consider the max operation:\n\\[ f(x,y) = \\max(x,y) \\quad \\rightarrow \\quad \\frac{\\partial f}{\\partial x} = \\begin{cases} 1 \u0026 \\text{if } x \\geq y \\\\ 0 \u0026 \\text{otherwise} \\end{cases} \\quad \\text{and} \\quad \\frac{\\partial f}{\\partial y} = \\begin{cases} 0 \u0026 \\text{if } x \\geq y \\\\ 1 \u0026 \\text{otherwise} \\end{cases} \\]Here, the (sub)gradient is 1 on the input that was larger and 0 on the other input. Intuitively, if \\( x = 4 \\) and \\( y = 2 \\), then the max is 4, and the function is not sensitive to the setting of \\( y \\). That is, if we were to increase it by a tiny amount \\( h \\), the function would keep outputting 4, and therefore the gradient is zero—there\u0026rsquo;s no effect. Of course, if we were to change \\( y \\) by a large amount (e.g., larger than 2), then the value of \\( f \\) would change, but the derivatives tell us nothing about the effect of such large changes on the inputs of a function; they are only informative for tiny, infinitesimally small changes, as indicated by the \\( \\lim_{h \\to 0} \\) in their definition.\nCompound Expressions with Chain Rule Let's now consider more complicated expressions that involve multiple composed functions, such as \\( f(x,y,z) = (x+y)z \\). While this expression is simple enough to differentiate directly, we'll take a particular approach that will be helpful for understanding the intuition behind backpropagation. Specifically, note that this expression can be broken down into two expressions: \\( q = x+y \\) and \\( f = qz \\). Moreover, we know how to compute the derivatives of both expressions separately, as seen in the previous section. \\( f \\) is just the multiplication of \\( q \\) and \\( z \\), so \\( \\frac{\\partial f}{\\partial q} = z \\), \\( \\frac{\\partial f}{\\partial z} = q \\), and \\( q \\) is the addition of \\( x \\) and \\( y \\) so \\( \\frac{\\partial q}{\\partial x} = 1 \\), \\( \\frac{\\partial q}{\\partial y} = 1 \\). However, we don’t necessarily care about the gradient on the intermediate value \\( q \\)—the value of \\( \\frac{\\partial f}{\\partial q} \\) is not useful. Instead, we are ultimately interested in the gradient of \\( f \\) with respect to its inputs \\( x, y, z \\). The chain rule tells us that the correct way to \u0026ldquo;chain\u0026rdquo; these gradient expressions together is through multiplication. For example, \\( \\frac{\\partial f}{\\partial x} = \\frac{\\partial f}{\\partial q} \\frac{\\partial q}{\\partial x} \\). In practice, this is simply a multiplication of the two numbers that hold the two gradients. Let\u0026rsquo;s see this with an example:\n# inputs x = -2; y = 5; z = -4 # forward pass q = x + y # q becomes 3 f = q * z # f becomes -12 # backward pass (backpropagation) in reverse order: # First backprop through f = q * z dfdz = q # df/dz = q, so gradient on z becomes 3 dfdq = z # df/dq = z, so gradient on q becomes -4 dqdx = 1.0 dqdy = 1.0 # Now backprop through q = x + y dfdx = dfdq * dqdx # The multiplication here is the chain rule! dfdy = dfdq * dqdy We are left with the gradient in the variables \\([ \\frac{\\partial f}{\\partial x}, \\frac{\\partial f}{\\partial y}, \\frac{\\partial f}{\\partial z} ]\\), which tell us the sensitivity of the variables \\( x, y, z \\) on \\( f \\). This is the simplest example of backpropagation. Going forward, we will use a more concise notation that omits the \\( df \\) prefix. For example, we will simply write \\( dq \\) instead of \\( \\frac{\\partial f}{\\partial q} \\), and always assume that the gradient is computed on the final output. Image is from cs231n notes by Karpathy. Visualization of above example as circuit. The forward pass, left to right, (shown in green) computes values from inputs to output. The backward pass, right to left, then performs backpropogation which starts at the end and recursively applies the chain rule to compute the gradients (shown in red) all the way to the inputs. Notice, in the addition node, the gradient (-4) flowed to both prior node equally, this is the basic idea of skip connections in ResNet. Intutive understanding of backpropogation Backpropogation is a local process. Every gate in a circuit diagram gets some inputs and can right away compute two things: 1. its output value and 2. the local gradient of its output with respect to its inputs. The gates can do this completely independentlty without being aware of any of the details full circuit. Once the forward pass is over, during backprop the gate will eventually learn about the gradients of its output value on the final output of the entire circuit. And with the chain rule you take the gradient and multiply it by the gradient of its outputwith respect to its inputs.\nThis extra multiplication (for each input) due to the chain rule can turn a single and relatively useless gate into a cog of a complex circuit such as an entire neural network.\nIn the above example image, the intuition behind how backpropagation works using an example of a simple circuit with an add gate and a multiply gate.\nThe add gate receives inputs [-2, 5] and computes an output of 3. Since it\u0026rsquo;s performing addition, its local gradient for both inputs is +1.\nThe rest of the circuit computes the final value, which is -12.\nDuring the backward pass, the add gate learns that the gradient for its output was -4. If we imagine the circuit as wanting to output a higher value, then the add gate \u0026ldquo;wants\u0026rdquo; its output to be lower due to the negative sign, with a force of 4.\nTo continue the chain rule and propagate gradients, the add gate multiplies the gradient (-4) to all of the local gradients for its inputs. This results in the gradient on both inputs (x and y) being -4.\nThis process has the desired effect: If the inputs were to decrease (in response to their negative gradients), the add gate\u0026rsquo;s output would decrease, causing the multiply gate\u0026rsquo;s output to increase.\nBackpropogation can thus be thought of as gates communicating to each other (through the gradient signal) whether they want their outputs to increase or decrease (and how strongly), so as to make the final output value higher. The derivative of the sigmoid function and its application in backpropagation for a neuron in a neural network.\nThe sigmoid function is defined as:\n\\[ \\sigma(x) = \\frac{1}{1 + e^{-x}} \\] The derivative of the sigmoid function with respect to its input **x** is derived as follows: \\[ \\begin{align*} \\sigma(x) \u0026= \\frac{1}{1 + e^{-x}} \\\\ \\frac{d\\sigma(x)}{dx} \u0026= \\frac{e^{-x}}{(1 + e^{-x})^2} \\\\ \u0026= \\frac{1}{1 + e^{-x}} \\cdot \\frac{e^{-x}}{1 + e^{-x}} \\\\ \u0026= (1 - \\sigma(x)) \\cdot \\sigma(x) \\end{align*} \\] This simplification of the gradient of the sigmoid function yields a straightforward expression: \\((1 - \\sigma(x)) \\cdot \\sigma(x)\\). In practical applications, this simplified expression is advantageous as it allows for efficient computation and reduces numerical issues.\nThe provided Python code demonstrates the backpropagation process for a neuron:\nw = [2, -3, -3] # assume some random weights and data x = [-1, -2] # Forward pass dot = w[0]*x[0] + w[1]*x[1] + w[2] f = 1.0 / (1 + math.exp(-dot)) # Sigmoid function # Backward pass through the neuron # starting from end ddot = (1 - f) * f # Gradient on dot variable, using the sigmoid gradient derivation dx = [w[0] * ddot, w[1] * ddot] # Backprop into x dw = [x[0] * ddot, x[1] * ddot, 1.0 * ddot] # Backprop into w # We\u0026#39;re done! We have the gradients on the inputs to the circuit Backprop in practice: Staged computation Suppose that we have a function of the form:\n\\[ f(x,y) = \\frac{x + \\sigma(y)}{\\sigma(x) + (x + y)^2} \\] To be clear, this function is completely useless and it’s not clear why you would ever want to compute its gradient, except for the fact that it is a good example of backpropagation in practice Here is the forward pass:\nx = 3 y = -4 # forward pass sigy = 1 / (1 + math.exp(-y)) # (1) num = x + sigy # (2) sigx = 1 / (1 + math.exp(-x)) # (3) xy = x + y # (4) xy2 = xy**2 # (5) den = sigx + xy2 # (6) invden = 1/den # (7) f = num * invden # (8) We have to compute gradients for sigy, num, sigx, xy, xy2, den, invden.\n# backprop f = num * invden dnum = invden # df/dnum = 1*invden + num*0 = invden dinvden = num # df/dinvden # backprop invden = 1/den dden = dinvden * (-1/(den**2)) # self-learned-rule-of-thumb! # for chain rule, instead of getting confused on # what to multiply because there are branches # instead multiply the d of LHS, in above the example # the function was # invdev = 1/den # we mul the dinvdev, which is dLHS. # backprop den = sigx + xy2 dsigx = dden * 1 dxy2 = dden * 1 # backprop xy2 = xy**2 dxy = dxy2 * (2 * xy) # backprop xy = x + y dx = dxy * 1 dy = dxy * 1 # backprop sigx = 1 / (1 + math.exp(-x)) dx += dsigx * ((1-sigx) * sigx) # Notice, +=, see notes below # backprop num = x + sigy dx += dnum * 1 dsigy = dnum * 1 # backprop sigy = 1 / (1 + math.exp(-y)) dy += dsigy * ((1-sigy) * sigy) # Done! Cache forward pass variables. To compute the backward pass it is very helpful to have some of the variables that were used in the forward pass. In practice you want to structure your code so that you cache these variables, and so that they are available during backpropagation. If this is too difficult, it is possible (but wasteful) to recompute them.\nGradients add up at forks. The forward expression involves the variables x,y multiple times, so when we perform backpropagation we must be careful to use += instead of = to accumulate the gradient on these variables (otherwise we would overwrite it). This follows the multivariable chain rule in Calculus, which states that if a variable branches out to different parts of the circuit, then the gradients that flow back to it will add.\nImage is from cs231n notes by Karpathy. An example circuit demonstrating the intuition behind the operations that backpropagation performs during the backward pass in order to compute the gradients on the inputs. Sum operation distributes gradients equally to all its inputs. Max operation routes the gradient to the higher input. Multiply gate takes the input activations, swaps them and multiplies by its gradient The add gate always takes the gradient on its output and distributes it equally to all of its inputs, regardless of what their values were during the forward pass. This follows from the fact that the local gradient for the add operation is simply +1.0, so the gradients on all inputs will exactly equal the gradients on the output because it will be multiplied by x1.0 (and remain unchanged). In the example circuit above, note that the + gate routed the gradient of 2.00 to both of its inputs, equally and unchanged. For this reason the add gate is also called gradient highway.1\nThe max gate routes the gradient. Unlike the add gate which distributed the gradient unchanged to all its inputs, the max gate distributes the gradient (unchanged) to exactly one of its inputs (the input that had the highest value during the forward pass). This is because the local gradient for a max gate is 1.0 for the highest value, and 0.0 for all other values. In the example circuit above, the max operation routed the gradient of 2.00 to the z variable, which had a higher value than w, and the gradient on w remains zero.1\nThe multiply gate is a little less easy to interpret. Its local gradients are the input values (except switched), and this is multiplied by the gradient on its output during the chain rule. In the example above, the gradient on x is -8.00, which is -4.00 x 2.00.1\nUnintuitive effects and their consequences.\nNotice that if one of the inputs to the multiply gate is very small and the other is very big, then the multiply gate will do something slightly unintuitive: it will assign a relatively huge gradient to the small input and a tiny gradient to the large input. Note that in linear classifiers where the weights are dot producted \\(w^T x_i\\) (multiplied) with the inputs, this implies that the scale of the data has an effect on the magnitude of the gradient for the weights. For example, if you multiplied all input data examples \\(x_i\\) by 1000 during preprocessing, then the gradient on the weights will be 1000 times larger, and you’d have to lower the learning rate by that factor to compensate. This is why preprocessing matters a lot, sometimes in subtle ways! And having intuitive understanding for how the gradients flow can help you debug some of these cases. Gradients for vectorized operations Gradients for vectorized operations extend concepts to matrix and vector operations, requiring attention to dimensions and transpose operations.\nMatrix-matrix multiplication presents a challenge: Forward pass:\nW = np.random.randn(5, 10) X = np.random.randn(10, 3) D = W.dot(X) Suppose we have the gradient on D:\ndD = np.random.randn(*D.shape) # same shape as D dW = dD.dot(X.T) # transpose of X dX = W.T.dot(dD) Tip: Utilize dimension analysis to derive gradient expressions. The resulting gradients must match the respective variable sizes. For instance, dW must match the size of W and depends on the matrix multiplication of X and dD.\nStart with small, explicit examples to derive gradients manually, then generalize to efficient, vectorized forms. This approach aids understanding and application of vectorized expressions.\nGradient checking TLDwrote; cs231n notes on gradient checking\nSources Inspired by CS231n notes\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nThis blog is written as notes to this youtube lecture.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nCS231n Lecture on backprop can be found here.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://akash5100.github.io/posts/2024-04-02-understanding_loss.backward/","summary":"\u003cp\u003eRelying on any library is not good; therefore, this time, removing this black box and understand what\u0026rsquo;s happening inside. Writing a backward pass manually would be helpful. Backpropagation is not something that works magically or automatically if you are hoping to debug. In other words, it is easy to fall into the trap of abstracting away the learning process, believing that you can simply stack arbitrary layers together and backprop will magically make them work on your data. So let\u0026rsquo;s look at a few explicit examples where this is not the case in quite unintuitive ways.\u003c/p\u003e","title":"Understanding loss.backward()"},{"content":"Table of contents Strange output of coupling Source Understanding why training deep neural networks can be fragile is crucial. Issues like dead neurons or saturation of non-linearity, and the vanishing or exploding gradients have caused problem for deep learning for years. However, there\u0026rsquo;s a beacon of hope that emerged around 2015: Batch Normalization.\nHere is a visualization of problem using graph. Say we have 1000 datapoints and each has 500 embeddings (latent factors).\nD = torch.randn(1000, 500) Y = torch.randn(1000, 1) * 0.1 And we have 10 layers each with 500 * 500 neurons.\nl1 = Linear(500, 500) l2 = Linear(500, 500) l3 = Linear(500, 500) l4 = Linear(500, 500) l5 = Linear(500, 500) l6 = Linear(500, 500) l7 = Linear(500, 500) l8 = Linear(500, 500) l9 = Linear(500, 500) l10 = Linear(500, 1) And we train it.\n... h1 = l1(D) h1 = h1.tanh() h2 = l2(h1) h2 = h2.tanh() ... h10 = l10(h9) h10 = h10.tanh() If we see the statistics of hidden layer\u0026rsquo;s activations, h1, h2, the standard deviation of the layer is getting close to one. Standard deviation is the distance between the mean and data point, which implies that the data is shrinking closer and closer to mean.\nAnd that\u0026rsquo;s what happens exactly:\nH0 -\u0026gt; mean: -0.0003, std: 0.2132 H1 -\u0026gt; mean: 0.0001, std: 0.0476 H2 -\u0026gt; mean: -0.0000, std: 0.0106 H3 -\u0026gt; mean: 0.0000, std: 0.0024 H4 -\u0026gt; mean: -0.0000, std: 0.0005 H5 -\u0026gt; mean: 0.0000, std: 0.0001 H6 -\u0026gt; mean: 0.0000, std: 0.0000 H7 -\u0026gt; mean: -0.0000, std: 0.0000 H8 -\u0026gt; mean: 0.0000, std: 0.0000 H9 -\u0026gt; mean: -0.0001, std: 0.0000 Visualization showing hidden activations being squashed, a phenomenon known as neuron saturation Activations collasping to 0 (if initialized small) and it will saturate at ends if initialized bigger The Batch Normalization paper marked a significant milestone in deep learning. It was the first normalization layer technique to tackle the fragility of deep neural net training. Released by the Google team, specifically Google DeepMind, this technique provided a groundbreaking solution to the problem of activation saturation in weights initialization.\nAs shown above in a tanh activation: if the weights are too large, activations saturate at 1 and -1, and if they\u0026rsquo;re too close to zero, they saturate in the middle. Neither scenario is ideal.\nImage from the original BatchNorm paper Batch Normalization paper is the first normalization layer technique that is used in deep learning. It addressed the problem of activations saturation in the weights that we initialize. We want the weights to be roughly Gaussian. The basic idea of this paper was \u0026ldquo;if we want the activation of the hidden state to be Gaussian, then why don\u0026rsquo;t we take that activation and normalize them to be Gaussian\u0026rdquo;. Sounds funny but it actually works, we take the mean and standard deviation of the hidden state activations and we subtract the mean from each data point and divide them with standard deviation normalizes them to be roughly Gaussian.\n# if \u0026#34;h2\u0026#34; is the activation of hidden layer \u0026#34;l2\u0026#34;, then h2 = l2(h1) # get the acts bx = b1(h2) # pass them to batchNorm layer h2 = h2.tanh() # non-linearity # The paper formula looks like xbs # mini batch xmean = xbs.mean(1, keepdim=True) # batch mean xvar = xbs.var(1, keepdim=True) # batch var h_normalized = (h2 - xmean) / (xvar + epsilon)**0.5 But this will take away the flexibilty of weight and it will never learn, to fix this we take this normalized hidden state and multiply them with learnable weights (called Gamma) and add bias called beta, This gives neural net the full ability to learn the new data points. The gain amplifies (visualize vertically in an x/y plane for understanding) and bias helps the neural net as an offset (visualize horizontally, as when we add something to a number it shiftes to-and-forth the number line).\n# initialization of Gamma and beta gamma = torch.ones(hidden_sz) beta = torch.zeros(hidden_sz) gamma * h_normalized + beta # similar to XW + B :) 3 linear layer with 3 batchnorm layer, slightly better acts? The problem of activation for large neural net will become very quickly intractable to keep activation of hidden Layer roughly Gaussian as to keep them unsaturated. But sprinkling multiple batch mobilization layer is easy and effective but the comes with a terrible cost.\nThe training the neural net with mini-batch, we did that for efficiency and training fast and parallelly. But in BatchNorm, we are again COUPLING each data points in batches in the forward and backward pass. Each logits is not just a function of previous hidden layer, but also the function of the data point that came in that particular Batch. To understand this, if we react for any one of those input example it is going to change slightly depending upon what the other input is in those batch. So, H will also change because it will be impacted by H-mean and H-Standard Deviation. It will slightly jitter the edge depending upon the value of mean and standard deviation.\nTLDR: As the mean and var are calculated along the dimension, this means we are again coupling the datapoint. So any changes in any one of the input in a batch will jitter the activation of the batchnorm layer.\nWe could think that this is a bug, but it\u0026rsquo;s actually good for training and neural net because it acts as a regularizer and kinda adds noise to the input data and sensitivity to weight initialization. It prevents overfitting and also helps to generalize (it jitters the input data. Because each input data is subtracted by mean and then divided by standard deviation calculated across the batch for each data point).\nIt works so well. which made it hard to move on to different techniques because no one likes the property of coupling batches but it is the first Normalization layer technique. It has regularization effect, stable training. It worked quite well. Because of The regularization effect.\nStrange output of coupling Let\u0026rsquo;s say we trained a model and we want to deploy how can we do that if in the forward pass expects a mean and standard deviation of batch? That means it expects a batch as an input and not a data point. So the authors of the research paper proposed this solution:\nOption 1.\nCalibrate mean and standard deviation of training data after training as an additional step (between training and validation)\nOption 2.\nKeep track of standard deviation and mean in a variable in the training Loop. (called Running variable)\nrunning_mean = 0.999 * running_mean + 0.001 * xmean running_std = 0.999 * running_std + 0.001 * xstd Note: There is a epsilon. In the paper and we add that epsilon when we calculate the standard deviation. We add that epsilon to variance, in the denominator. To add stability/safety in the calculation.\nSecond note: We don\u0026rsquo;t need bias for the layer before a batch normalization layer. Because in the next step we take the mean of the activations and subtract it. It basically cancels out. The bias is useless. This don\u0026rsquo;t break anything but just waste of computation. Because it never take parts in gradients. So whenever in a neural net if you are using batch normalization, in the layer prior to batch normalization layer we set the bias to False.\nHere is the full implementation of multi layer neural net with batch normalization layer:\nclass Linear: def __init__(self, fan_in, fan_out, bias=True): self.weight = torch.randn((fan_in, fan_out)) * 0.1 # / fan_in**0.5 # note: kaiming init self.bias = torch.zeros(fan_out) if bias else None def __call__(self, x): self.out = x @ self.weight if self.bias is not None: self.out += self.bias return self.out def parameters(self): return [self.weight] + ([] if self.bias is None else [self.bias]) class BatchNorm1d: def __init__(self, dim, momentum=0.1, eps=1e-5): self.training = True self.eps = eps self.momentum = momentum self.gamma = torch.ones(dim) self.beta = torch.zeros(dim) self.running_mean = torch.zeros(dim) self.running_var = torch.ones(dim) def __call__(self, x): if self.training: if x.ndim == 2: dim = 0 elif x.ndim == 3: dim = (0,1) xmean = x.mean(dim, keepdim=True) xvar = x.var(dim, unbiased=True, keepdim=True) #Notice: unbiased=True # In statistics, Bessel\u0026#39;s correction is the use of n − 1 instead of n in the formula for the sample variance and sample standard deviation else: xmean = self.running_mean xvar = self.running_var dx = (x - xmean)/torch.sqrt(xvar + self.eps) self.out = self.gamma * dx + self.beta if self.training: with torch.no_grad(): self.running_mean = (1.0 - self.momentum) * self.running_mean + self.momentum * xmean self.running_var = (1.0 - self.momentum) * self.running_var + self.momentum * xvar return self.out def parameters(self): return [self.gamma, self.beta] class Tanh: def __call__(self, x): self.out = torch.tanh(x) return self.out def parameters(self): return [] # Hyperparameters n = 3 # trigram model emb_sz = 10 n_hidden = 100 vocab_sz = len(chars) # it\u0026#39;s 27 # ----------------------------------------------- C = torch.randn(vocab_sz, emb_sz, generator=g) layers = [ Linear(emb_sz*n, n_hidden, bias=False), BatchNorm1d(n_hidden), Tanh(), Linear(n_hidden, n_hidden, bias=False), BatchNorm1d(n_hidden), Tanh(), Linear(n_hidden, n_hidden, bias=False), BatchNorm1d(n_hidden), Tanh(), Linear(n_hidden, n_hidden, bias=False), BatchNorm1d(n_hidden), Tanh(), Linear(n_hidden, n_hidden, bias=False), BatchNorm1d(n_hidden), Tanh(), Linear(n_hidden, vocab_sz, bias=False), BatchNorm1d(vocab_sz) ] # ----------------------------------------------- with torch.no_grad(): layers[-1].gamma *= 0.1 for l in layers[:-1]: if isinstance(l, Linear): l.w *= 1.0 parameters = [C] + [p for l in layers for p in l.parameters()] print(sum(p.nelement() for p in parameters)) for p in parameters: p.requires_grad = True Training log:\nepoch 0/20000 | loss: 3.3056 | perx: 27.2652 epoch 100/20000 | loss: 3.2400 | perx: 25.5348 epoch 200/20000 | loss: 3.1552 | perx: 23.4585 epoch 300/20000 | loss: 3.1343 | perx: 22.9729 epoch 400/20000 | loss: 2.9765 | perx: 19.6192 epoch 500/20000 | loss: 3.0426 | perx: 20.9598 epoch 600/20000 | loss: 2.9243 | perx: 18.6212 epoch 700/20000 | loss: 2.9431 | perx: 18.9746 epoch 800/20000 | loss: 2.7778 | perx: 16.0831 epoch 900/20000 | loss: 2.7860 | perx: 16.2167 epoch 1000/20000 | loss: 2.7030 | perx: 14.9249 epoch 1100/20000 | loss: 2.8019 | perx: 16.4762 epoch 1200/20000 | loss: 2.7259 | perx: 15.2699 epoch 1300/20000 | loss: 2.7281 | perx: 15.3041 epoch 1400/20000 | loss: 2.6197 | perx: 13.7310 epoch 1500/20000 | loss: 2.6625 | perx: 14.3316 epoch 1600/20000 | loss: 2.6108 | perx: 13.6103 epoch 1700/20000 | loss: 2.7052 | perx: 14.9580 epoch 1800/20000 | loss: 2.6034 | perx: 13.5096 epoch 1900/20000 | loss: 2.5848 | perx: 13.2610 epoch 2000/20000 | loss: 2.1021 | perx: 8.1833 Source Inspired from this lecture. Complete implementation: github link ","permalink":"https://akash5100.github.io/posts/2024-03-10-batch_normalization/","summary":"\u003ch3 id=\"table-of-contents\"\u003eTable of contents\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#strange-output-of-coupling\"\u003eStrange output of coupling\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#source\"\u003eSource\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eUnderstanding why training deep neural networks can be fragile is crucial. Issues like dead neurons or saturation of non-linearity, and the vanishing or exploding gradients have caused problem for deep learning for years. However, there\u0026rsquo;s a beacon of hope that emerged around 2015: Batch Normalization.\u003c/p\u003e\n\u003cp\u003eHere is a visualization of problem using graph. Say we have 1000 datapoints and each has 500 embeddings (latent factors).\u003c/p\u003e","title":"Batch Normalization"},{"content":"Table of contents Peeking inside a hidden layer But how can we decide which number to multiply? (Initialization) Summary I implemented some techniques used in a research paper for n-gram language modeling. It is a simple MLP-based trigram language model that takes three characters\u0026rsquo; tokens as input, passes them into an embedding layer, then a hidden linear layer, and finally predicts the next character\u0026rsquo;s token.\nHere are the hyperparameters I used:\nn = 3 #trigram model emb_sz = 2 #latent factor n_hidden = 100 vocab_sz = len(chars) # it\u0026#39;s 27 Here is the architecture:\nSource: https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf\nC = torch.randn(vocab_sz, emb_sz) w1 = torch.randn(n*emb_sz, n_hidden) b1 = torch.randn(n_hidden) w2 = torch.randn(n_hidden, vocab_sz) b2 = torch.randn(vocab_sz) # Setting up gradient requirements parameters = [C, w1, b1, w2, b2] for p in parameters: p.requires_grad = True # --------------------------------------------------- # training epoch = 200_000 bs = 32 # batch sz for i in range(epoch): # mini batch idx = torch.randint(0, xtrain.shape[0], size=(bs,)) xbs, ybs = xtrain[idx], ytrain[idx] # batch # forward pass emb = C[xbs] hpreact = emb.view(emb.shape[0], -1) @ w1 + b1 h = torch.tanh(hpreact) logits = h @ w2 + b2 loss = F.cross_entropy(logits, ybs) perplexity = torch.exp(loss) # backward pass for p in parameters: p.grad = None loss.backward() # update lr = 0.01 if i \u0026lt; 100_000 else 0.001 for p in parameters: p.data -= p.grad.data * lr # track lossi.append(loss.log10().item()) if i%10000 == 0: print(f\u0026#34;epoch: {i:7d}/{epoch:7d} | loss: {loss.item():.4f} | perplexity: {perplexity.item():.4f}\u0026#34;) epoch: 0/ 200 | loss: 9.9113 | perplexity: 20157.0859 epoch: 10/ 200 | loss: 4.8961 | perplexity: 133.7642 epoch: 20/ 200 | loss: 3.5018 | perplexity: 33.1759 epoch: 30/ 200 | loss: 3.0638 | perplexity: 21.4093 epoch: 40/ 200 | loss: 2.8681 | perplexity: 17.6032 epoch: 50/ 200 | loss: 2.8150 | perplexity: 16.6932 Firstly, I want to explain why we need to initialize the weights of our model with good values. As observed, the initial loss is very high, and then subsequent losses are very low. The weights we initialized are significantly incorrect, as some are very confidently wrong while others can be overly confident. This necessitates spending time and computational resources to correct them before starting unbiased learning. Thus, improving parameter initialization can help us skip this unnecessary step, which otherwise wastes energy and computation.\nHockey stick loss which is due to bad initial weights initialization Let\u0026rsquo;s visualize the logits in a histogram:\nplt.hist(logits.tolist(), 50); Logits values are too big to converge The mean is near 100 or so. We can achieve our objectives if the logits obtained in the final layer are low instead of very high. To accomplish this, we can initialize the bias of the final layer to zero, allowing the neural network to learn how to offset it by itself, and initialize the weights with small numbers.\nC = torch.randn(vocab_sz, emb_sz) w1 = torch.randn(n*emb_sz, n_hidden) b1 = torch.randn(n_hidden) w2 = torch.randn(n_hidden, vocab_sz) * 0.01 b2 = torch.randn(vocab_sz) * 0 plt.hist(logits[0].tolist(), 50); Logits values are near zero We should not initialize weights with zero, just as the bias, because this would cause all neurons to update in the same way, resulting in symmetric and redundant weights throughout the training process.\nisn't this better than a hockey stick loss? Peeking inside a hidden layer The second issue I want to address concerns the hidden layer. Consider the forward pass\nidx = torch.randint(0, xtrain.shape[0], size=(bs,)) emb = C[xtrain[idx]] embinp = emb.view(emb.shape[0], -1) h = torch.tanh(embinp @ w1 + b1) logits = h @ w2 + b2 loss = F.cross_entropy(logits, ytrain[idx]) In this code snippet, bs represents the batch size, C is the embedding matrix, h is the output of the hidden layer (shaped BSxN_hidden), and w1, b1, w2, b2 are the linear layer\u0026rsquo;s weights and biases, with cross-entropy loss calculated at the end. Refresher: cross-entropy loss involves calculating the softmax of logits and then calculating its log likelihood, commonly used as a loss function in classification tasks. Everything seems fine, but if we examine the hidden layer\u0026rsquo;s numbers, something is amiss. Here\u0026rsquo;s the training loss:\nepoch: 0/ 200000 | loss: 2.7297 | perplexity: 15.3288 epoch: 10000/ 200000 | loss: 2.5840 | perplexity: 13.2500 Perplexity, is the exponential of cross-entropy loss (loss.exp()). The exponential of loss, which changes with very few points, is much easier to understand when spread using exponential.\nA deeper problem lurks within this neural net and its initialization. To visualize this, let\u0026rsquo;s flatten the hidden layer (h) and plot it in a histogram. We observe that most numbers lie near 1 and -1 instead of in between, which occurs because tanh is a squashing function that compresses everything into its flat region at 1 or -1. This means that these neurons are either highly active or highly inactive. As Andrej Karpathy pointed out, \u0026ldquo;if you\u0026rsquo;re not very experienced with gradient descent, you might think this is okay, but if you are experienced with the black magic of gradient descent, then you might be sweating already\u0026rdquo;.\n\u0026gt;\u0026gt;\u0026gt;h tensor([[-1.0000, 0.9997, -1.0000, ..., 1.0000, -1.0000, 0.2273], [ 0.3959, 0.9912, -1.0000, ..., -0.9820, 0.6797, -0.9998], [ 0.8695, -0.9592, -0.4954, ..., 0.4124, -0.8499, 0.9920], ..., [ 0.9987, 0.9952, -0.9996, ..., 0.8290, -0.9994, -1.0000], [-0.9993, 0.9991, -1.0000, ..., -0.6726, -1.0000, 1.0000], [ 0.9954, -0.2170, -1.0000, ..., -0.9923, 0.0163, -0.8219]], Most of them are near 1 and -1.\nplt.hist(h.view(-1).tolist(), 50); Saturated Tanh We can actually do the same small number multiplication to the W1 and B1 which gives us hidden state (h).\nWe can mitigate this issue by applying the same small number multiplication to W1 and B1, which produces a more desirable hidden state (h).\nw1 = torch.randn(n*emb_sz, n_hidden) * 0.1 b1 = torch.randn(n_hidden) * 0.01 #later plt.hist(h.view(-1).tolist(), 50); better Tanh This is because, as we can observe, the differentiation of tanh is as follows:\nx = data t = (math.exp(2*x) - 1)/(math.exp(2*x) + 1) out = t # backward grad = (1 - t**2) * out.grad Regardless of whether we increase or decrease the input data to the tanh function, 1 - t^2 would be zero, resulting in zero gradients. Notice that the gradients always decrease. If the weight initialization is poor and all data points lie above 1 or below -1, they will all be squashed into the flat region of 1 and -1. Consequently, the gradients will be destroyed. To further illustrate this, let\u0026rsquo;s visualize our hidden layer using matplotlib.\nWe take the absolute value of each value and observe how frequently they are fully activated.\nplt.figure(figsize=(16,16)) plt.imshow((h.abs()\u0026lt;0.99).tolist(), cmap=\u0026#39;gray\u0026#39;) Visualization of Hidden Layer Activations with Tanh Function In this plot, white cells indicate dead neurons. If a complete column is white, it will completely destroy any gradients flowing through it, and weights beyond that column will never learn because they will never receive gradients.\nSimilar issues occur with other activation functions, such as Sigmoid, Relu, and ELU, because they also have a squashing plane. A research paper (https://arxiv.org/pdf/1502.01852.pdf) extensively studied this phenomenon, particularly in the context of Relu and PRelu in convolutional neural nets. They found that if a Relu neuron never activates for any input in the dataset, its weights and biases will never receive gradients, rendering it ineffective. Conversely, if it receives excessively large gradients (possible with high learning rates), it will be knocked out of the data manifold, preventing it from learning from the rest of the training data.\nActivations They also proposed a solution to this problem: when initializing weights and biases in a Gaussian distribution, if the resulting standard deviation shifts too much, we need to retain the original standard deviation. We can achieve this by multiplying the weight matrix by a small number during initialization. Surprisingly, the number we multiply is the new standard deviation for that matrix.\nX1 = torch.randn(1000,100) \u0026gt;\u0026gt;\u0026gt; X2 = torch.randn(100,1000) \u0026gt;\u0026gt;\u0026gt; X1.mean(), X1.std() (tensor(-0.0070), tensor(1.0002)) \u0026gt;\u0026gt;\u0026gt; X2.mean(), X2.std() (tensor(-0.0032), tensor(1.0020)) \u0026gt;\u0026gt;\u0026gt; (X1@X2).mean() tensor(-0.0096) \u0026gt;\u0026gt;\u0026gt; (X1@X2).std() tensor(10.0116) \u0026gt;\u0026gt;\u0026gt; X2 = torch.randn(100,1000) * 0.2 \u0026gt;\u0026gt;\u0026gt; (X1@X2).mean() tensor(0.0020) \u0026gt;\u0026gt;\u0026gt; (X1@X2).std() tensor(2.0032) So, the standard deviation of the matrix changes based on the number we multiply.\nBut how can we decide which number to multiply? It turns out there\u0026rsquo;s a mathematical principle. For example, in Relu, half of the data is discarded to 0 for any input in the dataset. We can then amplify the remaining half of the data with a gain. Most researchers currently use the square root of the fan-in to multiply with the weights (as mentioned above). The fan-in represents the input features of a layer.\nw1 = torch.randn(n*emb_sz, n_hidden) * (n*emb_sz)**0.5 b1 = torch.randn(n_hidden) # for w1: n*emb_sz is fan-in (feature in) # x**0.5 is handy for calculating the square root Another popular initialization method is Kaiming initialization, based on the same theory, but with different gains for each activation function. Here is a link to the documentation: https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.kaiming_normal_\nPyTorch calculates the gain as follows: https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.calculate_gain\nA gain might be necessary because activation functions like Relu and Tanh squash the input. Researchers today typically use the square root of the fan-in to multiply with the weights, as demonstrated above. Because there are other normalization techniques such as batch normalization. instance normalization, layer normalization, group normalization and stronger optimization algorithms besides stochastic gradient descent, such as RMS Prop and Adam.\nSummary The activation function squashes the weights, so we should be concerned about how we initialize them. For example, in the tanh function, we don\u0026rsquo;t want them to saturate to one or negative one. To maintain good performance, we normalize them with a gain, which can typically be found as the square root of the fan-in. There are additional mathematical principles to consider, such as the Kaiming initialization, which also utilizes a gain multiplied by a constant for different activations like Sigmoid, Linear, and ReLU.\n","permalink":"https://akash5100.github.io/posts/2024-03-04-dead_neurons/","summary":"\u003ch3 id=\"table-of-contents\"\u003eTable of contents\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#peeking-inside-a-hidden-layer\"\u003ePeeking inside a hidden layer\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#but-how-can-we-decide-which-number-to-multiply\"\u003eBut how can we decide which number to multiply? (Initialization)\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#summary\"\u003eSummary\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eI implemented some techniques used in a research paper for n-gram language modeling. It is a simple MLP-based trigram language model that takes three characters\u0026rsquo; tokens as input, passes them into an embedding layer, then a hidden linear layer, and finally predicts the next character\u0026rsquo;s token.\u003c/p\u003e\n\u003cp\u003eHere are the hyperparameters I used:\u003c/p\u003e","title":"Dead Neurons"},{"content":"Table of contents Reset Gate Update Gate Memory Cell Final Memory Summary Sources GRU was first proposed in 2014 (RNN was in 1986, LSTM in 1995). GRU raised the question of whether we need to be that flexible like LSTM to learn the sequence. GRU is less flexible than LSTM but it is good enough for sequence learning.\nGRU redesigned the LSTM cell by introducing reset gate, update gate, and new memory cell; therefore, the number of gates were reduced from four to three. It was empirically shown in (Chung et al., 2014) that the performance of LSTM improves by using GRU cells. Later in 2017, the GRU was further simplified by merging the reset and update gates into a forget gate (Heck \u0026amp; Salem, 2017). Nowadays, GRU is the most commonly used LSTM structure.\nGRU simplified the LSTM cell in order to make the flexibility of LSTM.\nabove, both h(t-1) are same\u0026ndash; to simplify the arrows\nINSERT CODE Reset Gate The reset gate considers the effect of the input x (t) and the previous hidden state h (t-1) and outputs the signal:\nr = sig(W * h(t-1) + U * x(t) + b)\nwhere W and U are trainable weight matrix and the Bias b. It controls the amount of forgetting/resetting the previous information with respect to the new-coming information.\nUpdate Gate This gate also takes the input x (t) and the previous hidden state h (t-1) and outputs the signal:\nz = sig(W * h(t-1) + U * x(t) + b)\nIt controls the amount of using the new input data for updating the cell by the coming information of sequence.\nMemory Cell This gate takes the input at current time slot, x(t), and the hidden state of the last time slot, h(t−1) and outputs the signal:\nH = tanh(W * (r * h(t-1)) + U * x(t) + b)\nwhere W, U, and the bias b are the learnable weights for the new memory cell.\nThis gate considers the effect of the input and the previous hidden state to represent the new information of current input.\nThe new memory cell in the GRU cell is similar to the new memory cell in the LSTM cell. Note that, in the LSTM cell, the hidden state and the new memory cell were different; however, the hidden state of the GRU cell replaces the new memory signal in the LSTM cell.\nFinal Memory After the computations of outputs of the update gate (z) and the new memory cell (H), we calculate the final memory or the hidden state (h(t)).\nh(t) = (1-z) * h(t-1) + z * H Where:\nh(t−1) is the previous hidden state. z is the output of the update gate. H is the output of the new memory cell. This equation combines the previous hidden state with the new memory cell based on the output of the update gate.\nIf the update gate outputs a value close to 1, indicates high activations. This means that the model has decided to rely more on the new input data for updating the hidden state based on the current input.\nWhen the output of the update gate (z) is close to 0, it means that the model has decided to rely more on the previous hidden state h(t−1) and less on the new input data x(t) for updating the cell state h(t). In this case 1 -z is close to 1, indicating that the model is giving less weight to the new input data and more to the previous hidden state.\nSo, to calculate the final memory, we multiply the previous hidden state by (1−z) to control the amount of information carried over from the previous time step, and add z∗H to incorporate the new information based on the update gate\u0026rsquo;s decision.\nSummary How it improved LSTM?\nGRU (Gated Recurrent Unit) improved LSTM (Long Short-Term Memory) by simplifying the architecture and being faster to train. GRU has fewer gates and parameters compared to LSTM, making it more efficient and easier to implement.\nHowever, LSTM is considered more powerful and flexible than GRU, capable of capturing long-term dependencies better. While GRU is faster, it may not perform as well as LSTM in tasks requiring extensive memory retention. Both models have their strengths and are used based on the specific requirements of the task at hand. Researchers and engineers often experiment with both LSTM and GRU to determine which one suits their use case best, as there isn\u0026rsquo;t a clear winner between the two. 1\nIs it used widely than LSTM LSTM might still be preferred over GRU in some cases:\nComplexity: While GRU is simpler in architecture, LSTM\u0026rsquo;s additional complexity may allow it to capture more intricate patterns in the data, especially in tasks requiring longer-term memory. Performance: In certain tasks or datasets, LSTM may outperform GRU in terms of accuracy or convergence speed. Familiarity: LSTM has been around for longer and is more widely used and understood in the research community and industry. As a result, researchers and practitioners may default to LSTM due to familiarity. An analogy to understand + and x of activation matrices:\nIn LSTM and GRU networks, adding (+) matrices is like mixing colors to create new shades, while multiplying (*) matrices is like adjusting the brightness or intensity of those colors. Sources Stackexchange: GRU vs LSTM\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://akash5100.github.io/posts/2024-02-16-gru/","summary":"\u003ch3 id=\"table-of-contents\"\u003eTable of contents\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#reset-gate\"\u003eReset Gate\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#update-gate\"\u003eUpdate Gate\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#memory-cell\"\u003eMemory Cell\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#final-memory\"\u003eFinal Memory\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#summary\"\u003eSummary\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#sources\"\u003eSources\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eGRU was first proposed in 2014 (RNN was in 1986, LSTM in 1995).\nGRU raised the question of whether we need to be that flexible like LSTM to learn the sequence. GRU is less flexible than LSTM but it is good enough for sequence learning.\u003c/p\u003e\n\u003cp\u003eGRU redesigned the LSTM cell by introducing reset gate, update gate, and new memory cell; therefore, the number of gates were reduced from four to three. It was empirically shown in (Chung et al., 2014) that the performance of LSTM improves by using GRU cells. Later in 2017, the GRU was further simplified by merging the reset and update gates into a forget gate (Heck \u0026amp; Salem, 2017). Nowadays, GRU is the most commonly used LSTM structure.\u003c/p\u003e","title":"Gated Recurrent Unit (GRU)"},{"content":"Table of contents Regularizing LSTM 1. Dropout 2. Activation Regularization and Temporal Activation Regularization Gradient Clipping (good to know) Summary LSTM were designed to deal with the issue of exploding or vanishing gradients in RNNs. This is challenging because, neural nets generally struggle to learn long-term dependencies.\nLSTM introduces gate mechanism and has 2 hidden state, that allows information to flow unmodified over many timesteps. Each LSTM cell has a set of gates (forget, input, cell/memory and out) that carefully regulate the information into and out of the cell.\nThe forget gate modulates what information gets discarded from the cell state. The input gate decides what new information gets stored in the cell state. And the output gate decides what information propagates to the next steps.\nThis gated mechanism gives the LSTM cell explicit control over what is preserved, changed, or forgotten in the cell state over potentially very long sequences. This helps preserve gradient flow over multiple time steps. The key innovation is the gated cell that preserves information in tact over time.\nimage1\nThe first gate is called a forget gate. It\u0026rsquo;s a linear layer followed by a sigmoid, so its output will consist of scalars btw 0 and 1. We multiply this result with the cell state to determine which information to keep and which to throw away: values closer to 0 are discarded and values closer to 1 are kept.\nThe second gate is the input gate. It works with the third gate which is generally called cell gate\u0026ndash; to update the cell state (the second hidden state). Similar to the forget gate, the input gate decided which elements of the cell state to update (values close to 1) or not (values close to 0). The third gate determines what those updated values are, in the range of -1 to 1. (tanh). This result is added to the cell state.\nThe last gate is the output gate. It determines which information from the cell state to use to generate the output.\nThe cell state goes through the tanh before being combined with the sigmoid output from the output gate, and this result is the new hidden state.\nHidden state is used to predict next word where as cell state is used to preserve memory.\nclass LSTMcell(Module): def __init__(self, ni, nh): self.forget_gate = nn.Linear(ni + nh, nh) self.input_gate = nn.Linear(ni + nh, nh) self.cell_gate = nn.Linear(ni + nh, nh) self.output_gate = nn.Linear(ni + nh, nh) def forward(self, input, state): h, c = state h = torch.cat([input, h], dim=1) forget = torch.sigmoid(self.forget_gate(h)) c = c * forget inp = torch.sigmoid(self.input_gate(h)) cell = torch.tanh(self.cell_gate(h)) c = c + inp * cell # final c state out = torch.sigmoid(self.output_gate(h)) h = out * torch.tanh(c) # final h state return h, (h,c) We can then refactor the code, in terms of performance, it\u0026rsquo;s better to do one big matrix multiplication than four small ones (because GPU works better in doing things parallel). The stacking takes time (since we have to move one of the tensors around the GPU to have it all in a contiguous array), so we use two seperate layers for the input and the hidden state.\nclass LSTM(Module): def __init__(self, ni, nh): self.i = nn.Linear(ni, nh*4) self.h = nn.Linear(ni, nh*4) def forward(self, x, hc): h, c = hc i, f, g, o = (self.i(input) + self.h(h)).chunk(4, 1) i, f, g, o = i.sigmoid(), f.sigmoid(), g.tanh(), o.sigmoid() c = (f * c) + (i * g) h = o * c.tanh() return h, (h,c) This is how the chunk method works: t = torch.arange(0, 10); t [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] t.chunk(2) ([0,1,2,3,4], [5,6,7,8,9]) Here, It divides the 't' straight into 2 halves. For example if the matrix is 4x4 and you created chunk(2), it will give you 2x4 (upper half), 2x4 (lower half). If you chunk it by (2,1), which means after every chunk, you want to leave 1 sliding window, if you visualize it, it will give you vertically first half (4x2), vertically second half (4x2). same, if you have a 3dim shaped tensor, eg. 4x4x4. Regularizing LSTM Recurrent neural networks, in general, are hard to train, because of the problem of vanishing activations and gradients we saw before. Using LSTM (or GRU) cells makes training easier than with vanilla RNNs, but they are still very prone to overfitting.\nData augmentation for text data is currently not a well-explored space.\n1. Dropout Dropout is a regularization technique that was introduced by Geoffrey Hinton, the basic idea is to randomly change some activations to zero at training time. This makes sure all neurons actively work toward the output.\nFor regularization of Neural networks, if we apply dropout with a probability p, we rescale all activations by dividing them by 1-p (each activations has the probabilty p of getting zero\u0026rsquo;ed, so 1-p acts will be left active)\nThe Bernoulli distribution models the probability of success or failure in a single trial, where success (typically denoted as 1) has a probability p and failure (denoted as 0) has a probability 1−p. It\u0026rsquo;s commonly used for situations with only two possible outcomes, like flipping a biased coin or a loaded die where success might be getting a specific result like a head or a six.\nWhy we need to rescale the remaining activations?\nWhen applying dropout, we randomly deactivate (set to zero) a fraction of neurons in a neural network during training. This helps prevent overfitting by forcing the network to learn redundant features.\nNow, when we deactivate neurons, it reduces the overall output of the layer. To compensate for this reduction in output, we scale (multiply) the remaining activations by a factor to maintain the same output expected output. This scaling is essential to ensure that the model doesn\u0026rsquo;t become overly reliant on the presence of all neurons during the training and thus maintains its ability to generalize well to unseen data.\nUnderstanding it with an example!\nWhen we apply dropout with probability p, it means that during each training iteration, each neuron in the network has a probability p of being \u0026ldquo;dropped out\u0026rdquo; or set to zero. This process is randomly applied and help prevent overfitting by making the network more robust and less reliant on specific neurons.\nNow, when we rescale all activations by dividing them by 1-p, it means that we adjust the remaining activations to compensate for the dropout. Since, on average, a fraction p of neurons will be zeroed out, we need to scale up the remaining activations to maintain the same expected output. Dividing by 1-p accomplishes this scaling, ensuring that the overall output of the layer remains consistent despite the dropout.\nIf x is the activations value, after applying dropout with probability p the re-scaled activation x' would be:\nx\u0026rsquo; = x / (1-p)​\nsay,\nx = 10 (there are 10 activations), p = 0.2\nSo, there will be 10 activations (randomly generate or trained)\n\u0026gt;\u0026gt;\u0026gt; torch.sigmoid(torch.randn(10)) tensor([0.1107, 0.7174, 0.4452, 0.6757, 0.7396, 0.7361, 0.6020, 0.1646, 0.2716, 0.6765]) the activations, which are \u0026lt; p are dropped (set to zero).\nSo in the above example, element at indices 0, 7 which are \u0026lt; 0.2 are dropped.\nNext step is to calculate the rescaled activations.\n0.7, 0.4, 0.6, 0.7, 0.7, 0.6, 0.2, 0.6\nrescaled version of them would be:\n0.8, 0.5, 0.7, 0.8, 0.8, 0.7, 0.25, 0.7\nThese rescaled activations compensate for the dropout!\nUsing bernoulli distribution to dropout activations in a pytorch\u0026rsquo;s layer\nclass Dropout(Module): def __init__(self, p): self.p = p def forward(self, x): if not x.training: return x mask = x.new(*x.shape).bernoulli_(1-p) return x * mask.div_(1-p) \u0026gt;\u0026gt;\u0026gt; a = torch.randn(10) \u0026gt;\u0026gt;\u0026gt; a.new # creates new tensor with same datatype (maintains consistency) \u0026gt;\u0026gt;\u0026gt; a = a.new(*a.shape) # unpack the shape and create a tensor of same shape=a \u0026gt;\u0026gt;\u0026gt; a = a.bernaulli_(p) # draw random binary nums using bernaulli distribution \u0026gt;\u0026gt;\u0026gt; mask = a.div_(1-0.2) # divide each input by 1-p tensor([0.0000, 1.2500, 1.2500]) # this mask contains every information that we need to dropout-- # multiplying it with 0 will get 0 the acts # and 1.2 (for eg) will amplify them! \u0026gt;\u0026gt;\u0026gt; a * mask tensor([0.0000, 2.4414, 2.4414]) Bernoulli distribution\u0026ndash; any number to either 0 or 1\nUsing dropout before passing the output of our LSTM to final layer will help reduce overfitting.\nFastAi also uses dropout in its default CNN head, and it is also available in its tabular module. So I think it means, dropout technique is widely applicable to train the sleeping neurons.\n2. Activation Regularization and Temporal Activation Regularization In weight decay, to aim to make the weights as small as possible (we do this by add a small penalty to the loss, which results in increasing the gradient which results in weights getting small in order to reduce the loss).\nIn Activation regularization (AR), we will try to make the final activations produced by the LSTM as small as possible! (instead of weights)\nWe can do this by adding the means of the squares of the activations along with a multiplier alpha (which is like wd for weight decay).\nloss += alpha * acts.pow(2).mean()\nTemporal Activation Regularization (TAR) is linked to the fact we are predicting tokens in a sentence. That means it\u0026rsquo;s likely that the outputs of our LSTM model should somewhat make sense when we read them in order. TAR encourages this behavior by adding a penalty to the loss to make the difference between two consecutive activations as small as possible.\nWe calculate the difference between every consecutive activations: (remember, activations in a layer are just another matrix, a stacked batch of matrices)\nloss += beta * (acts[:, 1:] - acts[:, :-1]).pow(2).mean()\nalpha and beta are two hyperparameters to tune. To make this work, we need our model with dropout to return 3 things:\nproper output activations of LSTM pre-dropout activations of LSTM post-dropout AR is often applied on the dropped-out (post) acts\u0026ndash; ensuring we only penalize the activations that are used (not zeros).\nWhile TAR is often applied on the non-dropped-out (pre) acts\u0026ndash; because zeros in the dropped out acts create big differences between two consecutive acts.\nWeight-tied Regularized LSTM\nAn useful trick that can be applied from AWD-LSTM paper is weight tying23\nclass LMModel(nn.Module): def __init__(self, n_hidden, n_layers, vocab_sz, p): self.i_h = nn.Embedding(vocab_sz, n_hidden) self.rnn = nn.LSTM(n_hidden, n_hidden, n_layers, batch_first=True) self.drop = nn.Dropout(p) self.h_o = nn.Linear(n_hidden, vocab_sz) self.h_o.weight = self.i_h.weight #weight tying: AWD-LSTM # using the same embedding weight matrix in the output self.h = [torch.zeros(n_layers, bs, n_hidden) for _ in range(2)] def forward(self, x): raw, h = self.rnn(self.i_h(x), self.h) out = self.drop(raw) self.h = [h_.detach() for h_ in h] return self.h_o(out), raw, out def reset(self): for h in self.h: h.zero_() model = LMModel7(len(vocab), 64, 2, 0.5) # Adding AR and TAR regularization class RNNRegularizer: def __init__(self, model: nn.Module, alpha=0., beta=0.): self.alpha, self.beta = alpha, beta self.m = model def after_loss(self): if not self.model.train(): return if self.alpha: # adding mean of the squares along with a multiplier alpha. for p in self.m.parameters(): p.grad += self.alpha * p.data.pow(2).mean() if self.beta: for name, module in self.model.named_modules() h = module.raw_out if len(h) \u0026gt; 1: diff_mean = (h[:, 1:] - h[:, :-1]).float().pow(2).mean() for p in module.parameters(): p.grad += self.beta * diff_mean reg = RNNRegularizer(model, 2, 1) loss.backward() reg.after_loss() optim.step() AWD-LSTM architecture uses dropout in a lot more places:\nembedding dropout (just after the embedding layer) input dropout (after the embedding layer concat with input) weight dropout (weights of the LSTM at each training step) hidden dropout (hidden state between two layers) This makes it even more regularized.\nFastai\u0026rsquo;s implementation: Since fine-tuning those five dropout values (including the dropout before the output layer) is complicated, we have determined good defaults and allow the magnitude of dropout to be tuned overall with the drop_mult parameter you saw (which is multiplied by each dropout). let\u0026rsquo;s implement this from scratch later.\nGradient Clipping (good to know) Gradient clipping is a technique that is used to prevent the exploding gradient problem during the training of deep neural networks.\nworking:\nthe error gradient is clipped to a threshold during the backward pass the clipped grads are used to update the weights. problem with gradient clipping:\ngradient clipping can result in changing of the direction of gradient in the plane, which results in wrong local minima. This is a tradeoff.\nSummary each word is associated with n_hidden latent factors.\nwords are passed through an embedding layer of size 30 x 64, representing vocab sz and n_hidden dimensions.\nIn our LSTM architecture, input sequences are structured as 64 x 16 matrices, where 16 is the sequence length of each input, and 64 is the batch size.\nEach word is transformed into embeddings of shape 64 x 16 x 64 (there are 16 words in each input and bs=64)\nHidden states are initialized with dimensions of n_layers x bs x n_hidden, n_layers is the number of RNN layer stacked in pytorch, here set are 2 x 64 x 64.\nLSTM updates both the cell state (c) and the hidden state (h) within its layers\n(how?) input x which is now 64 x 16 x 64 is concatenated with hidden state we decide the size of hidden state as: n_layers x bs x n_hidden. One for each layer, and for each batch. the concatenate will be of shape: 64 x 16 x (64 + 64) = 64 x 16 x 128. we also create a cell state of same shape. f * c\u0026ndash; to forget bunch of thing i * g\u0026ndash; which part of the memory to update and what values to insert in that update. c = (f * c) + (i * g) \u0026ndash; perform the actual update produce output with tanh, deciding the filtering. Essentially, the LSTM processes and updates information of these states, that we created.\nThree types of regularization we use in RNNs\nDropout Activation Regularization (AR) Temporal Activation Regularization (TAR) AWD-LSTM uses all three. Weight-tied LSTM (introduced in the AWD-LSTM paper)\u0026ndash; we use same weight matrix that we used in the input embedding in the output self.ho.weight = self.ih.weight\nGradient clipping\nSources\nRNN Survery Paper\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nweight tying\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nAWD-LSTM\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://akash5100.github.io/posts/2024-02-07-lstm/","summary":"\u003ch3 id=\"table-of-contents\"\u003eTable of contents\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#regularizing-lstm\"\u003eRegularizing LSTM\u003c/a\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#1-dropout\"\u003e1. Dropout\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#2-activation-regularization-and-temporal-activation-regularization\"\u003e2. Activation Regularization and Temporal Activation Regularization\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#gradient-clipping-good-to-know\"\u003eGradient Clipping (good to know)\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#summary\"\u003eSummary\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eLSTM were designed to deal with the issue of exploding or vanishing gradients in RNNs. This is challenging because, neural nets generally struggle to learn long-term dependencies.\u003c/p\u003e\n\u003cp\u003eLSTM introduces gate mechanism and has 2 hidden state, that allows information to flow unmodified over many timesteps. Each LSTM cell has a set of gates (forget, input, cell/memory and out) that carefully regulate the information into and out of the cell.\u003c/p\u003e","title":"Long Short-Term Memory (LSTMs)"},{"content":"Table of contents We can remove the hardcoded part by replacing it with a loop Improving our RNN First thing first, let\u0026rsquo;s solve the resetting of hidden state Creating Multilayer RNN Exploding and disappearing Activations If we have data like: (from the example of previous blog)\nX \u0026ndash; Y\n'one', '.', 'two' \u0026ndash; '.'\n'.', 'three', '.' \u0026ndash; 'four'\nWhere 3 words are used as input to predict 1 word from a vocabulary as an output, we can create a neural network architecture that takes three words as input and returns a prediction of the probability for each possible next word in the vocabulary.\nWe will use three standard linear layers but with two tweaks!\nThe first tweak is that the first linear layer will use only the first word\u0026rsquo;s embedding as activations, the second layer will use the second word\u0026rsquo;s embedding plus the first layer\u0026rsquo;s output activation and the third layer will use the third word\u0026rsquo;s embedding plus the second layer\u0026rsquo;s output activations. The key effect is that the model will take into account the information from the words that came before it. The \u0026lsquo;information\u0026rsquo; is actually the higher dimensionality of data points. This is created by the embedding layer. Here is a video that explains it well in the first part.\nThe second tweak is that each of these three layers will use the same weight matrix. The goal of this specific structure is to train the model\u0026rsquo;s embeddings in a way that a single set of parameters can be used consistently across all positions in the sequence (for example, embedding and learned weights for a word like \u0026ldquo;the\u0026rdquo; can be used again and again and the result doesnot depend on position of that word in the sequence). This means, the output of the network is designed not to depend on the specific position of a word in the sequence. In other words, activation values will change as data moves through the layers, but the layer weights themselves will not change from layer to layer.\nLet\u0026rsquo;s take the same example from above:\nemb means the embedding layer, so a word like \u0026lsquo;one\u0026rsquo; is fed into the embedding layer.\nWe can now create the language model module that.\nclass LMModel1(nn.Module): def __init__(self, vocab_sz, n_hidden): self.ih = nn.Embedding(vocab_sz, n_hidden) # we need embs for every word in the vocab and n_hidden is a random num self.hh = nn.Linear(n_hidden, n_hidden) self.ho = nn.Linear(n_hidden, vocab_sz) def forward(self, x): # x is batch of length 64, with 3 words each h = F.relu(self.hh(self.ih(x[:,0]))) h = h + self.ih(x[:,1]) h = F.relu(self.hh(h)) h = h + self.ih(x[:,2]) h = F.relu(self.hh(h)) return self.ho(h) ih (input to hidden) -\u0026gt; hh (hidden to hidden) -\u0026gt; ho (hidden to output)\nWe can remove the hardcoded part by replacing it with a loop class LMModel2(nn.Module): def __init__(self, vocab_sz, n_hidden): self.ih = nn.Embedding(vocab_sz, n_hidden) self.hh = nn.Linear(n_hidden, n_hidden) self.ho = nn.Linear(n_hidden, vocab_sz) def forward(self, x): h = 0 for i in range(3): h = h + (self.ih(x[:,i])) h = F.relu(self.hh(h)) return self.ho(h) we see a set of activations is being updated each time through the loop stored in the variable h\u0026ndash; this is called the hidden state\nJargon: Hidden State The activations that are updated at each step of a recurrent neural network.\nJargon: Recurrent NN (Looping Neural Net) A neural network that is defined using a loop like this is called a recurrent Neural Network. RNN is not a complicated new architecture but simply a refactoring of a multilayer neural network using a for loop.\nImproving our RNN Looking at the code, one thing seems problematic is that we are initializing our hidden state to zero for every new input sequence. Why is that problematic?\nResetting the activations to zero for every sequence means starting with a \u0026ldquo;blank slate\u0026rdquo; for each new input sequence. It doesn\u0026rsquo;t allow the model to remember information from previous sequences it has processed. Imagine you reading a story in a book, but when you turn a page you forget what you read, lol.\nAnother thing that can be improved in our RNN is, why only predict the 4th word after 3 words? why not predict the 2nd and 3rd words?\nFirst thing first, let\u0026rsquo;s solve the resetting of hidden state We are basically throwing away the information we have about the sentences we have seen so far, this could be easily fixed by saving the state as a class variable.\nself.h = 0, and update it each time.\nBut we will be creating a not very noticable, but important to deal problem. Any guesses?\nHere is what will happen:\nDuring the forward pass, the RNN processes the input sequence step by step, generating hidden states (h) for each time step, Let\u0026rsquo;s say you have an input sequence of length T and an RNN with hidden size H. At each time step t, the RNN takes the input x_t and then previous hidden state h_{t-1}, and produces the current hidden state h_t and the output y_t. after processing the entire input seqs, you compare the predicted outputs y_t with the true labels y_{true, t} for each step t. You compute the loss function that measures the difference between the predicted and true labels. Backward pass\u0026ndash; Now you need to backpropagate the error through the RNN to update the weights and biases. BPTT (backpropagation through time) does this by unrolling the RNN through time and applying standard backprop at each time step. at each time step t, you compute the error term delta_t by applying the chain rule to the loss function with respect to the output y_t and the hidden state h_t. You then propagate the error term backward through time, using the error term of the current time step t to compute the error term of the previous time step t-1. This is done by applying the chain rule to the RNN\u0026rsquo;s update equations. Finally, you update the weights and biases of the RNN using the computed error terms and an optimizer (e.g., gradient descent) at each time step t. repeat\u0026ndash; repeat steps 2 and 3 for multiple epochs, until loss converges. So, for a word, we created embeddings and activations is stored in hidden state. For a new word, we are actually incorporating the activation of that word into the already existing hidden state. This accumulation occurs over time, and if there are 1000 tokens, the size will grow to incorporate information from 1000 tokens. But in real world, we might have 1 million tokens.\nThis is going to be slow and infact we wont be able to store even one mini-batch on the GPU.\nThe solution to this problem is to tell Pytorch that we dont want to backpropagate the derivatives through the entire implicit neural network. Instead, we will keep just last three layers of gradients.\nTo remove the gradient history in Pytorch, we use detach method.\nHere is the RNN, now stateful.\nclass LMModel3(nn.Module): def __init__(self, vocab_sz, n_hidden): self.ih = nn.Embedding(vocab_sz, n_hidden) self.hh = nn.Linear(n_hidden, n_hidden) self.ho = nn.Linear(n_hidden, vocab_sz) self.h = 0 def forward(self, x): for i in range(3): self.h = self.h + (self.ih(x[:,i])) self.h = F.relu(self.hh(self.h)) out = self.ho(self.h) self.h = self.h.detach() return out def reset(self): self.h = 0 # Later, at the beginning of each epoch and before each validation phase # this will be used. This will make sure we start with a clean state # before reading those continuous chunks of text. n_hidden = 64 vocab_sz = len(vocab) This model will have the same activatiosn whatever sequence length we pick, because the hidden state will remember the last activation from the previous batch. The only thing different is the gradients will be computed at each step will consider only the sequence length instead of the whole stream. This approach is called batchpropagation through time (BPTT).\nHere is the gradual flow of data 64 x 1 -\u0026gt; 64 -- 64 -\u0026gt; 64 -- 30 (vocab sz)\nBut this would be the shape of hidden state: 64 x 64. This could be explained as, the output of first layer is 64, and for each element in the batch, the bs=64. (bs, n_hidden).\nThe gradients for the hidden state (self.h) will be detached after processing each sequence. This means that gradients will not be calculated or updated for the hidden state during backpropagation through time (BPTT) for subsequent sequences. However, gradients for other parameters in the model, such as the parameters of the embedding layer (self.ih) or the linear layers (self.hh and self.ho), will still be calculated and updated normally based on the loss computed at each time step.\nWe can use our previous group_chunks function to create sequence.\nBut wait, we can improve the our model, instead of predicting 1 word after every 3 word, what if we can predict next word after every single word?\nThis is simple to add in! We first need to change our data, so that the dependent variable has each of the three next words after each of our three input words.\nsl = 16 seqs = [ tensor(nums[i: i+sl]), tensor(nums[i+1: i+sl+1]) for i in range(0, len(nums)-sl-1, sl)] # this range will jump 16 index # lets see [\u0026#34; \u0026#34;.join([vocab[o] for o in s]) for s in seqs[0]] [\u0026#39;one . two . three . four . five . six . seven . eight .\u0026#39;, \u0026#39;. two . three . four . five . six . seven . eight . nine\u0026#39;] cut = int(len(seq) * 0.8) train_ds = group_chunks(seqs[:cut], bs) valid_ds = group_chunks(seqs[cut:], bs) class LMModel4(nn.Module): def __init__(self, vocab_sz, n_hidden): self.ih = nn.Embedding(vocab_sz, n_hidden) self.hh = nn.Linear(n_hidden, n_hidden) self.ho = nn.Linear(n_hidden, vocab_sz) self.h = 0 def forward(self, x): outs = [] for i in range(sl): self.h = self.h + (self.ih(x[:,i])) self.h = F.relu(self.hh(self.h)) outs.append(self.ho(self.h)) self.h = self.h.detach() return torch.stack(outs, dim=1) def reset(self): self.h = 0 n_hidden = 64 vocab_sz = len(vocab) The input x, to the RNN model now of length 16, instead of 3.\nWe are trying to predict next word after each word. So we loop through for each word in a sequence and feed that word in the embedding and pass it into the linear layer, while updating the hidden state. and it gives one output, which is of length equal to the length of vocab (later we can softmax it, which gives a single word with highest probability).\nBut for now, we actually save/append that 30 shaped tensor into an array, and we do that for each word in the sequence. So for sequence length = 16, the outs array will be of length 16 and each element is a tensor of shape bs, vocab_sz\u0026ndash; 64, 30.\nTo summarize, for every word in the sequence of length 16, we made the NN predict the next word (everything is possible because how we created the data splits) and we save every next word prediction and stack it.\nSo the final output of model will be of shape [sl x bs x vocab_sz], but we can stack them to 1st dimension using dim=1, so it becomes\u0026ndash; [bs, sl, vocab_sz].\n# understanding stack with dim=1 \u0026gt;\u0026gt;\u0026gt; a = torch.tensor([1,3,5]) \u0026gt;\u0026gt;\u0026gt; b = torch.tensor([2,4,6]) \u0026gt;\u0026gt;\u0026gt; torch.stack([a,b]) tensor([[1, 3, 5], [2, 4, 6]]) \u0026gt;\u0026gt;\u0026gt; torch.stack([a,b], dim=1) tensor([[1, 2], [3, 4], [5, 6]]) Let\u0026rsquo;s say we have 4 data points, and it gave us output of 5 (vowels). The shape would be 2, 5.\nLikewise, we stack all the 64 batch and calculate the loss together. Since we stacked on dim=1. Here is the loss function:\ndef loss_func(preds, targs): # targs-- bs, sl # preds-- bs, sl, vocab return F.cross_entropy(preds.view(-1, len(vocab)), targs.view(-1)) # here, 5 is the length of vocab, (= the vowels) Before we can compare, we need to reshape and flatten them.\nLet\u0026rsquo;s say this is our output:\nafter flattening (starting with sl, dim=1) it will look like:\nThe flattened second dim is of shape [64 * 16, 30] = [1024, 30]\nThe targets is of shape bs, sl = 64, 16. We flatten them by (-1). The preds-targs will be of shape (6416, 30) - (6416,). We can use CrossEntropy loss normally in this.\nWe only have one linear layer between the hidden state and the output activations in our basic RNN, so maybe we\u0026rsquo;ll get better results with more layers.\nCreating Multilayer RNN In, a multilayer RNN, we pass the activations from one RNN into a second RNN.\nWe can save our time and use PyTorch\u0026rsquo;s RNN class, which implements exactly the same.\n# Args: # | input_size: The number of expected features in the input `x` # | hidden_size: The number of features in the hidden state `h` # | num_layers: Number of recurrent layers. E.g., setting ``num_layers=2`` # | would mean stacking two RNNs together to form a `stacked RNN`, # | with the second RNN taking in outputs of the first RNN and # | computing the final results. Default: 1 # batch_first: If ``True``, then the input and output tensors are # | provided as `(batch, seq, feature)` instead of `(seq, batch, feature)`. # | Note that this does not apply to hidden or cell states. See the # | Inputs/Outputs sections below for details. Default: ``False`` class LMModel5(nn.Module): def __init__(self, vocab_sz, n_hidden, n_layers): self.ih = nn.Embedding(vocab_sz, n_hidden) self.rnn = nn.RNN(n_hidden, n_hidden, n_layers, batch_first=True) self.ho = nn.Linear(n_hidden, vocab_sz) self.h = torch.zeros(n_layers, bs, n_hidden) def forward(self, x): res, h = self.rnn(self.ih(x), self.h) self.h = h.detach() return self.ho(res) def reset(self): self.h.zero_() n_hidden = 64 vocab_sz = len(vocab) n_layers = 2 # And if we train this: learn = Learner(dls, LMModel5(len(vocab), 64, 2), loss_func=CrossEntropyLossFlat(), metrics=accuracy, cbs=ModelResetter) learn.fit_one_cycle(15, 3e-3) It disappointing than our single layer RNN, why so? The reason is that we have a deeper model, leading to exploding or vanishing activations.\nExploding and disappearing Activations In Practice, creating accurate RNN model is difficult. We will get better results if we call detach less often and have more layers\u0026ndash; this will give our RNN a longer time to learn from and richer features to create. This means our model is more deep and training this kind of deep model is a key challenge.\nThis is challenging because of what happens when you multiply by a matrix many times. If we multiply a number many times, eg, if you keep multiplying 1 with 2, 2, 4, 8, 16 and after 32 steps, you already at 4,294,967,296. A similar issue happens if you multiply by 0.5, you get 0.5, 0.25, 0.125 and after 32 steps, 0.00000000023. As you can see, multiplying a number even slightly higher or lower than 1 results in an explosion or disappearence of our starting number, after just few multiplications.\nBecause matrix multiplication is just multiplyiong number and then adding them up, exactly same thing happens\u0026ndash; and that\u0026rsquo;s all a deep neural network is\u0026ndash; each extra layer is another matrix multiplication. This means that it is very easy for a deep nn to end up with extremely large or extremly small numbers.\nThis is a problem, because the way computer stores floating numbers, it become less and less accurate the further away the number gets from zero.\nThis inaccuracy often leads to the gradient calculated for updating the weights end up as zero or infinity. This is referred to as the exploding or vanishing gradients problem. That means, in SGD the weights are either not updated at all or jump to infinity, either way that doesn\u0026rsquo;t improve with training.\nOne option is to change the definition of a layer in a way that makes it less likely to have exploding activations. (How? idk.)\nAnother option is by being careful about initialization.\nFor RNNs, two types of layers are frequently used to avoid exploding activations: gated recurrent units (GRUs) and long short-term memory (LSTM) layers. Both of these are available in PyTorch and are drop-in replacements for the RNN layer.\n","permalink":"https://akash5100.github.io/posts/2024-02-06-tweaking_mlp_to_make_it_rnn/","summary":"\u003ch3 id=\"table-of-contents\"\u003eTable of contents\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#we-can-remove-the-hardcoded-part-by-replacing-it-with-a-loop\"\u003eWe can remove the hardcoded part by replacing it with a loop\u003c/a\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#improving-our-rnn\"\u003eImproving our RNN\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#first-thing-first-lets-solve-the-resetting-of-hidden-state\"\u003eFirst thing first, let\u0026rsquo;s solve the resetting of hidden state\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#creating-multilayer-rnn\"\u003eCreating Multilayer RNN\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#exploding-and-disappearing-activations\"\u003eExploding and disappearing Activations\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eIf we have data like: (from the example of previous blog)\u003c/p\u003e\n\u003cp\u003e\u003ccode\u003eX\u003c/code\u003e \u0026ndash; \u003ccode\u003eY\u003c/code\u003e\u003c/p\u003e\n\u003cp\u003e\u003ccode\u003e'one', '.', 'two'\u003c/code\u003e \u0026ndash; \u003ccode\u003e'.'\u003c/code\u003e\u003c/p\u003e\n\u003cp\u003e\u003ccode\u003e'.', 'three', '.'\u003c/code\u003e \u0026ndash; \u003ccode\u003e'four'\u003c/code\u003e\u003c/p\u003e\n\u003cp\u003eWhere 3 words are used as input to predict 1 word from a vocabulary as an output, we can create a neural network architecture that takes three words as input and returns a prediction of the probability for each possible next word in the vocabulary.\u003c/p\u003e","title":"Stateful Recurrent Neural Network"},{"content":"Here are the main steps for language modeling:\nTokenization\u0026ndash; Converting Text into list of words (creating the vocab) Numericalization\u0026ndash; Converting each word in vocab to number, by replacing them with their indices (simple!) The next steps are: Language Model Data Creation (X\u0026amp;Y) and Language Model Creation\nJargon: Token One element of a list created by the tokenization process. It could be a word, part of a word (a subword), or a single character.\nTo split the text into words, and to make the vocab, we need to convert text to words.\n\u0026ldquo;This is an example.\u0026rdquo; -\u0026gt; \u0026rsquo;this\u0026rsquo;, \u0026lsquo;is\u0026rsquo;, \u0026lsquo;an\u0026rsquo;, \u0026rsquo;example\u0026rsquo;, \u0026lsquo;.\u0026rsquo;\nBut there are different problem with this, text has lot of details. For instance, what if we have a chemistry long organic compound, oxybenzosomethingpentane, or some chinese font or japanese that don\u0026rsquo;t use bases at all, or in english, a word like \u0026ldquo;don\u0026rsquo;t\u0026rdquo;? German and Polish languages can be made so long by concatenating small pieces. How would you split it?\nThere is no one correct answer to these question, there is no one approach to tokenization, so we use 3 main techniques:\nWord based: what we saw above subword based: \u0026ldquo;occasion\u0026rdquo; -\u0026gt; \u0026lsquo;oc ca sion\u0026rsquo; (it splits words into more smaller parts) character based: \u0026lsquo;akash\u0026rsquo; -\u0026gt; a k a s h (individual character) SpaCy is a opensource tokenization library.\nspacy([\u0026#39;The U.S. dollar $1 is $1.00.\u0026#39;]) [\u0026#39;The\u0026#39;,\u0026#39;U.S.\u0026#39;,\u0026#39;dollar\u0026#39;,\u0026#39;$\u0026#39;,\u0026#39;1\u0026#39;,\u0026#39;is\u0026#39;,\u0026#39;$\u0026#39;,\u0026#39;1.00\u0026#39;,\u0026#39;.\u0026#39;] Fastai has some rules, that it puts on top of this spacy tokens. They put some tokens starting with \u0026lsquo;xx\u0026rsquo;, these are special tokens. Pytorch names special tokens like \u0026lsquo;\u0026rsquo;.\nThe most common item is, xxbos and xxeos, is beginning of stream and end of stream.\nxxmaj replaces the capitals letters, like\n\u0026lsquo;L\u0026rsquo; -\u0026gt; \u0026lsquo;xxmaj\u0026rsquo;, \u0026rsquo;l\u0026rsquo;\nxxrep replaces repeating words,\n\u0026lsquo;!!!\u0026rsquo; -\u0026gt; \u0026lsquo;xxrep\u0026rsquo;, \u0026lsquo;3\u0026rsquo;, \u0026lsquo;!\u0026rsquo; (3 * !)\nsome other rules are,\nreplacing html to text replacing repeating words replacing useless spaces, removes repeating spaces wrapping spaces aroung \u0026lsquo;/\u0026rsquo; and \u0026lsquo;#\u0026rsquo;. \u0026lsquo;#\u0026rsquo; -\u0026gt; \u0026rsquo; # \u0026rsquo; \u0026amp; \u0026lsquo;/\u0026rsquo; -\u0026gt; \u0026rsquo; / ' lowercasing all character adding \u0026lsquo;bos\u0026rsquo; and \u0026rsquo;eos\u0026rsquo; at the beginning and end of a complete sequence. here is a useful link: text_proc_rules\nSubword Tokenization\nIn addition to the word tokenization approach seen in the preceding section, another popular tokenization method is subword tokenization. Word tokenization relies on an assumption that spaces provide a useful separation of components of meaning in a sentence. However, this assumption is not always appropriate. For instance, consider this sentence: 私の名前はアカシュ・ヴェルマです (\u0026lsquo;My name is Akash Verma\u0026rsquo; in Japanese). That’s not going to work very well with a word tokenizer, because there are no spaces in it! Languages like Chinese and Japanese don’t use spaces, and in fact they don’t even have a well-defined concept of a “word.” Other languages, like Turkish and Hungarian, can add many subwords together without spaces, creating very long words that include a lot of separate pieces of information.\nTo handle these cases, its best to use subword tokenization. This proceeds in two steps:\nFind most commonly occuring groups of letters, these became vocab. tokenize the corpus using this vocab. When using the library like Spacy or Fastai, we instantiate our subword tokenizer with what size of vocab it should be. \u0026ndash;\u0026gt; SubwordTokenizer(vocab_sz=2000). We want to create vocab for our text corpus, so passing our text corpus and \u0026rsquo;train\u0026rsquo; it. \u0026ndash;\u0026gt; setup(txts). This creates vocab for our text corpus and then we can use that trained subwordtokenizer to tokenize things!\nThe length of each token depends on the size of vocab, if we created a vocab of size smaller, token size is smaller too that means it will break a single word into many words. Whereas if the vocab size to too big, it will consider a complete word as a token.\nsmall vocab size -\u0026gt; \u0026lsquo;y o u d is c over ed\u0026rsquo; -\u0026gt; more tokens\nlarge vocab size -\u0026gt; \u0026lsquo;you discover ed\u0026rsquo; -\u0026gt; less tokens\nPicking a vocab size represents a compromise, a larger vocab size means fewer tokens per sentence, which means faster training and less memory and less state for the model to remember. But on the downside, it means larger embedding matrices, which require more data to learn.\n\u0026ldquo;more data to learn\u0026rdquo;, meaning, for example, if the text contains a name \u0026lsquo;akash\u0026rsquo; this would be considered as one token, whereas if the vocab size is small, it would be \u0026lsquo;ak ash\u0026rsquo; and \u0026lsquo;ash\u0026rsquo; could be an already repeating word in the corpus. To solve this problem, we can replace words like this (which occurs rarely) with an unknown special token, \u0026lt;UKN\u0026gt; or xxukn. This can reduce the embedding matrix size. But then, this means there is also less data for the new rare words (that we replaced with ukn flag), now how can we learn about those rare word in the corpus?\nThis last issue is better handeled by setting a minimum frequence threshold; for example, min_freq=3 means that any word appearing fewer than three times is replaced with xxukn.\nI learned creating RNN from scratch using this dataset: Human Numbers dataset\nIt contains 2 file, train.txt and valid.txt. Merging the two to make it a big corpus. and use it for language modeling.\nwith open(path/\u0026#39;train.txt\u0026#39;) as f: dat = f.read() print(dat[:52]) print(\u0026#39;\\n\u0026#39;) with open(path/\u0026#39;valid.txt\u0026#39;) as f: dat2 = f.read() print(dat2[:52]) dat += dat2 # outputs: # one # two # three # four # five # six # seven # eight # nine # eight thousand one # eight thousand two # eight thousa We will take all those lines and concatenate them in one big stream. From one number to next, we use \u0026ldquo;.\u0026rdquo; as a seperator.\nThis is how our stream will look like: \u0026ldquo;one . two . three . [\u0026hellip;]\u0026rdquo;\ntext = \u0026#34;. \u0026#34;.join(dat.split(\u0026#34;\\n\u0026#34;)) tokens = text.split(\u0026#34; \u0026#34;) tokens.remove(\u0026#34;\u0026#34;) # how does this get in? # tokens will look like: # [\u0026#34;one\u0026#34;, \u0026#34;.\u0026#34;, \u0026#34;two\u0026#34;, \u0026#34;.\u0026#34;, [...]] We need to create vocab, creating a frequency map of all the tokens we have:\nfreq = list({i for i in tokens}) freq = {w: 0 for w in freq} for i in tokens: freq[i]+=1 Now, we have a vocab. Converting them into number. Because fastai numericalize the tokens by ordering them with there frequency of occurance (if a token occurs 100000 times it will be in 0th index of our vocab) I tried to replicate that:\nfreq1 = dict(sorted(freq.items(), key=lambda item: item[1], reverse=True)) vocab = list(freq1); vocab[:3] word2idx = {w: i for i,w in enumerate(freq1)}; word2idx # a map to convert tokens to numbers {\u0026#39;.\u0026#39;: 0, \u0026#39;hundred\u0026#39;: 1, \u0026#39;thousand\u0026#39;: 2, \u0026#39;six\u0026#39;: 3, \u0026#39;five\u0026#39;: 4, \u0026#39;two\u0026#39;: 5, \u0026#39;nine\u0026#39;: 6, \u0026#39;four\u0026#39;: 7, \u0026#39;seven\u0026#39;: 8, \u0026#39;three\u0026#39;: 9, \u0026#39;one\u0026#39;: 10, \u0026#39;eight\u0026#39;: 11, \u0026#39;eighty\u0026#39;: 12, \u0026#39;forty\u0026#39;: 13, \u0026#39;ninety\u0026#39;: 14, \u0026#39;twenty\u0026#39;: 15, \u0026#39;sixty\u0026#39;: 16, \u0026#39;thirty\u0026#39;: 17, \u0026#39;fifty\u0026#39;: 18, \u0026#39;seventy\u0026#39;: 19, \u0026#39;sixteen\u0026#39;: 20, \u0026#39;fourteen\u0026#39;: 21, \u0026#39;eleven\u0026#39;: 22, \u0026#39;thirteen\u0026#39;: 23, \u0026#39;eighteen\u0026#39;: 24, \u0026#39;seventeen\u0026#39;: 25, \u0026#39;ten\u0026#39;: 26, \u0026#39;fifteen\u0026#39;: 27, \u0026#39;nineteen\u0026#39;: 28, \u0026#39;twelve\u0026#39;: 29} Numericalize\u0026ndash; Replacing the tokens with their indices.\nnums = [word2idx[i] for i in tokens] nums[:20] # our first 20 tokens. [10, 0, 5, 0, 9, 0, 7, 0, 4, 0, 3, 0, 8, 0, 11, 0, 6, 0, 26, 0] Now that our dataset ready data modeling for Language Model should be easy.\nWe going to predict each word based on previous 3 words, so the sequence would be a list of 3 word as independent variable and the label would be the next word after each sequence (dependent variable).\n[(tokens[i:i+3], tokens[i+3]) for i in range(0, len(tokens)-4, 3)][:3] # outputs: [([\u0026#39;one\u0026#39;, \u0026#39;.\u0026#39;, \u0026#39;two\u0026#39;], \u0026#39;.\u0026#39;), ([\u0026#39;.\u0026#39;, \u0026#39;three\u0026#39;, \u0026#39;.\u0026#39;], \u0026#39;four\u0026#39;), ([\u0026#39;four\u0026#39;, \u0026#39;.\u0026#39;, \u0026#39;five\u0026#39;], \u0026#39;.\u0026#39;)] We can do this with nums which is what the model will actually use:\nseqs = [tensor(nums[i:i+3]), tensor(nums[i+3]) for i in range(0, len(nums)-4, 3)] seqs[:5] [(tensor([10, 0, 5]), 0), (tensor([0, 9, 0]), 7), (tensor([7, 0, 4]), 0), (tensor([0, 3, 0]), 8), (tensor([ 8, 0, 11]), 0)] We can use 80% of sequence as training and 20% as validating dataset.\nbs = 64 cut = int(len(seqs)*0.8) train_ds = seqs[:cut] valid_ds = seqs[cut:] Because the activations on a architecture like RNN is carried forward within batches, if sequence 1 of batch 1 and batch 2 are flowing, our Neural Net will learn better. The activations being carried forward in RNN is called Hidden state (we\u0026rsquo;ll study this in detail).\nIf bs is the batch size, and m is the length of each sequence in a batch,\nbs = 64 m = len(seqs)//bs m, bs, len(seqs) (328, 64, 21031) The first batch will be composed of samples:\n(0, m*1, m*2, m*3, ... m*(bs-1))\nThe second batch will be:\n(0, m+1*1, m+1*2, m+1*3, ... m+1*(bs-1))\nThis way at each epoch, the model will see a chunk of contiguous text of size 3*m.\nLets understand this with an example:\n\u0026gt;\u0026gt;\u0026gt; a = \u0026#34;\u0026#34;\u0026#34;my name is akash and akash is my name both means same thing right ? ?\u0026#34;\u0026#34;\u0026#34; \u0026gt;\u0026gt;\u0026gt; a = a.split(\u0026#34; \u0026#34;); a [\u0026#39;my\u0026#39;, \u0026#39;name\u0026#39;, \u0026#39;is\u0026#39;, \u0026#39;akash\u0026#39;, \u0026#39;and\u0026#39;, \u0026#39;akash\u0026#39;, \u0026#39;is\u0026#39;, \u0026#39;my\u0026#39;, \u0026#39;name\u0026#39;, \u0026#39;both\u0026#39;, \u0026#39;means\u0026#39;, \u0026#39;same\u0026#39;, \u0026#39;thing\u0026#39;, \u0026#39;right\u0026#39;, \u0026#39;?\u0026#39;, \u0026#39;?\u0026#39;] \u0026gt;\u0026gt;\u0026gt; bs = 2; m = len(a)//bs; m 8 # there will be 2 batches and each of them will have sequence of length 8 \u0026gt;\u0026gt;\u0026gt; a[:m] [\u0026#39;my\u0026#39;, \u0026#39;name\u0026#39;, \u0026#39;is\u0026#39;, \u0026#39;akash\u0026#39;, \u0026#39;and\u0026#39;, \u0026#39;akash\u0026#39;, \u0026#39;is\u0026#39;, \u0026#39;my\u0026#39;] \u0026gt;\u0026gt;\u0026gt; a[m:2*m] [\u0026#39;name\u0026#39;, \u0026#39;both\u0026#39;, \u0026#39;means\u0026#39;, \u0026#39;same\u0026#39;, \u0026#39;thing\u0026#39;, \u0026#39;right\u0026#39;, \u0026#39;?\u0026#39;, \u0026#39;?\u0026#39;] \u0026gt;\u0026gt;\u0026gt; a[0 + m*0] \u0026#39;my\u0026#39; \u0026gt;\u0026gt;\u0026gt; a[0 + m*1] \u0026#39;name\u0026#39; \u0026gt;\u0026gt;\u0026gt; a[1 + m*0] \u0026#39;name\u0026#39; \u0026gt;\u0026gt;\u0026gt; a[1 + m*1] \u0026#39;both\u0026#39; Its like what we discussed above: a[each token in a sequence + m * for each batch]\nWe can create a function for that:\ndef group_chunks(ds, bs): new_ds = [] m = len(ds)//bs for i in range(m): new_ds += (ds[i + m*j] for j in range(bs)) return new_ds new_a = group_chunks(a, bs=2) \u0026gt;\u0026gt;\u0026gt; for i in range(m+bs): ... print(new_a[i], new_a[i+bs], new_a[i+bs*2]) ... my name is name both means name is akash both means same is akash and means same thing akash and akash same thing right and akash is thing right ? Creating the with our seqs.\ntrain_ds = group_chunks(seqs[cut:], bs) valid_ds = group_chunks(seqs[:cut], bs) This is the way, we organize the data into batches for sequential processing in a model, particularly in the context of recurrent neural networks (RNNs) or sequence models.\n","permalink":"https://akash5100.github.io/posts/2024-02-05-tokenization_for_lm/","summary":"\u003cp\u003eHere are the main steps for language modeling:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eTokenization\u0026ndash; Converting Text into list of words (creating the vocab)\u003c/li\u003e\n\u003cli\u003eNumericalization\u0026ndash; Converting each word in vocab to number, by replacing them with their indices (simple!)\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eThe next steps are: Language Model Data Creation (X\u0026amp;Y) and Language Model Creation\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eJargon: Token\u003c/strong\u003e \u003cbr/\u003e One element of a list created by the tokenization process. It could be a word, part of a word (a subword), or a single character.\u003c/p\u003e","title":"Tokenize \u0026 Numericalize"},{"content":"A language model is a model that is trained to guess the next word in a text (having read the ones before). This kind of task is called self-supervised learning.\nJargon: self supervised learning Training a model using labels that are embedded in the independent variable, rather than requiring external labels.\nSelf supervised learning is not usually used for the model that is trained directly, but instead is used for pretraining a model used for transfer learning. Self supervised learning is used to train a base model, and that base model is used to train different model for specific task like text classification!1\nWherever possible, you should aim to start your neural network training with a pre-trained model, and fine tune it. You really don’t want to be starting with random weights, because that’s means that you’re starting with a model that doesn’t know how to do anything at all! With pretraining, you can use 1000x less data than starting from scratch.2\nHere is an example, say our goal is to train text classifier on IMDb\u0026rsquo;s reviews, either positive or negative.\nwe can do,\nWikipedia's pretrained model -\u0026gt; trained on IMDb classifier\nBut we can achieve better resilt by,\nWikipedia's pretrained model -\u0026gt; Finetune on IMDb corpus -\u0026gt; trained on IMDb classifier\nWe can finetune on IMDb corpus by getting all the text file and training the model on that big text chunk. This is known as Universal Language Model Fine-Tuning (ULMFiT)3\nPretext \u0026amp; Downstream tasks (in transfer learning)\nThe task that pretrained model performs is called pretext task and what the model (that we want) performs after fintuning is called downstream task.\nThe most important question that needs to be answered in order to use self-supervised learning in computer vision is: what pretext task should you use? It turns out that there are many you can choose from.\nChoosing a pretext task\nThe task that the base model is going to perform (pretext task) should be something that is useful/related to the task that the final model after finetuning (downstream task) is going to perform.\nThe relationship between pretext and downstream tasks is that the pretext task is designed to encourage the network to learn useful features for the downstream task, and the downstream task is used to evaluate the quality of the learned features.\nTake an example: autoencoder\u0026ndash; This is a model which can take an input image, converted into a greatly reduced form (using a bottleneck layer), and then convert it back into something as close as possible to the original image.\nThis model is using compression as a pretext task.\nHowever, solving this task requires not just regenerating the original image content, but also regenerating any noise in the original image. Therefore, if your downstream task is something where you want to generate higher quality images, then this would be a poor choice of pretext task.\nYou should also ensure that the pretext task is something that a human could do.\nFor instance, you might use as a pretext task the problem of generating a future frame of a video. But if the frame you try to generate is too far in the future then it may be part of a completely different scene, such that no model could hope to automatically generate it.\nhow would you approach the task of applying a neural network to a language modeling problem?\nThe data is texts, we want to teach our neural network to predict next word. How can we do that? I think it all depends on what we feed to neural net, the structure of data and labels. We can somehow convert the text to numbers because thats what computer understands, We can then preprocess the data in such a way that the next word is the label. How? Idk. but if we do that we might be able to predict next word. Maybe the output will consist of list of probabilities of words like we did in multi-class prediction (Crossentropy and softmax).\nHere is the answer:\nmake list of all possible levels of that categorical variable\u0026ndash; creating vocab In the created vocab, replace the words with their index. creating embedding matrix for each item in the vocab. using this embedding matrix as first layer in the NN. (we did the same thing in Recommendation system!) A dedicated embedding matrix can take input to the index of vocab (created in step 2), this is faster and much efficient approach than normal one-hot encoded indexing the vocab.4 For now, understanding the architecture of this model doesn\u0026rsquo;t matter, this is just for understanding how we used embeddings! 30 is the vocab size. and we predicted next word after that 3 word sequence.\nThe main steps can be named as:\nTokenization\u0026ndash; Converting Text into list of words (the vocab) Numericalization\u0026ndash; Converting each word in vocab to number, by replacing them with their indices (simple!) Language Model Data Creation (X\u0026amp;Y)\u0026ndash; Hmmm Language Model Creation\u0026ndash; Training an Architecture like LSTM on DataLoaders we created, yo what? (ignore) Next, I would be creating notes on Tokenization and Numericalization.\nSources Jurgen introduced it way before\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nFastai blog post\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nULMFiT\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nFastbook course\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://akash5100.github.io/posts/2024-02-03-selfsup_learning/","summary":"\u003cp\u003eA language model is a model that is trained to guess the next word in a text (having read the ones before). This kind of task is called self-supervised learning.\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eJargon: self supervised learning\u003c/strong\u003e \u003cbr /\u003e Training a model using labels that are embedded in the independent variable, rather than requiring external labels.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eSelf supervised learning is not usually used for the model that is trained directly, but instead is used for pretraining a model used for transfer learning. Self supervised learning is used to train a base model, and that base model is used to train different model for specific task like text classification!\u003csup id=\"fnref:1\"\u003e\u003ca href=\"#fn:1\" class=\"footnote-ref\" role=\"doc-noteref\"\u003e1\u003c/a\u003e\u003c/sup\u003e\u003c/p\u003e","title":"Self-Supervised \u0026 Transfer Learning in Language Models"},{"content":"Alright who cares about the follow up blog, here is a notion notes to save me. This is the notes for tweaking Random forest on a tabular dataset to squeeze some performance, I tried this on a Kaggle compe and it worked out pretty badly. Score of something like 2.xx. Maybe I Will improve it when time comes. zzz\nAs we already initialized the embeddings of our users and products for example, We take the result of the embedding lookup and concatenate those activations together. This gives us a matrix that we can then pass through linear layers and nonlinearities in the usual way.\nSince we\u0026rsquo;ll be concatenating the embeddings rather than the dot product, the two embedding matrices can have different sizes i.e, different numbers of latent factors.12\nclass CollabNN(nn.Module): def __init__(self, user_sz, item_sz, n_acts=100, y_range=(0, 5.5)): super(CollabNN, self).__init__() self.user_factors = nn.Embedding(*user_sz) self.item_factors = nn.Embedding(*item_sz) self.layers = nn.Sequential( nn.Linear(user_sz[1]+item_sz[1], n_acts), nn.ReLU(), nn.Linear(n_acts, 1)) self.y_range = y_range def forward(self, userid, itemid): embs = self.user_factors(userid), self.item_factors(itemid) x = self.layers(torch.cat(embs)) return sigmoid_range(x, *self.y_range) model = CollabNN((n_users+1, 74), (n_movies+1, 101), 50) optimizer = torch.optim.Adam(model.parameters(), lr=4e-3) loss_func = nn.MSELoss() What do we want to do in a single epoch # one epoch for i in dls: user, movie, r = i r = r.unsqueeze(dim=0) optimizer.zero_grad() out = model(user, movie) loss = loss_func(out, r) loss.backward() # calc grad optimizer.step() # update weights break Understanding the above architecture.\nLet\u0026rsquo;s say we decided the size of embedding matrix of users is n_users x 74 and the embeddings size of items is n_items x 101 (choosing 74 and 101 randomly, no special reason).\nWe initialized user_factors and item_factors. (The Embeddings)\nNext, to input the latent factors of each user+item into a linear layer, we created the input layer of size 74+101 and n_acts as output. The activation for this layer is ReLU!\nAfter that, there is another Linear layer in sequence, with n_acts as input and 1 as output.\nThis completes our model.\nWorking\nIn forward pass, we have userid and movieid. We get the embedding (latent factor) for that particular movie and user. So, if you assumed it would be of shape, 1 x 74 and 1 x 101, I think you are right!\nWe collect those two and pass it to the Sequential layer, finally we can sigmoid (0 - 1) or tanh(-1 - 1) the logits to get activations from the final layer (as it is a score ranging between 0-1, closer to 0 means eww movie and closer to 1 means wowii movie).\nI tried to create a simple diagram of the above working:\nSources Kaggle Notebook\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nFastbook\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://akash5100.github.io/posts/2024-02-01-deep_learning_for_collaborative_filtering/","summary":"\u003cp\u003eAlright who cares about the follow up blog, \u003ca href=\"https://akzsh.notion.site/Tabular-Data-analysis-and-Decision-Tree-9444c1ca59d7464dbc91e7cb6cb243fc?pvs=4\"\u003ehere\u003c/a\u003e is a notion notes to save me. This is the notes for tweaking Random forest on a tabular dataset to squeeze some performance, I tried this on a \u003ca href=\"https://www.kaggle.com/competitions/store-sales-time-series-forecasting/leaderboard\"\u003eKaggle compe\u003c/a\u003e and it worked out pretty badly. Score of something like 2.xx. Maybe I Will improve it when time comes. zzz\u003c/p\u003e\n\u003chr\u003e\n\u003cbr/\u003e\n\u003cp\u003eAs we already initialized the embeddings of our users and products for example, We take the result of the embedding lookup and concatenate those activations together. This gives us a matrix that we can then pass through linear layers and nonlinearities in the usual way.\u003c/p\u003e","title":"Embeddings in sequential Neural Network"},{"content":"Table of contents Collaborative Filtering A Tabular Dataset -\u0026gt; Movie Recommendation System? Embeddings? Speed up the calculation of scores Why sharp curves of learning weights = overfitting Weight Decay or L2 Regularization Movie Recommendation System with Embeddings (MovieLens Dataset) Direction and Distance of embeddings Otakus are kinda poison to our embeddings I will write a follow up blog about Regression and Random Forest, today completes the half of the first month of 2024, I learned lots of stuffs that I am interested and participated on a Kaggle compe, predicing Energy consumption and production using solar panels, just to try what I learned and now I am one step closer to understanding the \u0026ldquo;Attention\u0026rdquo;. I will continue this journey and still be writing this blog for future me like creating notes, maybe in more readable form.\nCollaborative Filtering A common problem to solve is having number of users and number of products, and you want to recommend which product the user will like or you have a new product in your database and you would like to recommend that product to you current users, but you cant recommend that to every user because someone might like it and some may hate it, you only want the first one to happen. A general solution to this problem called collaborative filtering, works like: look up what kind (genre) a user likes and find other user who have used/liked the same product and recommend the other product that those user have used/like.\nFor example, Netflix when we create an account asks, what kind of movie/genre you like.\nA Tabular Dataset -\u0026gt; Movie Recommendation System? MovieLens1 dataset consists of:\n100,000 ratings (1-5) from 943 users on 1682 movies. Each user has rated at least 20 movies. Simple demographic info for the users (age, gender, occupation, zip) There is the User table2 and a Movie (items) table called movie.csv. Merging those two we have:\nrating = pd.read_csv(path/\u0026#39;rating.csv\u0026#39;).drop(columns=[\u0026#39;timestamp\u0026#39;]) movie = pd.read_csv(path/\u0026#39;movie.csv\u0026#39;, usecols=(0,1)) # for now using, movie id and title rating = rating.merge(movie) rating.head() userId itemId rating 196 242 3 186 302 3 22 377 1 244 51 2 166 346 1 This crosstab table shows ratings a user gave to a movie. We would like our model to learn to fill the missing ones. Those missing cells are the place where the user have not seen that movie or not reviewed it.\nHow?\nIf we knew for each user to what degree they liked each category that a movie might fall into, such as genre, age, actors, etc then a simple way to fill these is multiplication of all of them and add them. For instance, lets assume these are ranging from -1 to +1, close to +1 means the user liked it, and vice versa.\nThe result of the multiplication and addition is called Score. If we have these numbers for each users and items (movies) we can dot product them and get the scores.\nEmbeddings? From where do we get these numbers for each user and item, the answer is we don\u0026rsquo;t, we learn them. These numbers are called latent factors; we refer to them as latent because, for us (humans), we don\u0026rsquo;t know their meaning.\nHere is how the overall image will look like:\nWe created 2 matrix, depending on how many feature each user and movie will have (here 4, act, com, scifi, rom. So n_user x 4 and 4 x n_movies), initialized both with random number because we are going to learn it anyway using SGD.\nSpeed up the calculation of scores We have 2 latent factor matrix, for user and item. To make predictin, we need to take the dot product, for specific user and item\u0026rsquo;s latent vectors. But deep learning models don\u0026rsquo;t know how to index into a matrix to lookup the vector for a specific user/movie. They only understand matrix multiplications. We can represent looking up the index as multiplying by one-hot-encoded vector of that row.\n# User Matrix users = torch.tensor([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6], [0.7, 0.8, 0.9]]) #(3 users x 3 factors) # One-hot encoded vector for 2nd user one_hot = torch.tensor([0, 1, 0]) # Matrix Multiplication users * one_hot # Result gives latent factors for only 2nd user tensor([[0.0, 0.0, 0.0], [0.4, 0.5, 0.6], [0.0, 0.0, 0.0]]) But this is inefficient, we can think how big will be the one-hot encoded matrices for users and, so libraries like PyTorch has Embedding Layers that do this lookup and retrieve the vector at that index, it creates the latent factor matrices in such a way that later we dont need to multiply one-hot encoded matrix to get latent factor for specific user/item. A simple array indexing gives us the row we want.\nWhy sharp curves of learning weights = overfitting We have non-linearity in labels, our universal function approxiator (neural nets) want to learn that non-linearity, So, what we learn? weights and bias. If the weights are increased more and more over time, what will happen?\nIt will be more and more sharp, that\u0026rsquo;s nothing but overfitting.\nWeight Decay or L2 Regularization Large weights in a neural network indicate that certain activations are becoming more active compared to others over time. We calculate loss to determine the disparity between our predictions and the target, and then use it to compute gradients. preds - targs We adjust the weights to minimize the loss, with the frequency of adjustments determined by the learning rate. Adding a large term to artificially inflate the loss can cause the gradient descent algorithm to decrease the weights more than necessary. This aggressive adjustment of weights occurs because the stochastic gradient descent algorithm perceives a steep gradient in the opposite direction when the loss is artificially increased. However, in practical scenarios, directly augmenting the loss would be inefficient. Instead, a large constant is added directly when calculating the gradient, effectively increasing its magnitude without inflating the loss. When the loss is higher, indicating poorer model performance, the gradient will typically have a larger magnitude, suggesting a steeper slope in the loss landscape. Following the direction opposite to the gradient with appropriate step size allows gradient descent to efficiently navigate the landscape and converge to a better set of model parameters, thereby minimizing the loss.\nWhether a weight becomes smaller or larger depends on the sign of the gradient for that weight and the direction opposite to it that is followed during optimization.\nWe calculate the gradient, the step function makes weights more less that it should be. This is called Weight Decay.\nIn practice, it would be very inefficient, we can do this increasing of loss task directly while calculating gradient.\nd/dp p^2 = 2p^(2-1) = 2p\nUnder-the-hood, this is the grad calculation, right?:\nparams.grad += 2 * params\nWe can add a constant wd Weight Decay, so big that it will make it twice as big, so we can skip the inefficient sum of weights whole squared part.\nparams.grad += 2 * params * wd \u0026lt;- Weight decay constant\nMovie Recommendation System with Embeddings (MovieLens Dataset) Follow kaggle notebook for more, but I will just create a Embedding layers and our Recommendation system\u0026rsquo;s model for scratch here:\nWhat the embedding is, guessed?\u0026ndash; random numbers for each users and for each movies. And they are learnable. Which means return_grad=True.\ndef create_embs(size): return nn.Parameters(torch.zeros(*size).normal_(0, 0.01)) torch.tensor.normal_?3\nclass DotProductModel(nn.Module): def __init__(self, n_users, n_items, n_factors, y_range=(0, 5.5)): self.user_factors = create_embs([n_users, n_factors]) self.item_factors = create_embs([n_items, n_factors]) self.user_bias = create_embs([n_users]) self.item_bias = create_embs([n_items]) self.y_range = y_range def forward(self, x): # assuming x is [userid, itemid] # we need dot product of user*item matrix users = self.user_factors[:, 0] # every row\u0026#39;s userid items = self.item_factors[:, 1] # every row\u0026#39;s itemid res = (users*items).sum(dim=1) # keep the dim to 1 # add bias res += self.user_bias[x[:,0]] + self.item_bias[x[:,1]] return sigmoid_range(res, *self.y_range) # unpack y_range and pass Direction and Distance of embeddings The embedding layers are hard to understand directly, but Principal Component Analysis (PCA) is used to get the underlying directions in matrix.\nCold Start Problem / Bootstrapping?\nThe biggest challenge with collaborative filtering in practice is, when you have no data for the user, how would you recommend it? Or when you have a complete new product in you database, whom would you recommend it?\nTaking the average just works, taking the average for the sci-fi may high compared to average of action factor, It would probably be better to pick a particular user to represent average taste lol.\nOr, like what Netflix, amazon prime do for new users? (create form, like what genre you like, what actors you like etc)\nOtakus are kinda poison to our embeddings Imagine, a small number of users ended up setting up recommendation for the complete database. For instance, people who watch anime, tends to only review anime\u0026rsquo;s and this result in the overall recommendation\u0026rsquo;s direction to incline toward anime\u0026rsquo;s. This result in getting some anime\u0026rsquo;s in Top ten movies.\nThis is a database problem, this can be solved by hiring a good database maintainer, no?\nIn the end, this is coming back to how to avoid disaster when rolling out any kind of machine learning system. It’s all about ensuring that there are humans in the loop; that there is careful monitoring, and a gradual and thoughtful rollout.\nYo, what we saw above is just a simple dot product, there are 2 ways that I learned:\nGet the latent factors (embeddings) and predict scores with simple Dot products. when data is simple and big I assume, for more speed and efficiency? Other is to get the latent factors and concat them (movies_embs + user_embs) and pass them to neural net layers. \u0026ndash;The Deep Learning Approach Look below the image of Google play\u0026rsquo;s rec sys.\nEntity Embedding Paper and Google Play\u0026rsquo;s Recommendation system\nGoogle Play\u0026rsquo;s recommendation system paper4\nThis entity embedding specifically refers to the second part, when we have embeddings for categorical variables, we can concatenate them (the embedding) with RAW categorical data, (either one hot encoded or ordinal) and use that result to feed into Neural network.\nthis below is from \u0026lsquo;Practical Deep Learning for Coders book\u0026rsquo; worth sharing\nOther thing that I learned, that want to remember is (might be wrong on this)\nBig orgs doesn\u0026rsquo;t create better architectures (their researchers do), How the researchers do? on addition to working on researches, they stay up-to-date with research on that specific domain they are working, for example NLP, Computer Vision etc. When they like some new discoveries, they keep that. On top level, this could be more easily understood if you think of this example: there are startups like Figma which provides better features from the already available products like Adobe, and later Adobe buys it. The same thing happens with architectures - maybe they see a breakthrough research, if open source research, they test it and if they get success they spent their resources in that, who knows.\nSources\nMovieLens dataset\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nMy kaggle Notebook\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nnormal_\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nWide and Deep Learning for Recommender System\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://akash5100.github.io/posts/2024-01-15-embeddings/","summary":"\u003ch3 id=\"table-of-contents\"\u003eTable of contents\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#collaborative-filtering\"\u003eCollaborative Filtering\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#a-tabular-dataset---movie-recommendation-system\"\u003eA Tabular Dataset -\u0026gt; Movie Recommendation System?\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#embeddings\"\u003eEmbeddings?\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#speed-up-the-calculation-of-scores\"\u003eSpeed up the calculation of scores\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#why-sharp-curves-of-learning-weights--overfitting\"\u003eWhy sharp curves of learning weights = overfitting\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#weight-decay-or-l2-regularization\"\u003eWeight Decay or L2 Regularization\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#movie-recommendation-system-with-embeddings-movielens-dataset\"\u003eMovie Recommendation System with Embeddings (MovieLens Dataset)\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#direction-and-distance-of-embeddings\"\u003eDirection and Distance of embeddings\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#otakus-are-kinda-poison-to-our-embeddings\"\u003eOtakus are kinda poison to our embeddings\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eI will write a follow up blog about Regression and Random Forest, today completes the half of the first month of 2024, I learned lots of stuffs that I am interested and participated on a Kaggle compe, predicing Energy consumption and production using solar panels, just to try what I learned and now I am one step closer to understanding the \u0026ldquo;Attention\u0026rdquo;. I will continue this journey and still be writing this blog for future me like creating notes, maybe in more readable form.\u003c/p\u003e","title":"Embeddings in Recommendation Systems"},{"content":"Table of contents Hyperparameters I coded the Titanic dataset in a simple MLP Regression and Summarizing Loss functions Source Multi label classification refers to the problem of identifying the categories of objects in images that may not contain exactly one type of object. So each data can have either single or multiple label(s). Example, a image has car, bicycle, person, tree.\nWhy we cant use softmax and NLL loss?\nSoftmax outputs values that sum to 1, and because the use of exp it tends to push on activation to be much larger than the others.\nNll loss, returns the value of just one activation: the single activation corresponding with the single label for an item. This doesn\u0026rsquo;t make sense when we have multiple labels.\nBinary Cross Entropy The binary cross entropy, measures the difference between the predicted probs and the actual labels, rewarding the accurate preds and penalizing the wrongs.\nImplementing it:\n# targets is one-hot encoded def binary_cross_entropy(x, targets, sigmoid=True): if sigmoid: x = x.sigmoid() x = torch.where(targets==1, x, 1-x) # if label is 1, return x (prob) else 1-x return -x.log().mean() this means, for every prediction, check if the label is correct or not. If the label is 1, return the predicted probability; otherwise, return 1 minus the predicted probability. Then, take the logarithm and calculate the mean over all predictions to get the Binary Cross Entropy loss. This function helps the model learn by penalizing deviations from the actual labels and encouraging accurate predictions.\nAccuracy for multi-label dataset (Threshold) We need the accuracy function to calculate the accuracy for all the labels for single image. After applying the sigmoid to our activation, we need to decide which ones are 0s and which ones are 1s, with the help of threshold. Each value above the threshold will be considered as a 1 and each value lower than the threshold will be considered 0.\n\u0026gt;\u0026gt;\u0026gt; threshold = 0.7 \u0026gt;\u0026gt;\u0026gt; ((inp\u0026gt;threshold)==targets.bool()).float().mean() Hyperparameters Hyperparameters are parameters that are not learned during the training process but need to be set before training. Droprate, Epoch, Learning Rate, Threshold, Batch Size etc.\nSetting the threshold of accuracy? I learned a technique where, we create a gradually spread tensor. And for each generate threshold, we check accuracy.\nI coded the Titanic dataset in a simple MLP What\u0026rsquo;s in the notebook? summarizing\u0026mdash;\nThe dataset is csv format with some information about every passenger. I have to predict whether passenger survived or not. A binary output. The first few question I asked myself are:-\nWhat should be the output layer activation? \u0026ndash; Sigmoid!\nWhat I will use for the activation for hidden layers? - Relu or Leaky Relu! (I know it is known for better and fast learning)\nWhat loss function I should be using?\nOkay this is interesting, I thought MSE because it is good for big outliers and another thing is I thought its a regression problem, but the problem involves binary classification, not regression. (Regression typically deals with predicting continuous values)\nThere are some data cols that I dont need so I will be dropping them out before feeding data to neural net.\nDealing with non-numerical data in the dataset.\nSummarizing it, the process is like- Dataset -\u0026gt; Data preprocessing (there are 10 cols and 800 approx rows) -\u0026gt; 800x10 -\u0026gt; 10x128 (relu) -\u0026gt; 128x1 (sigmoid). If output is \u0026gt;0.5 then its True else False.\nclass TitMLP(nn.Module): def __init__(self): super(TitMLP, self).__init__() self.l1 = nn.Linear(10, 128, bias=False) self.l2 = nn.Linear(128, 1, bias=False) self.sigmoid = nn.Sigmoid() def forward(self, x): x = F.relu(self.l1(x)) x = self.l2(x) x = self.sigmoid(x) return x Regression and Summarizing Loss functions Like I made mistake, I can use MSE for Titanic, but no. Think more about what the problem actually is and what loss function I should use.\nCross Entropy \u0026ndash; Single Label classification Binary Cross Entropy \u0026ndash; Multi Label classifcation MSE \u0026ndash; Regression Source Notebook ","permalink":"https://akash5100.github.io/posts/2023-12-22-titanic/","summary":"\u003ch3 id=\"table-of-contents\"\u003eTable of contents\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#hyperparameters\"\u003eHyperparameters\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#i-coded-the-titanic-dataset-in-a-simple-mlp\"\u003eI coded the Titanic dataset in a simple MLP\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#regression-and-summarizing-loss-functions\"\u003eRegression and Summarizing Loss functions\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#source\"\u003eSource\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eMulti label classification refers to the problem of identifying the categories of objects in images that may not contain exactly one type of object. So each data can have either single or multiple label(s). Example, a image has car, bicycle, person, tree.\u003c/p\u003e\n\u003chr\u003e\n\u003cbr /\u003e\n\u003cp\u003e\u003cstrong\u003eWhy we cant use softmax and NLL loss?\u003c/strong\u003e\u003c/p\u003e","title":"Cross Entropy in Classification"},{"content":"Table of contents Understanding Softmax Logarithm Finding a right Learning Rate - A technique Unfreezing \u0026amp; Transfer Learning Unfreezing? Discriminative learning rate To learn the foundation very clearly, I coded MLP, from scratch and trained MNIST dataset. (it was a 3 vs 7 model, a binary classifier). For that I used a Linear function in each neuron, and Relu as activation. and for the final layer I used Sigmoid. I wanted to expand this model from just a binary classifier to multi-class classifier (where each instance belongs to one and only one class) I learned about Softmax activation that can be used in the final layer and then creating a loss function for MNIST model. The first step is to load the dataset into numpy array. def fetch(url): import requests, os, numpy, hashlib, gzip fp = os.path.join(\"/tmp\", hashlib.md5(url.encode('utf-8')).hexdigest()) if os.path.isfile(fp): with open(fp, \"rb\") as f: dat = f.read() else: with open(fp, \"wb\") as f: dat = requests.get(url).content f.write(dat) return numpy.frombuffer(gzip.decompress(dat), dtype=numpy.uint8).copy() X_train = fetch(\u0026quot;http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz\u0026quot;)[0x10:].reshape((-1, 28, 28)) Y_train = fetch(\u0026quot;http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz\u0026quot;)[8:] X_test = fetch(\u0026quot;http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz\u0026quot;)[0x10:].reshape((-1, 28, 28)) Y_test = fetch(\u0026quot;http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz\u0026quot;)[8:] With this\u0026ndash; X_train, Y_train and X_test, Y_test ready to train and validate. The next step is implementing Softmax for the final layer of MLP that I code, so Understanding Softmax The normalization of data:\n\u0026gt;\u0026gt;\u0026gt; a = [2,3,4] \u0026gt;\u0026gt;\u0026gt; [i/sum(a) for i in a] [0.2222222222222222, 0.3333333333333333, 0.4444444444444444] \u0026gt;\u0026gt;\u0026gt; sum([i/sum(a) for i in a]) 1.0 \u0026gt;\u0026gt;\u0026gt; This is done to ensure that the activation are all between 0 and 1, and that they sum to 1.\nSoftmax is similar to Sigmoid, Sigmoid gives back a number between 0 and 1, but what if we have more categories in our target, such as 0-9 digits, that means we will need more activation than just a single column, we need activation per category.\nThis is basically Softmax but in practice, we do this:\nfrom math import exp \u0026gt;\u0026gt;\u0026gt; a = [2,3,4] \u0026gt;\u0026gt;\u0026gt; e = [exp(i) for i in a] \u0026gt;\u0026gt;\u0026gt; e [7.38905609893065, 20.085536923187668, 54.598150033144236] # the number bigger than others, is now exponentially bigger than others :) \u0026gt;\u0026gt;\u0026gt; [i/sum(e) for i in e] \u0026gt;\u0026gt;\u0026gt; sum([i/sum(a) for i in a]) 1.0 We take exponential of each item in the list. This ensures that all our numbers are positive and then dividing by the sum ensures we are going to have a bunch of numbers that add up to 1.\nExponential also has a nice property: if one of the number in our activations x is slightly bigger than others, the exponential will amplify this since it grows exponentially, which means that in the softmax, that number(which is bigger than others) will be closer to 1. (see in the above example)\nso the softmax can be coded as:\ndef softmax(x): e = [exp(i) for i in x] return [i/sum(e) for i in e] # or more better def softmax(x): return [exp(i) / sum(exp(j) for j in x) for i in x] Let\u0026rsquo;s test this, for the blog I would be using pytorch instead of my scratch implementation as understanding is all that matters.\nfor example we have 4 image predictions and have 10 possible class (0-9)\n\u0026gt;\u0026gt;\u0026gt; import torch \u0026gt;\u0026gt;\u0026gt; acts = torch.randn((4, 10)) \u0026gt;\u0026gt;\u0026gt; acts tensor([[-1.4481, -0.1757, -0.7788, -1.0553, 0.4467, -0.6139, 0.4489, 0.8270, 0.0285, 0.1228], [ 0.1568, 1.3101, 0.7205, -0.0426, -1.8226, -1.1551, 0.9280, 0.7243, -1.7663, 1.4627], [ 1.5100, -0.8356, 0.1867, -1.0825, 0.0181, -0.9047, 0.4021, 1.0330, 1.9762, -1.5650], [ 0.4135, 0.3576, -0.3664, 1.2406, -1.0209, 1.3133, -1.3655, -1.4199, 0.5766, 1.4867]]) acts.sigmoid() tensor([[0.1903, 0.4562, 0.3146, 0.2582, 0.6099, 0.3512, 0.6104, 0.6957, 0.5071, 0.5307], [0.5391, 0.7875, 0.6727, 0.4893, 0.1391, 0.2396, 0.7167, 0.6735, 0.1460, 0.8119], [0.8191, 0.3025, 0.5466, 0.2530, 0.5045, 0.2881, 0.5992, 0.7375, 0.8783, 0.1729], [0.6019, 0.5885, 0.4094, 0.7757, 0.2649, 0.7881, 0.2033, 0.1947, 0.6403, 0.8156]]) now Softmaxing them:\n\u0026gt;\u0026gt;\u0026gt; sm_acts = acts.softmax(dim=1) \u0026gt;\u0026gt;\u0026gt; sm_acts tensor([[0.0235, 0.0839, 0.0459, 0.0348, 0.1564, 0.0541, 0.1567, 0.2287, 0.1029, 0.1131], [0.0670, 0.2124, 0.1178, 0.0549, 0.0093, 0.0181, 0.1450, 0.1183, 0.0098, 0.2475], [0.2303, 0.0221, 0.0613, 0.0172, 0.0518, 0.0206, 0.0761, 0.1429, 0.3671, 0.0106], [0.0846, 0.0800, 0.0388, 0.1935, 0.0202, 0.2081, 0.0143, 0.0135, 0.0996, 0.2475]]) \u0026gt;\u0026gt;\u0026gt; sm_acts[0].sum() tensor(1.) Say, target (label) of the 4 images are:\ntarget = torch.tensor([1,4,5,6]) # first image is the image of 1, the second 4 and so on. \u0026gt;\u0026gt;\u0026gt; sm_acts[0] # first image tensor([0.0235, 0.0839, 0.0459, 0.0348, 0.1564, 0.0541, 0.1567, 0.2287, 0.1029, 0.1131]) \u0026gt;\u0026gt;\u0026gt; sm_acts[0][1] # this is the prediction for 1st image tensor(0.0839) This could be traversed more easily using indexing in python.\n\u0026gt;\u0026gt;\u0026gt; idx = range(4) # we have 4 image \u0026gt;\u0026gt;\u0026gt; sm_acts[idx, target] tensor([0.0839, 0.0093, 0.0206, 0.0143]) # the first image\u0026#39;s prediction is 0.0839 and so on. (match them with target) :) and pytorch provide a function that does exactly the same thing\n\u0026gt;\u0026gt;\u0026gt; import torch.nn.functional as F \u0026gt;\u0026gt;\u0026gt; F.nll_loss(sm_acts, target, reduction=\u0026#39;none\u0026#39;) tensor([-0.0839, -0.0093, -0.0206, -0.0143]) well, its negative. umhmmm???\nWhen applying log afterwards, we will have negative numbers, so: Negative Log Likelihood\n\u0026gt;\u0026gt;\u0026gt; -sm_acts[idx, target] tensor([-0.0839, -0.0093, -0.0206, -0.0143]) Logarithm We are using probabilities, and probabilities cannot be smaller than 0 or greater than 1. Than means our model will not care whether it predicts 0.99 or 0.999. Those numbers are very close together \u0026ndash; but in another sense, 0.999 is 10 time more confident than 0.99. So we want to transform our numbers between 0 and 1 to instead be between negative infinity and infinity. Logarithm, exactly does that.\nSome important points to remember about (why log)-\nLogarithm is used to create modification in either very very small number or very very large number. Logarithms are used to transform exponential growth or decay into linear relationships for easier analysis. operator : opposite\n+ : - x : / exponential : logarithm x^2 : 2 log x natural log has base e and common log has base 10\nWhen we first take the softmax, and then the log likelihood\nTaking the logarithm of the predictions (logits -\u0026gt; softmax -\u0026gt; preds) is a common step in computing the cross-entropy loss or negative log-likelihood loss. In the example, we calculated the softmax probabilities (sm_acts).\nSo, we take log of probabilites (sm_acts) and then use target to calculate loss:\n\u0026gt;\u0026gt;\u0026gt; torch.log(sm_acts) tensor([[-3.7508, -2.4781, -3.0813, -3.3581, -1.8553, -2.9169, -1.8534, -1.4753, -2.2740, -2.1795], [-2.7031, -1.5493, -2.1388, -2.9022, -4.6777, -4.0118, -1.9310, -2.1345, -4.6254, -1.3963], [-1.4684, -3.8122, -2.7920, -4.0628, -2.9604, -3.8825, -2.5757, -1.9456, -1.0021, -4.5469], [-2.4698, -2.5257, -3.2493, -1.6425, -3.9021, -1.5697, -4.2475, -4.3051, -2.3066, -1.3963]]) ChatGPT's explanation of logarithm: The logarithm of probabilities has the effect of compressing the range of values. Specifically: - Values close to 1 in the original probabilities become values close to 0 in the log-transformed probabilities. - Values between 0 and 1 become negative in the log-transformed probabilities. - Very small values become more negative in the log-transformed probabilities, making them stand out and avoiding numerical instability. In summary, taking the logarithm helps in numerical stability and provides a more interpretable scale, where differences in values correspond to changes in the original probabilities. So, after log, we see the preds:\n\u0026gt;\u0026gt;\u0026gt; import torch.nn.functional as F \u0026gt;\u0026gt;\u0026gt; l_sm_acts = torch.log(sm_acts) # log of preds \u0026gt;\u0026gt;\u0026gt; preds = l_sm_acts[idx, target] # log of preds for target \u0026gt;\u0026gt;\u0026gt; preds tensor([-2.4781, -4.6777, -3.8825, -4.2475]) \u0026gt;\u0026gt;\u0026gt; -preds # negate it to make output positive tensor([2.4781, 4.6777, 3.8825, 4.2475]) \u0026gt;\u0026gt;\u0026gt; -preds.mean() # mean tensor(3.8215) # this is the measure of difference between preds and target \u0026gt;\u0026gt;\u0026gt; ################ Torch #################### \u0026gt;\u0026gt;\u0026gt; F.nll_loss(torch.log(sm_acts), target) tensor(3.8215) \u0026gt;\u0026gt;\u0026gt; we first take the softmax, and then the log likelihood of that\u0026ndash; that combination is called cross-entropy loss.\nsoftmax (preds) -\u0026gt; log -\u0026gt; mean of the logs of preds\nThe mean of the negated log probabilities is a way to measure the dissimilarity between the predicted probabilities and the true labels.\nMinimizing this loss during training helps the model improve its ability to correctly classify inputs.\nFinding a right Learning Rate - A technique This is just a note to a technique from a research by Leslie Smith.\nStart with very very small LR and exonentially increase it untill we see the loss rising again (for a mini-batch) The Learning Rate with minimum loss is 1e-1 (1 * 10**-1) so Learning Rate for the model we can say is 1e-1/10, which is 1e-2 (1 * 10**-2) for a LR, we care only magnitude Unfreezing \u0026amp; Transfer Learning Using a pretrained model (trained weights) to use it for a new task, is called Transfer Learning.\nUnfreezing? In the context of neural networks, \u0026ldquo;unfreezing\u0026rdquo; refers to allowing the weights of certain layers to be updated during training. Typically, when you load a pretrained model, you freeze the weights of the earlier layers, preserving the learned features. This is done to avoid destroying the valuable information encoded in these layers.\nHowever, as you move towards the end of the network, the layers become more task-specific. Unfreezing these later layers allows the model to adapt and learn representations that are more relevant to your specific problem.\nhere is an example:\nfirst few layers \u0026ndash; it learns the details like curves, shapes, line.. we can use that. (freeze) last final layers \u0026ndash; it learns specific details like cat, dog etc. We dont need that in digit classifier for example. (so unfreeze the weights) Discriminative learning rate The learning rate determines how quickly the model should adapt during training. In the example above, we allocate less training time to the initial layers, as they capture more general features. Conversely, we allocate more time to the later layers, allowing them to learn task-specific features more thoroughly.\nSo, these are the steps to transfer learning: - load pretrained model - freeze initial layers - unfreeze later layers - train some time (I saw 3 epoch in a ImageNet example) - unfreeze all and set discriminative learning rate - train again","permalink":"https://akash5100.github.io/posts/2023-12-11-nll_loss/","summary":"\u003ch3 id=\"table-of-contents\"\u003eTable of contents\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#understanding-softmax\"\u003eUnderstanding Softmax\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#logarithm\"\u003eLogarithm\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#finding-a-right-learning-rate---a-technique\"\u003eFinding a right Learning Rate - A technique\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#unfreezing--transfer-learning\"\u003eUnfreezing \u0026amp; Transfer Learning\u003c/a\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#unfreezing\"\u003eUnfreezing?\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#discriminative-learning-rate\"\u003eDiscriminative learning rate\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eTo learn the foundation very clearly, I coded MLP, from scratch and trained MNIST dataset. (it was a 3 vs 7 model, a binary classifier). For that I used a Linear function in each neuron, and Relu as activation. and for the final layer I used Sigmoid. I wanted to expand this model from just a binary classifier to multi-class classifier (where each instance belongs to one and only one class) I learned about Softmax activation that can be used in the final layer and then creating a loss function for MNIST model.\n\u003cbr /\u003e\u003c/p\u003e","title":"Logarithms in Deep Learning"},{"content":"A artificial neural network can learn (almost) anything, and so its called a universal function approximator. To understand how it works, we need to know function.\nFunction, let\u0026rsquo;s say f(x) is just a system of inputs and outputs, a number in, a number out.\nx -\u0026gt; f(x) -\u0026gt; y\nWe give a input x, and it outputs y. We can plot all the functions on a graph, where it gives an output for an input. What is important is, if you know a function you can always calculate the output (y) for a given input (x).\nBut let\u0026rsquo;s say, we dont know what the function is, but we know some of the inputs and outputs. Is there a way we can reverse engineer that to produce that function?\nWe can still capture some x\u0026rsquo;s and y\u0026rsquo;s, making predictions. What we need to do is function approximation and more generally a function approximator.\nf(x) ≈ function approximator ≈ N(x)\nThat is what a neural network is.\nNeural networks are made up of neurons and a neuron itself is just a function, it can take any number of input and gives one output. Each inputs are multiplied together by a weight and added together along with a bias. The weights and bias makes up the parameters of the neuron, and values of weights and bias can change as the network learn.\nN(x1, x2, ...,xn) = w1*x1 + w2*x2 + ... wn*xn + b # or simply N(x) = w*x + b Neuron is building block and it can be combined with other neuron to form a more complicated function, one built from lots of linear functions.\nThere is one big problem, linear function can combine to give a linear function. We need to make something more than just a line, we need non linearity.\nLet's use ReLU, we use it as our activation function. means we simply apply it to our previous naive neuron. ReLU(x) = max(x,0) N(x) = max((w*x + b), 0) How do we find the weights and biases automatically? The most common algorithm for this is called Backpropogation.\nThe time I wrote this blog, I was coding a neural net(from scratch) to train MNIST dataset, a dataset of handdrawn numbers (0-9) so that the neural net can classify any input.\nTo train images, let\u0026rsquo;s say 3x3 pixel image (just an example)\nWe flatten the matrix (image represented on 2D array)\nand the same thing, let\u0026rsquo;s represent the pixels with 0-9.\nThis is the dot product of Input array and weights with a bias.\nTo take the dot product multiply each input by each weight and then add them all up.\nThis dot product is passed into an activation function, in this case a ReLU.\nmax(-0.26, 0) = 0 We feed the original inputs to a layer of neurons, each with their own weight and with their own learned value. The values of the weights and biases are calculated through the training process. We give the network input, and it produces an output. We compare the output with the actual output. This comparison is called the loss, which quantifies the difference between our prediction and the actual value.\nWe now have to find out, what changes we can make to weights and biases of these neural net to reduce the loss. As we dont know what the function is in the neuron, we need to make changes to weights to see that if that change makes the loss go up or down. (Bruh! millions of neuron and for each neuron how many calculations???)\nCalculating Gradient\nThe one magic step is the bit where we calculate the gradients. We use calculus as a performance optimization; it allows us to more quickly calculate whether our loss will go up or down when we adjust our parameters up or down. In other words, the gradients will tell us how much we have to change each weight to make our neural net better. Derivative of a function tells you how much a change in its parameters will change its result. The key point about a derivative is this: for any function, such as the quadratic function, we can calculate its derivative. The derivative is another function. It calculates the change, rather than the value. For instance, the derivative of the quadratic function at the value 3 tells us how rapidly the function changes at the value 3. More specifically, you may recall that gradient is defined as rise/run; that is, the change in the value of the function, divided by the change in the value of the parameter.\nWhen we know how our function will change, we know what we need to do to make it smaller. This is the key to machine learning: having a way to change the parameters of a function to make it smaller. Calculus provides us with a computational shortcut, the derivative, which lets us directly calculate the gradients of our functions.\nLife would probably be easier if backpropagation (backward pass) was just called calculate_gradient, but deep learning folks really do like to add jargon everywhere they can\nNow that we know what changes to make to weights, we do that and repeat the step.\nbtw, I forgot to write earlier \u0026ndash; we initialize the parameters to random values :)\nHere are the steps:\nadding some flavor to Gradient Decent makes it Stochastic gradient descent (SGD).\nChatGPT: Stochastic Gradient Descent (SGD) is like spice in the recipe of Gradient Descent. In standard Gradient Descent, you calculate the average gradient using the entire dataset, which can be computationally expensive. In SGD, you spice things up by randomly selecting a single data point or a small batch of data points for each iteration. This randomness introduces noise, but it speeds up the process, making it more like a spicy, fast-paced version of Gradient Descent.\n","permalink":"https://akash5100.github.io/posts/2023-11-04-algo_behind_universal_function_approximator/","summary":"\u003cp\u003eA artificial neural network can learn (almost) anything, and so its called a universal function approximator. To understand how it works, we need to know function.\u003c/p\u003e\n\u003cp\u003eFunction, let\u0026rsquo;s say f(x) is just a system of inputs and outputs, a number in, a number out.\u003c/p\u003e\n\u003cp\u003ex -\u0026gt; f(x) -\u0026gt; y\u003c/p\u003e\n\u003cp\u003eWe give a input x, and it outputs y. We can plot all the functions on a graph, where it gives an output for an input. What is important is, if you know a function you can always calculate the output (y) for a given input (x).\u003c/p\u003e","title":"Algorithm behind universal function approximator"},{"content":"While there are many blogs and tutorials available on accessing a Virtual Cloud Network in a VM instance, such as this comprehensive guide(by oracle). Some of the VM instances I\u0026rsquo;m working with are not vanilla; they come with the distributor\u0026rsquo;s packet blockers and they block traffics on all the ports.\nI worked around Iptables a default firewall for linux, I think. It is used to set up, maintain, and inspect the tables of IP packet filter rules in the Linux kernel. Several different tables may be defined. Each table contains a number of built-in chains and may also contain user-defined chains.\nI was deploying a Teamspeak server.\nThank you chatGPT:\nThe default ports for a TeamSpeak server are as follows: 1. **Voice Communication (UDP)**: - Default Port: 9987 - This is the default UDP port used for the TeamSpeak voice communication server. 2. **File Transfer (TCP)**: - Default Port: 30033 - This is the default TCP port used for file transfers in TeamSpeak. File transfers are often used for uploading and downloading files, such as server icons or avatars. 3. **Server Query (TCP)**: - Default Port: 10011 - This is the default TCP port used for server queries. Server queries are typically used for administrative purposes, allowing you to manage and configure the TeamSpeak server programmatically. These are the default ports used by TeamSpeak, but it\u0026#39;s worth noting that server administrators can configure different ports if needed. If you\u0026#39;re connecting to a TeamSpeak server or setting up your own, make sure to check the server\u0026#39;s configuration to confirm the specific ports it\u0026#39;s using. To make this endeavor successful, I needed to open specific ports on the VM instances, and that\u0026rsquo;s where iptables comes into play.\nAllowing traffic on restricted ports using iptables:\niptables -A INPUT -p tcp --dport 30033 -j ACCEPT\nAdding a rule to allow incoming TCP traffic on port 30033.\niptables -A INPUT -p tcp --dport 10011 -j ACCEPT\nGranting access to incoming TCP traffic on port 10011.\niptables -A INPUT -p udp --dport 9987 -j ACCEPT\nAllowing incoming UDP traffic on port 9987.\nAdditional steps that I performed (self-reminder)\nIn addition to these iptables commands, I\u0026rsquo;ve also found it valuable to implement the following supplementary actions:\nArchiving the current iptables rules ensures that I can revert to a previous configuration if needed. iptables-save \u0026gt; ~/iptables-rules\nTemporarily disabling the firewall rules can be helpful for troubleshooting or testing purposes, though it\u0026rsquo;s essential to be aware of the potential security implications. sudo iptables --flush\nCreating backups of the IPv4 and IPv6 iptables rule files is a safety net against unintended changes or errors. sudo mv /etc/iptables/rules.v4 /etc/iptables/rules.v4.bak \u0026amp;\u0026amp; sudo mv /etc/iptables/rules.v6 /etc/iptables/rules.v6.bak\nand reboot instance.\nwith this, iptable will stop blocking (YOUR) malicious and hostile data packets.\n","permalink":"https://akash5100.github.io/posts/2023-09-30-firewall/","summary":"\u003cp\u003eWhile there are many blogs and tutorials available on accessing a Virtual Cloud Network in a VM instance, such as this \u003ca href=\"https://docs.oracle.com/en/learn/lab_virtual_network/index.html#introduction\"\u003ecomprehensive guide(by oracle)\u003c/a\u003e. Some of the VM instances I\u0026rsquo;m working with are not vanilla; they come with the distributor\u0026rsquo;s packet blockers and they block traffics on all the ports.\u003c/p\u003e\n\u003cp\u003eI worked around \u003ca href=\"https://linux.die.net/man/8/iptables\"\u003eIptables\u003c/a\u003e a default firewall for linux, I think. It is used to set up, maintain, and inspect the tables of IP packet filter rules in the Linux kernel. Several different tables may be defined. Each table contains a number of built-in chains and may also contain user-defined chains.\u003c/p\u003e","title":"Traffic on blocked ports"},{"content":"I learned about black hole, how are they possible? (just started). On the sunday morning, I started reading a paper about BBC Reith Lecture. It all started with Albert Einstein writing a paper in 1939 claiming that stars could not collaspe under gravity because matter could not be compressed beyond a certain point, many scientist thought the same but an American scientist John Wheeler, who in many ways is the hero of the black hole story. In his work in 1950s and 1960s, emphasized that many stars would eventually collapse and pointed out problems that possibility posed for theoretical physics. Durring most of the life of a normal star, over billons of years, it will support itself against its own gravity by thermal pressure caused by nuclear processes which convert hydrogen into helium.\nEventually, however the star will exhaust its nuclear fuel. The star will now contract. In some cases it may be able to support itself a \u0026ldquo;white dwarf\u0026rdquo; star.\nSingularity\nRobert Oppenheimer after getting the fame of atomic bomb (i now become death the destroyer of worlds) investigated the problem of what would happen if a star whose mass is greater than a white dwarf or neutron star when they exhausted their nuclear fuel? he showed that such a star could not be supported by outward pressure; and that if you take pressure out of the calculation, a uniform spherically systematic symmetric star would contract to a single point of infinite density. such a point is called a singularity.\nA singularity is what you end up with when a giant star is compressed to an unimaginably small point.\nAfter this a lot of interesting discoveries occured (am not gonna tell) but a dramatic advance in our understanding of these mysterious phenomena came with a mathematical discovery in 1970. This was that the surface area of the event horizon aka the boundary around the black hole. It always increases when additional matter or radiation falls into the black hole. This property suggests that there is a resemblance between the area of the event horizon of a black hole and conventional newtonian physics, specifically the concept of entropy in thermodynamics. Entropy can be regarded as a measure of the disorder of a system, or equivalently as a lack of knowledge of its precise state. The famous Second Law of Thermodynamics says that entropy always increases with time. this discovery (in 1970) was the first hint of this crucial connection.\nEntropy means the tendency for anything that has order to become more disordered as time passes - so, for example bricks neatly stacked to form a wall (low entropy) will eventually end up in a heap of dust (high entropy). And this process is described by the Second Law of Thermodynamics.\n","permalink":"https://akash5100.github.io/posts/2023-09-15-entropy/","summary":"\u003cp\u003eI learned about black hole, how are they possible? (just started). On the sunday morning, I started reading a paper about BBC Reith Lecture. It all started with \u003cstrong\u003eAlbert Einstein\u003c/strong\u003e writing a paper in 1939 claiming that stars could not collaspe under gravity because matter could not be compressed beyond a certain point, many scientist thought the same but an American scientist \u003cstrong\u003eJohn Wheeler\u003c/strong\u003e, who in many ways is the hero of the black hole story. In his work in 1950s and 1960s, emphasized that many stars would eventually collapse and pointed out problems that possibility posed for theoretical physics. Durring most of the life of a normal star, over billons of years, it will support itself against its own gravity by thermal pressure caused by nuclear processes which convert hydrogen into helium.\u003c/p\u003e","title":"I relearned entropy"},{"content":"","permalink":"https://akash5100.github.io/about/","summary":"About Akash Verma","title":"About"}]