If we have data like: (from the example of previous blog) X – Y 'one', '.', 'two' – '.' '.', 'three', '.' – 'four' Where 3 words are used as input to predict 1 word from a vocabulary as an output, we can create a neural network architecture that takes three words as input and returns a prediction of the probability for each possible next word in the vocabulary. We will use three standard linear layers but with two tweaks! ...
Here are the main steps for language modeling: Tokenization– Converting Text into list of words (creating the vocab) Numericalization– Converting each word in vocab to number, by replacing them with their indices (simple!) The next steps are: Language Model Data Creation (X&Y) and Language Model Creation Jargon: Token One element of a list created by the tokenization process. It could be a word, part of a word (a subword), or a single character. ...
A language model is a model that is trained to guess the next word in a text (having read the ones before). This kind of task is called self-supervised learning. Jargon: self supervised learning Training a model using labels that are embedded in the independent variable, rather than requiring external labels. Self supervised learning is not usually used for the model that is trained directly, but instead is used for pretraining a model used for transfer learning. Self supervised learning is used to train a base model, and that base model is used to train different model for specific task like text classification!1 ...
Alright who cares about the follow up blog, here is a notion notes to save me. This is the notes for tweaking Random forest on a tabular dataset to squeeze some performance, I tried this on a Kaggle compe and it worked out pretty badly. Score of something like 2.xx. Maybe I Will improve it when time comes. zzz As we already initialized the embeddings of our users and products for example, We take the result of the embedding lookup and concatenate those activations together. This gives us a matrix that we can then pass through linear layers and nonlinearities in the usual way. ...
I will write a follow up blog about Regression and Random Forest, today completes the half of the first month of 2024, I learned lots of stuffs that I am interested and participated on a Kaggle compe, predicing Energy consumption and production using solar panels, just to try what I learned and now I am one step closer to understanding the “Attention”. I will continue this journey and still be writing this blog for future me like creating notes, maybe in more readable form. ...
Multi label classification refers to the problem of identifying the categories of objects in images that may not contain exactly one type of object. So each data can have either single or multiple label(s). Example, a image has car, bicycle, person, tree. Why we cant use softmax and NLL loss? Softmax outputs values that sum to 1, and because the use of exp it tends to push on activation to be much larger than the others. ...
To learn the foundation very clearly, I coded MLP, from scratch and trained MNIST dataset. (it was a 3 vs 7 model, a binary classifier). For that I used a Linear function in each neuron, and Relu as activation. and for the final layer I used Sigmoid. I wanted to expand this model from just a binary classifier to multi-class classifier (where each instance belongs to one and only one class) I learned about Softmax activation that can be used in the final layer and then creating a loss function for MNIST model. The first step is to load the dataset into numpy array. def fetch(url): import requests, os, numpy, hashlib, gzip fp = os.path.join("/tmp", hashlib.md5(url.encode('utf-8')).hexdigest()) if os.path.isfile(fp): with open(fp, "rb") as f: dat = f.read() else: with open(fp, "wb") as f: dat = requests.get(url).content f.write(dat) return numpy.frombuffer(gzip.decompress(dat), dtype=numpy.uint8).copy() X_train = fetch("http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz")[0x10:].reshape((-1, 28, 28)) Y_train = fetch("http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz")[8:] X_test = fetch("http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz")[0x10:].reshape((-1, 28, 28)) Y_test = fetch("http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz")[8:] With this– X_train, Y_train and X_test, Y_test ready to train and validate. The next step is implementing Softmax for the final layer of MLP that I code, so ...
A artificial neural network can learn (almost) anything, and so its called a universal function approximator. To understand how it works, we need to know function. Function, let’s say f(x) is just a system of inputs and outputs, a number in, a number out. x -> f(x) -> y We give a input x, and it outputs y. We can plot all the functions on a graph, where it gives an output for an input. What is important is, if you know a function you can always calculate the output (y) for a given input (x). ...