blanketglossary

Transformer (deep learning)

Definition

In deep learning, the transformer is a family of artificial neural network architectures based on the multi-head attention mechanism, in which text is converted to numerical representations called tokens, and each token is converted into a vector via lookup from a word embedding table. At each layer, each token is then contextualized within the scope of the context window with other (unmasked) tokens via a parallel multi-head attention mechanism, allowing the signal for key tokens to be amplified and less important tokens to be diminished.

Related concepts

15.ai AAAI Conference on Artificial Intelligence AI21 Labs AI accelerator AI agent AI alignment AI anthropomorphism AI boom AI bubble AI data center AI effect AI literacy AI nationalism AI safety AI slop AI takeover AI veganism AI winter Action selection Activation function Active learning (machine learning)Adobe Firefly Adversarial machine learning Agent2Agent Aidan Gomez Alan Turing Alec Radford AlexNet Alex Graves (computer scientist)Alex Krizhevsky Alibaba Group Allen Newell AlphaDev AlphaFold AlphaGenome AlphaGeometry AlphaGo AlphaGo (film)AlphaGo Zero AlphaGo versus Fan Hui AlphaGo versus Ke Jie AlphaGo versus Lee Sedol AlphaStar (software)AlphaZero Andrej Karpathy Andrew Ng Anomaly detection Anthropic Applications of artificial intelligence Apprenticeship learning Arthur Mensch Artificial Intelligence Act Artificial Intelligence Cold War Artificial general intelligence Artificial human companion Artificial intelligence Artificial intelligence and copyright Artificial intelligence and elections Artificial intelligence arms race Artificial intelligence content detection Artificial intelligence in architecture Artificial intelligence in education Artificial intelligence in fiction Artificial intelligence in healthcare Artificial intelligence in mental health Artificial intelligence in video games Artificial intelligence visual art Artificial neural network Artificial superintelligence Ashish Vaswani Association rule learning Attention (machine learning)Attention Is All You Need Attention is all you need Audio signal processing Aurora (text-to-image model)AutoGPT Autoencoder Automated machine learning Automated reasoning Automated theorem proving Automatic summarization Autoregressive model Autoregressive models BERT (language model)BF16 BIRCH BLOOM (language model)Backpropagation Bag-of-words model Baidu Batch learning Batch normalization Bayesian network Bernard Widrow Bias–variance tradeoff Blackwell (microarchitecture)Block matrix Boltzmann machine Boosting (machine learning)Bootstrap aggregating Byte pair encoding CUDA CURE algorithm CVPR Cache (computing)Canonical correlation Chain-of-thought prompting ChatGPT Chatbot Chatbot psychosis Chinchilla (language model)Christopher D. Manning ChromaDB Circuit (neural network)Claude (language model)Claude Shannon Cliff Shaw Cluster analysis Code generation Coefficient of determination Cohere Common Crawl Comparison of deep learning software Competition in artificial intelligence Complex number Computational learning theory Computational linguistics Computer chess Computer programming Computer vision Conditional random field Conference on Neural Information Processing Systems Confusion matrix Conjugate gradient method Constitutional AI Context window Contextualization (computer science)Convolution Convolutional neural network CrewAI Crowdsourcing Curriculum learning DALL-E DBRX DBSCAN Daniel Kokotajlo (researcher)Dario Amodei Data augmentation Data cleaning Data mining Data set David Silver (computer scientist)Decision tree learning DeepDream DeepSeek DeepSeek (chatbot)Deep learning Deep learning speech synthesis Demis Hassabis Density estimation Differentiable neural computer Diffusion model Diffusion process Dimensionality reduction Dot-product attention Dot product Double descent DreamBooth Dream Machine (text-to-video model)ECML PKDD ELMo Echo state network EfficientNet Electrochemical RAM EleutherAI ElevenLabs Elman network Elo rating system Empirical risk minimization Ensemble learning Environmental impact of artificial intelligence Ernie Bot Ethics of artificial intelligence Evaluation function Existential risk from artificial intelligence Expectation–maximization algorithm Explainable artificial intelligence FP16 Facial recognition system Factor analysis Fast weight Feature engineering Feature learning Feedforward neural network Fei-Fei Li Fine-tuning (deep learning)Fine-tuning (machine learning)Floating-point arithmetic Flux (text-to-image model)Foundation model Framework (computer science)Frank Rosenblatt François Chollet Future of Go Summit Fuzzy clustering GPT-1 GPT-2 GPT-3 GPT-4 GPT-4.1 GPT-4.5 GPT-4o GPT-5 GPT-5.5 GPT-J GPTZero GPT Image Gated Linear Unit Gated recurrent unit Gated recurrent units Gating mechanism Gato (DeepMind)Gemini (chatbot)Gemini (language model)Gemini Robotics Gemma (language model)Generative AI Generative adversarial network Generative artificial intelligence Generative engine optimization Generative model Generative pre-trained transformer Genie (AI model)Genie (world model)Geoffrey Hinton Glitch token GloVe Glossary of artificial intelligence Google Google AI Google Antigravity Google Assistant Google Brain Google DeepMind Google Gemini Google Labs Google Neural Machine Translation Google Pixel Google Translate Google Vids Google Workspace Gradient descent Grammar induction Gram–Schmidt process Grandmaster (chess)Graph neural network Graphical model Graphics processing unit Grok (chatbot)Hallucination (artificial intelligence)Handwriting recognition Hans Uszkoreit Herbert A. Simon Hidden Markov model Hierarchical clustering High Bandwidth Memory Higher-order neural network Highway network History of artificial intelligence Huawei PanGu Hugging Face Human-in-the-loop Human image synthesis Humanity's Last Exam Hyperparameter (machine learning)Hyperscale computing IBM Granite IBM Watson IBM Watsonx Ian Goodfellow Ideogram (text-to-image model)Ilya Sutskever Imagen (text-to-image model)Imitation learning In-context learning Inception (deep learning architecture)Independent component analysis Inductive bias Inference (machine learning)Inference engine Instruction tuning Integer Intelligent agent International Conference on Learning Representations International Conference on Machine Learning International Joint Conference on Artificial Intelligence Isolation forest Jais (language model)James Goodnight Jan Leike John Hopfield John McCarthy (computer scientist)John Schulman John von Neumann Joseph Weizenbaum Journal of Machine Learning Research Juergen Schmidhuber Jürgen Schmidhuber K-means clustering K-nearest neighbors algorithm Kernel machines Kling AI Knowledge distillation Kunihiko Fukushima LLM-as-a-Judge LLMs in higher education LMArena LSTMs LaMDA LangChain Language model Language model benchmark Language modeling Large language model Latent diffusion model Layer normalization LeNet Learning curve (machine learning)Learning rate Learning to rank Lethal autonomous weapon Lexical analysis Linear discriminant analysis Linear regression List of artificial intelligence companies List of artificial intelligence projects List of chatbots List of datasets for machine-learning research List of datasets in computer vision and image processing List of large language models Lists of open-source artificial intelligence software Llama.cpp Llama (language model)Local outlier factor Locality-sensitive hashing Logistic regression Long short-term memory Lookup table Loss function Loss functions for classification Lotfi A. Zadeh Low-rank approximation MMLU Machine Learning (journal)Machine learning Machine translation Mamba (deep learning architecture)Man bites dog Manus (AI agent)Marvin Minsky Master (software)Mean shift Mechanistic interpretability Memory paging Memtransistor Meta-learning (computer science)Meta AI Microsoft AI Microsoft Copilot Midjourney Minerva (model)Ming-Hsuan Yang MiniMax (company)Minimax Mira Murati Mistral AI Mixture of experts MobileNet Model Context Protocol Model compression MuZero Multi-agent reinforcement learning Multilayer perceptron Multimodal learning Music and artificial intelligence Mustafa Suleyman Naive Bayes classifier Named-entity recognition Nano Banana Nathaniel Rochester (computer scientist)Natural-language understanding Natural language Natural language generation Natural language processing Nemotron Network throughput Neural Turing machine Neural field Neural machine translation Neural network (machine learning)Neural radiance field Neural scaling law Neuro-symbolic AI Neuromorphic engineering Noam Shazeer Non-negative matrix factorization Normalization (machine learning)NotebookLM Nvidia Nvidia A100 Nvidia H100 OPTICS algorithm Oasis (Minecraft clone)Object-oriented programming Object hierarchy Occam learning Oliver Selfridge One-hot Online machine learning Ontology learning OpenAI OpenAI Codex (AI agent)OpenAI Codex (language model)OpenAI Five OpenAI o1 OpenAI o3 OpenAI o4-mini OpenVINO Open Neural Network Exchange Optical character recognition Oriol Vinyals Outline of machine learning Overfitting PaLM PagedAttention Parallel computing Parameter Paul Werbos Perceiver Perceptron Percy Liang Permutation matrix Perplexity Perplexity AI Phi (language model)Physics-informed neural networks Pipeline (Unix)Policy gradient method Precautionary principle Principal component analysis Probably approximately correct learning Project Debater Projection matrix Prompt engineering Prompt injection Proper generalized decomposition Protein structure prediction PyTorch Q-learning Quadratic function Quantum Artificial Intelligence Lab Quantum machine learning Quasi-Newton method Question answering Quoc V. Le Qwen RMSNorm Radial basis function kernel Random forest Random sample consensus ReLU Reasoning model Receiver operating characteristic Recraft Rectifier (neural networks)Recurrent neural network Recursive self-improvement Reflection (artificial intelligence)Regression analysis Regularization (mathematics)Regulation of artificial intelligence Regulation of artificial intelligence in the United States Reinforcement learning Reinforcement learning from human feedback Relevance vector machine Reservoir computing Residual neural network Restricted Boltzmann machine Retrieval-augmented generation Riffusion Robot Constitution Robot control Robotics Rule-based machine learning Runway (company)SGLang Sam Altman Seedance 2.0 Self-driving car Self-organizing map Self-play (reinforcement learning technique)Self-supervised learning Semantic analysis (machine learning)Semi-supervised learning Sentiment analysis Sepp Hochreiter Seppo Linnainmaa Seq2seq Sequence analysis Seymour Papert Shun'ichi Amari Sigmoid function Small-world network Small language model Softmax function Sora (text-to-video model)Sparrow (chatbot)Sparse dictionary learning Spectrogram Speculative decoding Speculative execution Speech-to-text Speech recognition Spiking neural network Stable Diffusion State of the art State–action–reward–state–action Statistical classification Statistical learning theory Statistical machine translation Stephen Grossberg Stochastic gradient descent Stochastic parrot Structured prediction Suno (platform)Supervised learning Support vector machine Symbolic artificial intelligence Synthetic data T-distributed stochastic neighbor embedding T5 (language model)Takeo Kanade Technology Innovation Institute Temporal difference learning TensorFlow TensorRT Tensor Processing Unit Text-to-image model Text-to-video model Text Summaries Text corpus Text summarization Textual entailment The MANIAC The Pile (dataset)Time series Timeline of artificial intelligence Timeline of machine learning Toeplitz matrix Tokenization Tokenization (lexical analysis)Topological deep learning Training, validation, and test data sets Transfer learning Transformer (deep learning architecture)U-Net Udio Uncanny valley Undetectable.ai Unsupervised learning VLLM Vanishing-gradient problem Vapnik–Chervonenkis theory Variational autoencoder Vector database Veo (text-to-video model)Vibe coding Vicuna LLM VideoPoet Virtual assistant Virtual politician Vision transformer Walter Pitts Warren Sturgis McCulloch WaveNet Weak artificial intelligence Web scraping Weight initialization Whisper (speech recognition system)Word2vec Word embedding Workplace impact of artificial intelligence World model (artificial intelligence)XAI (company)XLNet Xiaomi MiMo Yann LeCun Yoshua Bengio You.com

21 concepts already in your glossary