{"id":162,"date":"2024-05-09T12:52:31","date_gmt":"2024-05-09T12:52:31","guid":{"rendered":"https:\/\/wsw-int.de\/?p=162"},"modified":"2024-10-25T15:18:58","modified_gmt":"2024-10-25T15:18:58","slug":"xlstm-the-inventors-of-lstm-are-presenting-a-transformer-contender","status":"publish","type":"post","link":"https:\/\/multai.eu\/de\/xlstm-the-inventors-of-lstm-are-presenting-a-transformer-contender\/","title":{"rendered":"xLSTM: Die Erfinder des LSTM stellen einen Transformer-Kandidaten vor."},"content":{"rendered":"

Has Sepp Hochreiter done it again? After months of announcements, a group around the inventor of the LSTM finally published a paper<\/a> presenting \ud835\udc31\ud835\udc0b\ud835\udc12\ud835\udc13\ud835\udc0c to the world.<\/p>\n\n\n\n

Until the appearance of the Transformer in 2017, \ud835\udc0b\ud835\udc12\ud835\udc13\ud835\udc0c had been the go-to technology for a wide variety of sequence-related tasks, including text generation. Three limitations<\/p>\n\n\n\n

    \n
  1. the inability to revise storage decisions,<\/li>\n\n\n\n
  2. limited storage capacity, and<\/li>\n\n\n\n
  3. the need for sequential rather than parallel processing,<\/li>\n<\/ol>\n\n\n\n

    relegated LSTMs to second place behind Transformers.<\/p>\n\n\n\n

    The group proposes two types of new LSTM memory cells, baptized \ud835\udc2c\ud835\udc0b\ud835\udc12\ud835\udc13\ud835\udc0c and \ud835\udc26\ud835\udc0b\ud835\udc12\ud835\udc13\ud835\udc0c (see the graph from the original paper below). Both sLSTM and mLSTM are placed in a residual block (adding skip connections, similar to Transformers). These blocks can be stacked in various combinations, and thus constitute the complete \ud835\udc31\ud835\udc0b\ud835\udc12\ud835\udc13\ud835\udc0c architecture.<\/p>\n\n\n\n

    \"\"
    From the original paper.<\/figcaption><\/figure>\n\n\n\n