TAPAS模型解析表格問答系統

book_data = [
    {"chapter": 0, "name": "Introduction", "start_page": 1, "end_page": 11},
    {"chapter": 1, "name": "Text classification", "start_page": 12, "end_page": 48},
    {"chapter": 2, "name": "Named Entity Recognition", "start_page": 49, "end_page": 73},
    {"chapter": 3, "name": "Question Answering", "start_page": 74, "end_page": 120},
    {"chapter": 4, "name": "Summarization", "start_page": 121, "end_page": 140},
    {"chapter": 5, "name": "Conclusion", "start_page": 141, "end_page": 144}
]

table = pd.DataFrame(book_data)
table['number_of_pages'] = table['end_page'] - table['start_page']
table = table.astype(str)

table_qa = pipeline("table-question-answering")
queries = ["What's the topic in chapter 4?", "What is the total number of pages?", 
           "On which page does the chapter about question-answering start?", 
           "How many chapters have more than 20 pages?"]
preds = table_qa(table, queries)

for query, pred in zip(queries, preds):
    print(query)
    if pred["aggregator"] == "NONE":
        print("Predicted answer: " + pred["answer"])
    else:
        print("Predicted answer: " + pred["answer"])
    print('='*50)

asr = pipeline("automatic-speech-recognition")
from datasets import load_dataset
ds = load_dataset("superb", "asr", split="validation[:1]")

import soundfile as sf
def map_to_array(batch):
    speech, _ = sf.read(batch["file"])
    batch["speech"] = speech
    return batch
ds = ds.map(map_to_array)

pred = asr(ds[0]["speech"])
print(pred)
{'text': 'MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL'}

from transformers import CLIPProcessor, CLIPModel
from PIL import Image
import torch
import matplotlib.pyplot as plt

# 載入預訓練的 CLIP 模型和處理器
clip_ckpt = "openai/clip-vit-base-patch32"
model = CLIPModel.from_pretrained(clip_ckpt)
processor = CLIPProcessor.from_pretrained(clip_ckpt)

# 載入影像
image = Image.open("images/optimusprime.jpg")
plt.imshow(image)
plt.axis("off")
plt.show()

# 設定要比較的文字
texts = ["a photo of a transformer", "a photo of a robot", "a photo of agi"]
inputs = processor(text=texts, images=image, return_tensors="pt", padding=True)

# 進行預測
with torch.no_grad():
    outputs = model(**inputs)
    logits_per_image = outputs.logits_per_image
    probs = logits_per_image.softmax(dim=1)
    print(probs)

import torch
import torch.nn as nn
import torch.nn.functional as F

class SelfAttention(nn.Module):
    def __init__(self, embed_dim, num_heads):
        super(SelfAttention, self).__init__()
        self.embed_dim = embed_dim
        self.num_heads = num_heads
        self.head_dim = embed_dim // num_heads
        
        self.query_linear = nn.Linear(embed_dim, embed_dim)
        self.key_linear = nn.Linear(embed_dim, embed_dim)
        self.value_linear = nn.Linear(embed_dim, embed_dim)
        
        self.dropout = nn.Dropout(0.1)
    
    def forward(self, x):
        # x: [batch_size, sequence_length, embed_dim]
        batch_size = x.size(0)
        sequence_length = x.size(1)
        
        query = self.query_linear(x).view(batch_size, -1, self.num_heads, self.head_dim).transpose(1, 2)
        key = self.key_linear(x).view(batch_size, -1, self.num_heads, self.head_dim).transpose(1, 2)
        value = self.value_linear(x).view(batch_size, -1, self.num_heads, self.head_dim).transpose(1, 2)
        
        # 計算注意力分數
        attention_scores = torch.matmul(query, key.transpose(-1, -2)) / math.sqrt(self.head_dim)
        attention_weights = F.softmax(attention_scores, dim=-1)
        attention_weights = self.dropout(attention_weights)
        
        # 計算輸出
        output = torch.matmul(attention_weights, value).transpose(1, 2).contiguous().view(batch_size, sequence_length, self.embed_dim)
        
        return output

#### 內容解密：
1. **多頭注意力機制**：透過 `num_heads` 引數實作多頭注意力，每個頭獨立計算注意力分數，最後合併結果。
2. **查詢、鍵、值** 線性變換：使用 `query_linear`、`key_linear` 和 `value_linear` 將輸入 `x` 轉換為查詢、鍵和值向量。
3. **注意力分數計算**：透過點積計算查詢和鍵之間的相似度，並使用縮放因子 `math.sqrt(self.head_dim)` 進行標準化。
4. **Softmax 與 Dropout**：對注意力分數應用 Softmax 取得權重，並使用 Dropout 防止過擬合。
5. **輸出計算**：將注意力權重應用於值向量，得到最終輸出。

### Transformer 變體與應用

#### BERT 模型
BERT（Bidirectional Encoder Representations from Transformers）是根據 Transformer 編碼器的預訓練語言模型。它在多項 NLP 任務上取得了顯著的成果。

#### T5 模型
T5（Text-to-Text Transfer Transformer）將所有 NLP 任務統一為文字到文字的格式，進一步擴充套件了 Transformer 的應用範圍。

### Transformer 在實際案例中的表現

#### 文字分類別
Transformer 在文字分類別任務中表現出色，能夠捕捉文字中的關鍵資訊和上下文關係。

#### 問答系統
透過微調預訓練的 Transformer 模型，可以構建高效的問答系統，能夠準確地從文字中提取答案。

##### 此圖示說明瞭 Transformer 的整體架構，包括編碼器和解碼器的處理流程，以及自注意力機制和前饋神經網路的作用。