ローカルLLMでデータ加工したいということで、ollamaのstructured outputsをもう少し使ってみるメモです。多少複雑な構造でも使えそうなのか・・・？を試してみます。これまでのトライでは、日本語の場合、JSONのフォーマットが不正なエラーが生じるケースがありましたので、日本語では回避策をいれちゃいました。この辺りも徐々にFixされるのでしょう。英語の場合は問題なさそうですし、原文が英語なので素直に英語で回答させます。引き続き、ArXivにある論文のアブストラクトをもとにQAデータ作成を題材にします。
実行環境：
- Colab T4環境
- LLM動作はollamaで、モデルはollama公式のQwen2.5(7B)を利用しました。
結果：
英語でテストした結果、エラーもなくすんなりです。100件単位で数度繰り返しましたが出力したJSONフォーマット自体に問題はありませんでした。2byte系が末尾に来るときにはちょっくら注意でしょうか。ollamaありがたい。今回は、アブストラクトの情報を利用してQAデータセット作成を意識しました。文書全体を扱えるともっといい感じになるのかも。これでさらにQAデータセット作りがはかどります。
テスト内容概要：
手順概要
論文のアブストから、質問と回答、それに対する理由を可能な限りリストアップするものです。理由はさらに、重要性の説明、裏付け、客観性の評価を含みます。これらを定義します。
from typing import List, Dict
from pydantic import BaseModel, Fieldclass Reasoning(BaseModel):
importance: str = Field(..., description="重要性の説明")
evidence_support: str = Field(..., description="証拠による裏付け")
objectivity: str = Field(..., description="客観性の評価")class QAItem(BaseModel):
question: str = Field(..., description="質問内容")
answer: str = Field(..., description="回答内容")
reasoning: Reasoning = Field(..., description="回答の根拠付け")class QACollection(BaseModel):
QA_pairs: List[QAItem] = Field(..., description="複数のQ&Aペアのコレクション")
この構造に従って以下のようなデータを生成させます。
"qa_pairs": [
{
"question": "What is the general strategy employed for forecasting time series data?",
"answer": "The general strategy used for forecasting time series data involves a combined approach that leverages both standard forecasting techniques and a machine learning model to enhance accuracy and reliability.",
"reasoning": {
"importance": "Establishes the fundamental approach of the methodology",
"evidence_support": "Directly drawn from the description of the combined approach",
"objectivity": "Presents factual methodology without interpretation"
}
}
]
プロンプト（長いので中略）です。最後に構造そのものを例示しました。
query = (
{
"task": "Generate detailed, self-contained question and answer pairs with inline source evidence, based on the inputs",
"inputs": f"Title: {title}\nAbstract : {abstract}\n",
"instructions": (
"You are tasked with creating a set of detailed, self-contained question and answer pairs based on the given inputs. "
"Each Q&A pair should be understandable independently, without requiring external context. "
"Follow these systematic steps to generate high-quality Q&A pairs:\n\n"
""（中略）
"example_output": {
"qa_pairs": [
{
"question": "What is the general strategy employed for forecasting time series data?",
"answer": "The general strategy used for forecasting time series data involves a combined approach that leverages both standard forecasting techniques and a machine learning model to enhance accuracy and reliability.",
"reasoning": {
"importance": "Establishes the fundamental approach of the methodology",
"evidence_support": "Directly drawn from the description of the combined approach",
"objectivity": "Presents factual methodology without interpretation"
}
}
]
}
}
)
生成した回答例
UIP2P: Unsupervised Instruction-based Image Editing via Cycle Edit Consistency のアブストラクトから生成しました。
{'QA_pairs': [{'question': 'What is the main challenge that the proposed method addresses in instruction-based image editing?',
'answer': 'The main challenge addressed by the proposed method is the need for ground-truth edited images during training, which introduces biases and limits generalization ability. Existing supervised methods rely on datasets containing triplets of input image, edited image, and edit instruction, often generated from existing editing methods or human-annotations.',
'reasoning': {'importance': 'Identifies a key issue the method aims to solve, providing context for its significance.',
'evidence_support': 'Directly extracted from the abstract mentioning the need for ground-truth edited images and their limitations.',
'objectivity': 'Facts are presented without additional interpretation.'}},
{'question': 'What mechanism does the proposed method introduce to address these challenges?',
'answer': 'The proposed method introduces a novel editing mechanism called Cycle Edit Consistency (CEC), which applies forward and backward edits in one training step and enforces consistency in both image and attention spaces.',
'reasoning': {'importance': 'Highlights the core innovation of the study, crucial for understanding its unique contribution.',
'evidence_support': 'Directly extracted from the abstract mentioning CEC and its role in enforcing consistency.',
'objectivity': 'Facts are presented without additional interpretation.'}},
{'question': 'How does the proposed method differ from existing supervised methods?',
'answer': 'The proposed method differs from existing supervised methods by eliminating the need for ground-truth edited images during training, thus bypassing biases and limitations associated with human-annotations or pre-existing datasets of triplets.',
'reasoning': {'importance': 'Compares the new approach to traditional methods, providing clarity on its unique aspects.',
'evidence_support': 'Directly extracted from the abstract mentioning the need for ground-truth edited images and the limitations of supervised methods.',
'objectivity': 'Facts are presented without additional interpretation.'}},
{'question': 'What types of datasets can be used with this unsupervised method?',
'answer': 'This unsupervised method can use either real image-caption pairs or image-caption-edit triplets for training, unlocking the ability to train on a broader range of data.',
'reasoning': {'importance': 'Clarifies the flexibility and adaptability of the proposed approach in terms of dataset requirements.',
'evidence_support': 'Directly extracted from the abstract stating that the method can unlock training on datasets comprising real image-caption pairs or triplets.',
'objectivity': 'Facts are presented without additional interpretation.'}},
{'question': 'What empirical results does the study report about the proposed method?',
'answer': 'The study empirically demonstrates that their unsupervised technique performs better across a broader range of edits with high fidelity and precision compared to existing supervised methods.',
'reasoning': {'importance': "Provides concrete evidence of the method's effectiveness, essential for evaluating its practical utility.",
'evidence_support': 'Directly extracted from the abstract stating empirical results showing superior performance.',
'objectivity': 'Facts are presented without additional interpretation.'}},
{'question': 'What is the significance of this work in the field of instruction-based image editing?',
'answer': 'This work represents a significant advancement by addressing long-standing challenges with existing methods and proposing CEC, which allows for more scalable training on diverse datasets. This could unlock further innovations in unsupervised learning approaches for image editing.',
'reasoning': {'importance': "Summarizes the broader impact of the study's contributions to the field.",
'evidence_support': 'Directly extracted from the abstract emphasizing advancements and the potential for future work.',
'objectivity': 'Facts are presented without additional interpretation.'}}]}
いろいろと使えそう。
