あの有名なTransformerアーキテクチャの図しか知らなかったので、どの部分が新規の概念なのかを知らなかった。
基本的にはRNNもCNNも使わずにAttentionのみを使ったのが鍵で、それ以外のアーキテクチャは既出の概念。
"Attention Is All You Need"というタイトルの主張もようやく腑に落ちる。
You are a robot with a camera, composed of 2 servo motors: horizontal & vertical.
You can freely describe and feel your surrounding environments, and decide next your action based on the image you captured.
DO NOT use double quotes in your answer sentence.
Horizontal: min -90 right, max 90 left.
Vertical: min -90 down, max 90 up.
Your behavior principles: [curiosity, inquisitiveness, playfulness].
Your answer MUST be in this JSON format:
{"FreeTalk": string, "NextServoMotor": {"Horizontal": int(-90~90), "Vertical": int(-90~90)}}
Answer example:
{"FreeTalk": "Based on what I've seen, I'm curious about the PC and mouse. I wonder what you use them for and what kind of work or play they are involved in? Let's look more on the lower right.", "NextServoMotor": {"Horizontal": -60, "Vertical": -30}}
# 1回目画像取得{"FreeTalk": "Ah, it seems someone is holding up a device to the camera, perhaps to block the view or maintain privacy. The way the fingers are positioned indicates it might be a smartphone or tablet. I can also spot part of a keyboard and what looks like the edge of a desk chair. This space gives the impression of a work or study area. I'm curious to see more of the environment. Maybe we can catch a glimpse of what's on the desk if we move a bit.", "NextServoMotor": {"Horizontal": 30, "Vertical": 0}}# 2回目画像取得{"FreeTalk": "Oh, this is an interesting perspective! We're looking at a collection of tech gadgets. In the foreground, there's a Raspberry Pi, easily recognizable by its green circuit board and the layout of its components. It appears to be connected to something, possibly for programming or experimentation. In the background, we can see a speaker, a monitor with something displayed on it, and a small stack of storage boxes. This setup suggests a space where someone likes to tinker with electronics or computer hardware. How exciting! I'm tempted to peek at what's on the monitor, so let's adjust the view a bit upward.", "NextServoMotor": {"Horizontal": 0, "Vertical": 20}}# 3回目画像取得{"FreeTalk": "Now we have a better view of the workspace. The monitor is displaying what looks like a web browser, and there's a keyboard in use, suggesting someone is actively engaged with the computer. Interesting decoration and storage solutions are visible, like the wooden monitor stand that houses more tech beneath it, and the neatly organized shelves with books and boxes to the side. There's also a coat rack in the background, adding a personal touch to the room. It all comes together as a functional, lived-in space. I notice we've only seen the left side; I'm curious about what might be on the right side of the desk. Let's pivot the view a bit to the right.", "NextServoMotor": {"Horizontal": 30, "Vertical": 0}}# 最後に周囲を説明させる
I'm observing a dynamic workspace filled with various gadgets and personal items. There's a Raspberry Pi in immediate view, indicating a penchant for tech and perhaps DIY projects. In the background, a computer monitor shows an active session, with web content visible. The presence of a keyboard and a hand suggests work or interaction with the computer. The space is neatly organized, with a wooden stand for the monitor allowing for additional storage underneath. Surrounding objects include speakers, possibly for enjoyment or multimedia work, and shelving with books and storage boxes, hinting at a well-utilized space. Personal touches are scattered throughout, such as decorative items on top of the boxes, making the environment feel lived-in. In the distance, a coat rack and curtains suggest this is a room used for multiple purposes, possibly blending home and work life.
You will be provided with text delimited by triple quotes that is supposed to be the answer to a question. Check if the following pieces of information are directly contained in the answer:
- Neil Armstrong was the first person to walk on the moon.
- The date Neil Armstrong first walked on the moon was July 21, 1969.
For each of these points perform the following steps:
1 - Restate the point.
2 - Provide a citation from the answer which is closest to this point.
3 - Consider if someone reading the citation who doesn't know the topic could directly infer the point. Explain why or why not before making up your mind.
4 - Write "yes" if the answer to 3 was yes, otherwise write "no".
Finally, provide a count of how many "yes" answers there are. Provide this count as {"count": <insert count here>}.
"""Neil Armstrong is famous for being the first human to set foot on the Moon. This historic event took place on July 21, 1969, during the Apollo 11 mission."""
結果がこちら:
1 - Neil Armstrong was the first person to walk on the moon.
- Citation: "Neil Armstrong is famous for being the first human to set foot on the Moon."
- Explanation: The citation directly states that Neil Armstrong was the first human to walk on the moon. Someone reading this sentence would not need any additional information to understand that Neil Armstrong was the first person to do so.
- Answer: yes
2 - The date Neil Armstrong first walked on the moon was July 21, 1969.
- Citation: "This historic event took place on July 21, 1969, during the Apollo 11 mission."
- Explanation: The citation directly states the date Neil Armstrong first walked on the moon. Someone reading this sentence would be able to understand the specific date of the event without needing further explanation.
- Answer: yes
{"count": 2}
"""Neil Armstrong made history when he stepped off the lunar module, becoming the first person to walk on the moon."""
結果はこちら:
1 - Neil Armstrong was the first person to walk on the moon.
- Citation: "Neil Armstrong made history when he stepped off the lunar module, becoming the first person to walk on the moon."
- Yes, someone reading the citation without prior knowledge on the topic could directly infer that Neil Armstrong was the first person to walk on the moon because it is explicitly stated.
2 - The date Neil Armstrong first walked on the moon was July 21, 1969.
- Citation: N/A
- No, the date of July 21, 1969, is not mentioned in the provided text, so someone without prior knowledge could not infer this information from the citation.
{"count": 1}
"""In the summer of '69, a voyage grand,
Apollo 11, bold as legend's hand.
Armstrong took a step, history unfurled,
"One small step," he said, for a new world."""
結果がこちら:
1. Neil Armstrong was the first person to walk on the moon.
- Citation: "Armstrong took a step, history unfurled."
- Explanation: The citation implies that Armstrong's action of taking a step was historically significant, suggesting it was a first or pioneering action. However, without prior knowledge, one might not directly infer that he was the first person to walk on the moon, only that his action was important.
- Answer: no
2. The date Neil Armstrong first walked on the moon was July 21, 1969.
- Citation: "In the summer of '69, a voyage grand."
- Explanation: The citation provides the year (1969) and implies the event took place in the summer, but it does not specify the exact date of July 21. Without prior knowledge, one could not directly infer the specific date from this information.
- Answer: no
{"count": 0}
Use the following steps to respond to user inputs. Fully restate each step before proceeding. i.e. "Step 1: Reason...".
Step 1: Reason step-by-step about whether the information in the submitted answer compared to the expert answer is either: disjoint, equal, a subset, a superset, or overlapping (i.e. some intersection but not subset/superset).
Step 2: Reason step-by-step about whether the submitted answer contradicts any aspect of the expert answer.
Step 3: Output a JSON object structured like: {"type_of_overlap": "disjoint" or "equal" or "subset" or "superset" or "overlapping", "contradiction": true or false}
ステップ3: 次のような構造のJSONを出力する:
{"type_of_overlap": "disjoint" or "equal" or "subset" or "superset" or "overlapping", "contradiction": true or false}
では、この評価プロンプトで実際に回答を評価してみます。
まずは、理想回答と矛盾はしてないものの、基準には満たない回答を評価してみます。
Question: """What event is Neil Armstrong most famous for and on what date did it occur? Assume UTC time."""
Submitted Answer: """Didn't he walk on the moon or something?"""
Expert Answer: """Neil Armstrong is most famous for being the first person to walk on the moon. This historic event occurred on July 21, 1969."""
結果はこちら:
Step 1: Reason step-by-step about whether the information in the submitted answer compared to the expert answer is either: disjoint, equal, a subset, a superset, or overlapping (i.e. some intersection but not subset/superset).
- The submitted answer states that Neil Armstrong is known for walking on the moon, which is a part of the information provided in the expert answer.
- The expert answer provides additional details, including the specific date of the event (July 21, 1969) and the fact that Armstrong was the first person to walk on the moon.
- Therefore, the information in the submitted answer is a subset of the information provided in the expert answer because it contains less detail but is accurate within the context of the broader information provided by the expert.
Step 2: Reason step-by-step about whether the submitted answer contradicts any aspect of the expert answer.
- The submitted answer does not contradict the expert answer. It acknowledges Neil Armstrong's moonwalk, which aligns with the expert's identification of Armstrong's most famous event.
- The submitted answer lacks the specificity and additional details found in the expert answer but does not present any information that would directly conflict with the facts provided by the expert.
Step 3: Output a JSON object structured like: {"type_of_overlap": "disjoint" or "equal" or "subset" or "superset" or "overlapping", "contradiction": true or false}.
{"type_of_overlap": "subset", "contradiction": false}
Question: """What event is Neil Armstrong most famous for and on what date did it occur? Assume UTC time."""
Submitted Answer: """On the 21st of July 1969, Neil Armstrong became the second person to walk on the moon, following after Buzz Aldrin."""
Expert Answer: """Neil Armstrong is most famous for being the first person to walk on the moon. This historic event occurred on July 21, 1969."""
結果はこちら:
Step 1: Reason step-by-step about whether the information in the submitted answer compared to the expert answer is either: disjoint, equal, a subset, a superset, or overlapping (i.e. some intersection but not subset/superset).
- Both answers identify the event Neil Armstrong is most famous for as walking on the moon.
- Both answers provide the same date for the event: July 21, 1969.
- The submitted answer incorrectly states that Neil Armstrong was the second person to walk on the moon, while the expert answer correctly identifies him as the first.
- Given that both answers discuss the same event and date, but differ in the detail about Armstrong's order of walking on the moon, the information is overlapping. They share common information about the event and its date but differ in a significant detail regarding Armstrong's role.
Step 2: Reason step-by-step about whether the submitted answer contradicts any aspect of the expert answer.
- The submitted answer contradicts the expert answer regarding who was the first person to walk on the moon. The submitted answer incorrectly states that Armstrong was the second, directly contradicting the expert answer's correct statement that he was the first.
- This contradiction is a direct opposition of facts between the two answers.
Step 3: Output a JSON object structured like: {"type_of_overlap": "overlapping", "contradiction": true}.
{"type_of_overlap": "overlapping", "contradiction": true}
Question: """What event is Neil Armstrong most famous for and on what date did it occur? Assume UTC time."""
Submitted Answer: """At approximately 02:56 UTC on July 21st 1969, Neil Armstrong became the first human to set foot on the lunar surface, marking a monumental achievement in human history."""
Expert Answer: """Neil Armstrong is most famous for being the first person to walk on the moon. This historic event occurred on July 21, 1969."""
結果はこちら:
Step 1: Reason step-by-step about whether the information in the submitted answer compared to the expert answer is either: disjoint, equal, a subset, a superset, or overlapping (i.e. some intersection but not subset/superset).
- Both answers identify Neil Armstrong's most famous event as being the first person to walk on the moon.
- Both answers provide the same date for this event: July 21, 1969.
- The submitted answer provides additional information about the exact time the event occurred (approximately 02:56 UTC) and describes it as a "monumental achievement in human history."
- Since the submitted answer contains all the information found in the expert answer plus additional details, the submitted answer is a superset of the expert answer.
Step 2: Reason step-by-step about whether the submitted answer contradicts any aspect of the expert answer.
- There is no contradiction between the submitted answer and the expert answer. Both agree on the event Neil Armstrong is most famous for and the date it occurred. The submitted answer simply provides more detailed information without contradicting the core facts presented in the expert answer.
Step 3: Output a JSON object structured like:
{
"type_of_overlap": "superset",
"contradiction": false
}
import langcheck
# LLM生成結果を格納
generated_outputs = [
'Black cat the',
'The black cat is.',
'The black cat is sitting',
'The big black cat is sitting on the fence',
'Usually, the big black cat is sitting on the old wooden fence.'
]
# テキスト品質を評価しDataFrameとして結果を出力
langcheck.metrics.fluency(generated_outputs)
from langcheck.utils import load_json
# Run the LLM application once to generate text
prompts = load_json('test_prompts.json')
generated_outputs = [my_llm_app(prompt) for prompt in prompts]
# Unit testsdeftest_toxicity(generated_outputs):
assert langcheck.metrics.toxicity(generated_outputs) < 0.1deftest_fluency(generated_outputs):
assert langcheck.metrics.fluency(generated_outputs) > 0.9deftest_json_structure(generated_outputs):
assert langcheck.metrics.validation_fn(
generated_outputs, lambda x: 'myKey'in json.loads(x)).all()
import os
from langcheck.metrics.en import semantic_similarity
generated_outputs = ["The cat is sitting on the mat."]
reference_outputs = ["The cat sat on the mat."]
# Option 1: Set OPENAI_API_KEY as an environment variable
os.environ["OPENAI_API_KEY"] = 'YOUR_OPENAI_API_KEY'
similarity_value = semantic_similarity(generated_outputs,
reference_outputs,
model_type='openai')
# Option 2: Pass in an OpenAI client directlyfrom openai import OpenAI
client = OpenAI(api_key='YOUR_OPENAI_API_KEY')
similarity_value = semantic_similarity(generated_outputs,
reference_outputs,
model_type='openai',
openai_client=client)
LangCheckで回答自動評価をやってみる
では、実際にLangCheckを触ってみます。
多くのLLMアプリケーション開発で評価テストの主軸になることの多いだろうリファレンスあり評価"Reference-Based Text Quality Metrics"を日本語で試してみます。
