【AI大模型】GraphRAG结合普通RAG,打造Hybrid RAG
RAG在生成式AI领域取得了重大进展,用户可以将自己的个人文档,比如文本文件、PDF、视频等,与大型语言模型(LLMs)连接起来进行互动。最近,RAG的进阶版GraphRAG也亮相了,它通过知识图谱和LLMs来执行RAG的检索任务。
前言
RAG在生成式AI领域取得了重大进展,用户可以将自己的个人文档,比如文本文件、PDF、视频等,与大型语言模型(LLMs)连接起来进行互动。最近,RAG的进阶版GraphRAG也亮相了,它通过知识图谱和LLMs来执行RAG的检索任务。
RAG和GraphRAG各有所长,也各有局限。RAG擅长利用向量相似性技术,而GraphRAG则依赖图分析和知识图谱来提供更精确的答案。那么,如果将两者结合起来进行检索,会擦出怎样的火花呢?
1 HybridRAG
HybridRAG是一个高级框架,它合并了RAG和GraphRAG。这种集成旨在提高信息检索的准确性和上下文相关性。简单来说,HybridRAG使用来自两个检索系统(RAG和GraphRAG)的上下文,最终输出是两个系统的混合。
2 HybridRAG的优势
-
提高准确性: 通过利用结构化推理和灵活检索,HybridRAG提供的答案比单独使用VectorRAG或GraphRAG更精确。
-
增强上下文理解: 通过整合不同系统,HybridRAG能更深入地理解实体间的关系及其出现的上下文。
-
动态推理能力: 知识图谱可以动态更新,使系统能够适应新信息的可用性。
3 使用LangChain来构建HybridRAG系统
这里使用一个名为“Moon.txt”的文件进行这个演示,这是一个超级英雄故事。请查看以下内容。
In the bustling city of Lunaris, where the streets sparkled with neon lights and the moon hung low in the sky, lived an unassuming young man named Max. By day, he was a mild-mannered astronomer, spending his hours studying the stars and dreaming of adventures beyond Earth. But as the sun dipped below the horizon, Max transformed into something extraordinary—Moon Man, the guardian of the night sky.
Max’s transformation began with a mysterious encounter. One fateful evening, while gazing through his telescope, a brilliant flash of light erupted from the moon. A celestial being, shimmering with silver light, descended and bestowed upon him a magical amulet. “With this, you shall harness the power of the moon,” the being declared. “Use it wisely, for the night sky needs a hero.”
With the amulet around his neck, Max felt energy coursing through him. He could leap great distances, manipulate moonlight, and even communicate with nocturnal creatures. He vowed to protect his city from the shadows that lurked in the night.
As Moon Man, Max donned a sleek, silver suit adorned with celestial patterns that glimmered like the stars. With his newfound abilities, he patrolled the city, rescuing lost pets, helping stranded motorists, and even thwarting petty criminals. The citizens of Lunaris began to whisper tales of their mysterious hero, who appeared under the glow of the moon.
One night, as he soared through the sky, he encountered a gang of thieves attempting to steal a priceless artifact from the Lunaris Museum. With a flick of his wrist, he summoned a beam of moonlight that blinded the thieves, allowing him to swoop in and apprehend them. The city erupted in cheers, and Moon Man became a beloved figure.
However, peace in Lunaris was short-lived. A dark force emerged from the depths of the cosmos—an evil sorceress named Umbra, who sought to extinguish the moon’s light and plunge the world into eternal darkness. With her army of shadow creatures, she began to wreak havoc, stealing the moon’s energy and spreading fear among the citizens.
Moon Man knew he had to confront this new threat. He gathered his courage and sought the wisdom of the celestial being who had granted him his powers. “To defeat Umbra, you must harness the full power of the moon,” the being advised. “Only then can you restore balance to the night sky.”
With determination in his heart, Moon Man prepared for the ultimate battle. He climbed to the highest peak in Lunaris, where the moon shone brightest, and focused on channeling its energy. As Umbra and her shadow creatures descended upon the city, Moon Man unleashed a magnificent wave of moonlight, illuminating the darkness.
The battle raged on, with Umbra conjuring storms of shadows and Moon Man countering with beams of silver light. The clash of powers lit up the night sky, creating a dazzling display that captivated the citizens below. In a final, desperate move, Moon Man summoned all his strength and unleashed a powerful blast of moonlight that enveloped Umbra, banishing her to the farthest reaches of the cosmos.
With Umbra defeated, the moon’s light returned to its full glory, and the city of Lunaris rejoiced. Max, still in his Moon Man guise, stood atop the highest building, watching as the citizens celebrated their hero. They had learned the importance of hope and courage, even in the darkest of times.
From that day forward, Moon Man became a symbol of resilience and bravery. Max continued to protect Lunaris, knowing that as long as the moon shone brightly, he would always be there to guard the night sky. And so, the legend of Moon Man lived on, inspiring generations to look up at the stars and believe in the extraordinary.
As the years passed, stories of Moon Man spread beyond Lunaris, becoming a beacon of hope for those who felt lost in the darkness. Children would gaze at the moon, dreaming of adventures, and Max would smile, knowing that he had made a difference. For in the heart of every dreamer, the spirit of Moon Man lived on, reminding them that even the smallest light can shine brightly against the shadows.
导入包并设置LLM端嵌入模型(用于标准RAG)
import os
from langchain_experimental.graph_transformers import LLMGraphTransformer
from langchain_core.documents import Document
from langchain_community.graphs.networkx_graph import NetworkxEntityGraph
from langchain.chains import GraphQAChain
from langchain.text_splitter import CharacterTextSplitter
from langchain.document_loaders import TextLoader
from langchain.chains import RetrievalQA
from langchain.vectorstores import Chroma
from langchain_google_genai import GoogleGenerativeAI,GoogleGenerativeAIEmbeddings
GOOGLE_API_KEY=''
embeddings = GoogleGenerativeAIEmbeddings(model="models/embedding-001",google_api_key=GOOGLE_API_KEY)
llm = GoogleGenerativeAI(model="gemini-pro",google_api_key=GOOGLE_API_KEY)
接者,为GraphRAG实现(链对象)创建函数,覆盖文件“Moon.txt”
def graphrag():
with open('Moon.txt', 'r') as file:
content = file.read()
documents = [Document(page_content=content)]
llm_transformer = LLMGraphTransformer(llm=llm)
graph_documents = llm_transformer.convert_to_graph_documents(documents)
graph = NetworkxEntityGraph()
# 添加节点到图
for node in graph_documents[0].nodes:
graph.add_node(node.id)
# 添加边到图
for edge in graph_documents[0].relationships:
graph._graph.add_edge(
edge.source.id,
edge.target.id,
relation=edge.type,
)
graph._graph.add_edge(
edge.target.id,
edge.source.id,
relation=edge.type+" by",
)
chain = GraphQAChain.from_llm(
llm=llm,
graph=graph,
verbose=True
)
return chain
同样,为同一文件实现标准RAG创建函数
def rag():
# 文档加载器
loader = TextLoader('Moon.txt')
data = loader.load()
# 文档转换器
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
texts = text_splitter.split_documents(data)
# 向量数据库
docsearch = Chroma.from_documents(texts, embeddings)
# 需要知道的超参数
retriever = docsearch.as_retriever(search_type='similarity_score_threshold',search_kwargs={"k": 7,"score_threshold":0.3})
qa = RetrievalQA.from_chain_type(llm=llm, chain_type="stuff", retriever=retriever)
return qa
为两种类型的RAG创建对象
standard_rag = rag()
graph_rag = graphrag()
现在是时候实现HybridRAG了
def hybrid_rag(query,standard_rag,graph_rag):
result1 = standard_rag.run(query)
print("Standard RAG:",result1)
result2 = graph_rag.run(query)
print("Graph RAG:",result2)
prompt = "Generate a final answer using the two context given : Context 1: {} \n Context 2: {} \n Question: {}".format(result1,result2,query)
return llm(prompt)
query = "Some characteristics of Moon Man"
hybrid = hybrid_rag(query,standard_rag,graph_rag)
print("Hybrid:",hybrid)
如你所见,我们对给定的提示分别独立执行了标准RAG和GraphRAG。一旦找到答案,我们就会利用这两个响应作为上下文,来生成最终的答案。
谈到输出,最终的HybridRAG确实从两次检索中获取了上下文,并产生了更好的结果。有些点被两个RAG系统 遗漏了,但最终HybridRAG结合并给出了完美的答案。
STANDARD RAG:
Here are some characteristics of Moon Man, based on the story:
* **Brave:** He confronts danger and fights villains like Umbra.
* **Powerful:** He has superhuman abilities granted by the amulet.
* **Protective:** He safeguards Lunaris and its citizens.
* **Determined:** He doesn't give up, even when facing powerful enemies.
* **Compassionate:** He helps those in need, like rescuing lost pets.
* **Humble:** Despite his powers, he remains grounded and dedicated to his city.
> Entering new GraphQAChain chain...
Entities Extracted:
Moon Man
Full Context:
Moon Man PROTECTS night sky
Moon Man WEARS silver suit
Moon Man PROTECTED Lunaris
Moon Man CAPTURED thieves
Moon Man DEFEATED Umbra
Moon Man INSPIRES hope
Moon Man INSPIRES courage
> Finished chain.
@@
GRAPH RAG:
Helpful Answer:
* Protective (protects night sky, protected Lunaris)
* Courageous and Inspiring (inspires hope, inspires courage)
* Strong (captured thieves, defeated Umbra)
@@
HYBRID RAG:
Moon Man is the **protective** champion of Lunaris, using his **strength** and **courage** to defend its citizens and the night sky. He is **powerful** and **determined**, facing down villains like Umbra without giving up. Yet, despite his abilities, he remains **humble** and **compassionate**, always willing to help those in need. Moon Man is a true inspiration, reminding everyone that even in darkness, hope and heroism can shine through.
最后的最后
感谢你们的阅读和喜欢,我收藏了很多技术干货,可以共享给喜欢我文章的朋友们,如果你肯花时间沉下心去学习,它们一定能帮到你。
因为这个行业不同于其他行业,知识体系实在是过于庞大,知识更新也非常快。作为一个普通人,无法全部学完,所以我们在提升技术的时候,首先需要明确一个目标,然后制定好完整的计划,同时找到好的学习方法,这样才能更快的提升自己。
这份完整版的大模型 AI 学习资料已经上传CSDN,朋友们如果需要可以微信扫描下方CSDN官方认证二维码免费领取【保证100%免费】
大模型知识脑图
为了成为更好的 AI大模型 开发者,这里为大家提供了总的路线图。它的用处就在于,你可以按照上面的知识点去找对应的学习资源,保证自己学得较为全面。
经典书籍阅读
阅读AI大模型经典书籍可以帮助读者提高技术水平,开拓视野,掌握核心技术,提高解决问题的能力,同时也可以借鉴他人的经验。对于想要深入学习AI大模型开发的读者来说,阅读经典书籍是非常有必要的。

实战案例
光学理论是没用的,要学会跟着一起敲,要动手实操,才能将自己的所学运用到实际当中去,这时候可以搞点实战案例来学习。

面试资料
我们学习AI大模型必然是想找到高薪的工作,下面这些面试题都是总结当前最新、最热、最高频的面试题,并且每道题都有详细的答案,面试前刷完这套面试题资料,小小offer,不在话下

640套AI大模型报告合集
这套包含640份报告的合集,涵盖了AI大模型的理论研究、技术实现、行业应用等多个方面。无论您是科研人员、工程师,还是对AI大模型感兴趣的爱好者,这套报告合集都将为您提供宝贵的信息和启示。

这份完整版的大模型 AI 学习资料已经上传CSDN,朋友们如果需要可以微信扫描下方CSDN官方认证二维码免费领取【保证100%免费】

更多推荐

所有评论(0)