Scientific Journals

Install the Indexify Extractor SDK, Langchain Retriever and the Indexify Client¶

In [ ]:

%%capture
!pip install -U indexify-extractor-sdk indexify-langchain indexify

%%capture !pip install -U indexify-extractor-sdk indexify-langchain indexify

Start the Indexify Server¶

In [ ]:

!./indexify server -d

!./indexify server -d

Download an Embedding Extractor¶

On another terminal we'll download and start the embedding extractor which we will use to index text from the pdf document. We will also list all available extractors to showcase the full capabilities of indexify.

In [ ]:

!indexify-extractor download hub://embedding/minilm-l6
!indexify-extractor join-server
!indexify-extractor list

!indexify-extractor download hub://embedding/minilm-l6 !indexify-extractor join-server !indexify-extractor list

Download an Chunking Extractor¶

On another terminal we'll download and start the chunking extractor that will create chunks from the text and embeddings.

In [ ]:

!indexify-extractor download hub://text/chunking
!indexify-extractor join-server

!indexify-extractor download hub://text/chunking !indexify-extractor join-server

Download the PDF Extractor¶

On another terminal we'll install the necessary dependencies and start the PDF extractor which we will use to get text, bytes or json out of PDF documents.

Install Poppler on your machine

In [ ]:

!sudo apt-get install -y poppler-utils

!sudo apt-get install -y poppler-utils

Download and start the PDF extractor

In [ ]:

!indexify-extractor download hub://pdf/pdf-extractor
!indexify-extractor join-server

!indexify-extractor download hub://pdf/pdf-extractor !indexify-extractor join-server

Create Extraction Policies¶

Instantiate the Indexify Client

In [ ]:

from indexify import IndexifyClient, ExtractionGraph
client = IndexifyClient()

from indexify import IndexifyClient, ExtractionGraph client = IndexifyClient()

Create an Extraction Graph with policies to get texts and contents out of the PDF and to create chunks from the text and embeddings

In [ ]:

extraction_graph_spec = """
name: 'sciknowledgebase'
extraction_policies:
  - extractor: 'tensorlake/pdf-extractor'
    name: 'pdfextractor'
  - extractor: 'tensorlake/chunk-extractor'
    name: 'chunks'
    content_source: 'pdfextractor'
    input_params:
      chunk_size: 512
      overlap: 150
  - extractor: 'tensorlake/bge-base-en'
    name: 'bgeembedding'
    content_source: 'chunks'
"""
extraction_graph = ExtractionGraph.from_yaml(extraction_graph_spec)
client.create_extraction_graph(extraction_graph)

extraction_graph_spec = """ name: 'sciknowledgebase' extraction_policies: - extractor: 'tensorlake/pdf-extractor' name: 'pdfextractor' - extractor: 'tensorlake/chunk-extractor' name: 'chunks' content_source: 'pdfextractor' input_params: chunk_size: 512 overlap: 150 - extractor: 'tensorlake/bge-base-en' name: 'bgeembedding' content_source: 'chunks' """ extraction_graph = ExtractionGraph.from_yaml(extraction_graph_spec) client.create_extraction_graph(extraction_graph)

Upload a PDF File¶

In [ ]:

import requests
req = requests.get("https://arxiv.org/pdf/2310.16944.pdf")

with open('2310.16944.pdf','wb') as f:
    f.write(req.content)

import requests req = requests.get("https://arxiv.org/pdf/2310.16944.pdf") with open('2310.16944.pdf','wb') as f: f.write(req.content)

In [ ]:

client.upload_file("sciknowledgebase", path="2310.16944.pdf")

client.upload_file("sciknowledgebase", path="2310.16944.pdf")

What is happening behind the scenes¶

Indexify is designed to seamlessly respond to ingestion events by assessing all existing policies and triggering the necessary extractors for extraction. Once the PDF extractor completes the process of extracting texts, bytes, and JSONs from the document, it automatically initiates the embedding extractor to chunk the content, extract embeddings, and populate an index.

With Indexify, you have the ability to upload hundreds of PDF files simultaneously, and the platform will efficiently handle the extraction and indexing of the contents without requiring manual intervention. To expedite the extraction process, you can deploy multiple instances of the extractors, and Indexify's built-in scheduler will transparently distribute the workload among them, ensuring optimal performance and efficiency.

Perform RAG¶

Initialize the Langchain Retriever.

In [ ]:

from indexify_langchain import IndexifyRetriever
params = {"name": "sciknowledgebase.bgeembedding.embedding", "top_k": 15}
retriever = IndexifyRetriever(client=client, params=params)

from indexify_langchain import IndexifyRetriever params = {"name": "sciknowledgebase.bgeembedding.embedding", "top_k": 15} retriever = IndexifyRetriever(client=client, params=params)

Now create a chain to prompt OpenAI with data retrieved from Indexify to create a simple Q&A bot

In [ ]:

from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_openai import ChatOpenAI

from langchain_core.output_parsers import StrOutputParser from langchain_core.prompts import ChatPromptTemplate from langchain_core.runnables import RunnablePassthrough from langchain_openai import ChatOpenAI

In [ ]:

template = """Answer the question based only on the following context:
{context}

Question: {question}
"""
prompt = ChatPromptTemplate.from_template(template)

model = ChatOpenAI()

chain = (
    {"context": retriever, "question": RunnablePassthrough()}
    | prompt
    | model
    | StrOutputParser()
)

template = """Answer the question based only on the following context: {context} Question: {question} """ prompt = ChatPromptTemplate.from_template(template) model = ChatOpenAI() chain = ( {"context": retriever, "question": RunnablePassthrough()} | prompt | model | StrOutputParser() )

Now ask any question related to the ingested PDF document

In [ ]:

chain.invoke("What are the results on MT-Bench?")
# Results on MT-Bench show that ZEPHYR-7B surpasses LLAMA 2-CHAT-70B, the best open-access RLHF-based model.

chain.invoke("What are the results on MT-Bench?") # Results on MT-Bench show that ZEPHYR-7B surpasses LLAMA 2-CHAT-70B, the best open-access RLHF-based model.

In [ ]:

chain.invoke("what prompts are they using? Show all the prompts ")

chain.invoke("what prompts are they using? Show all the prompts ")

In [ ]: