Document searching and question answering

Using what we’ve learned so far, we can write a simple program that answers questions by looking up relevant documents and feeding it to a language model. In this example, paragraphs related to the Eiffel Tower (courtesy of Wikipedia) will be stored, and be used as example documents for our program.

Imports

Let’s start by importing the necessary components for our program.

from langdash import Langdash
from langdash.search.embedding_search import EmbeddingSearch
from langdash.infer import InferArgs
from langdash.response import RespInfer

Registering our models

For this program, we will use two separate models: one for generating embeddings, and one for generating text and answering our questions. We will use a suitable llama model for generation, and the multi-qa-MiniLM-L6-dot-v1 for embeddings.

ld.register_model(
  name="generation_model",
  model=Langdash.model_from_type(
    "llama_cpp", "/path/to/model.bin", n_ctx=4096
  ),
)

ld.register_model(
  name="embedding_model",
  model=Langdash.model_from_type(
    "sentence_transformers", "multi-qa-MiniLM-L6-dot-v1"
  )
)

Populating the database

First, the database needs to be created and populated. We create a new EmbeddingSearch database, and insert 5 different paragraphs about the Eiffel Tower.

db = EmbeddingSearch(ld.session_for_model("embedding_model"))

db.add(
  [
    "The Eiffel Tower is a wrought-iron lattice tower on the Champ de Mars in Paris, France. It is named after the engineer Gustave Eiffel, whose company designed and built the tower.",
    "The design of the Eiffel Tower is attributed to Maurice Koechlin and Émile Nouguier, two senior engineers working for the Compagnie des Établissements Eiffel. It was envisioned after discussion about a suitable centerpiece for the proposed 1889 Exposition Universelle, a world's fair to celebrate the centennial of the French Revolution. Eiffel openly acknowledged that inspiration for a tower came from the Latting Observatory built in New York City in 1853.",
    "The proposed tower had been a subject of controversy, drawing criticism from those who did not believe it was feasible and those who objected on artistic grounds. Prior to the Eiffel Tower's construction, no structure had ever been constructed to a height of 300 m, or even 200 m for that matter, and many people believed it was impossible.",
    "The Eiffel Tower was the world's tallest structure when completed in 1889, a distinction it retained until 1929 when the Chrysler Building in New York City was topped out. The tower also lost its standing as the world's tallest tower to the Tokyo Tower in 1958 but retains its status as the tallest freestanding (non-guyed) structure in France.",
    "More than 300 million people have visited the Eiffel tower since it was completed in 1889. In 2015, there were 6.91 million visitors. The tower is the most-visited paid monument in the world. An average of 25,000 people ascend the tower every day which can result in long queues.",
    "As one of the most famous landmarks in the world, the Eiffel Tower has been the inspiration for the creation of many replicas and similar towers. An early example is Blackpool Tower in England. The mayor of Blackpool, Sir John Bickerstaffe, was so impressed on seeing the Eiffel Tower at the 1889 exposition that he commissioned a similar tower to be built in his town. It opened in 1894 and is 158.1 m (519 ft) tall. Tokyo Tower in Japan, built as a communications tower in 1958, was also inspired by the Eiffel Tower.",
  ]
)

Creating our Q&A chain

Next, the Q&A chain. Just like in the text generation example, we write a instruction prompt and ask the model to generate our response. We will provide the question, as well as the relevant passage to the language model.

Since our chain is likely to be called again and again across documents, it is useful to have it cached. We can call the cached method to create a cached chain and specialize it on the generation_model model.

qa_chain = ld.chain(
  args={
    "document": str,
    "question": str
  },
  returns={
    "answer": str
  },
  nodes=[
    ld.format_args(
      """\
### Instruction:

You are given a passage on the Eiffel Tower:

{document}

Based on the passage above, answer the following question as short and precise as possible:

{question}

### Response:
"""
    ),
    ld.returns(
      "answer",
      "",
      infer_args=InferArgs(
        top_k=40,
        top_p=0.95,
        temperature=0.7,
      )
    ),
  ]
).cached("generation_model")
qa_chain.arguments_to_cache = {"document"}

Performing inference

Finally, we can write the function that performs inference. We get the first closest document in our embedding database, and feed it along with the question to the generation model.

def do_inference(question):
  try:
    _, document, _ = next(db.search(question))
    print("Document for inference:")
    print(document)
    print("---------")
    for resp in qa_chain.stream(
      args={
        "document": document,
        "question": question
      }
    ):
      if isinstance(resp, RespInfer):
        print(resp.tokstr, end="", flush=True)
    print("\n")
  except KeyboardInterrupt:
    print("(cut)")