Quick start#

Registering models#

Interaction with the library is done through a Langdash instance. Before doing anything else, let’s initialize an instance and register a pretrained language model. The register_model() method is used to define the model with its associated ID and model parameters, collectively called a model template. When passing in the model template, the Langdash.model_from_type() static method can be used to get the template from the name of the backend.

ld = Langdash()
ld.register_model(
  name="llama_cpp",
  model=Langdash.model_from_type(type="llama_cpp", "/path/to/ggml.bin", n_ctx=4096),
)

Langdash aims to support a variety of backends, but the primary target is lower-powered systems without a good GPU. At the time of writing, the currently supported backends include:

  • Generation backends: rwkv_cpp, llama_cpp, ctransformers (supporting the Dolly, GPT-Neo-X, GPT-2, GPT-J, MPT and Starcoder models), transformers

  • Embedding backends: sentence_transformers

See the text generation guide for specifics on how to use register_model.

Text generation#

Chains: the building blocks of Langdash#

Chains in Langdash provide a powerful way to interact with language models. A chain is simply a collection of nodes interpreted sequentially, which either inject tokens to the model or have it generate something; think of chains as functions.

Chains consist of arguments and return variables, as well as their corresponding types. Arguments will not be type checked, however return values will be casted from string to the specified type.

You can use chains to inject prompts to the model, and then have the model generate according to a template. Here is an example chain that prompts a language model to follow an instruction. It takes input in the form of an instruction text, and outputs a line containing the response.

instruct_chain = ld.chain(
  args={"instruction": str},
  returns={"response": str},
  nodes=[
    ld.format_args("# Instruction: {instruction}\n"),
    "# Response: "
    ld.returns(
      "response",
      end="\n",
      infer_args=InferArgs(),
    ),
  ],
)

The chain consists of three nodes:

  1. The first node formats the prompt using the format_args method.

  2. The second node injects the line # Response: to the language model.

  3. The third node simply returns the response text.

Optionally, inference parameters can be passed to the return node, see the full documentation on InferArgs for more.

You can call the chain like so, by providing the model name where it will be executed, and the arguments of the chain:

instruct_chain.call(ctx="model", args={"instruction": "Explain what a raven is."})

Which will return an LDReturns object, containing the return values specified, and the number of tokens injected or generated:

LDResult(returns={'response': 'A raven is a type of wild bird known for its piercing cry of \"Rra-ra-ra-ra.\" These birds are also known for their incredible agility and speed, often catching fish from waterways with ease. Rra-ra-ra-ra can be seen in the natural world and is commonly seen in the parklands and forests of Canada, the United States, Australia, and New Zealand.'}, prompt_tokens=32, completion_tokens=87)

Langdash provides a simple way to construct complex chains by concatenating nodes that can produce longer or more diverse sequences. Langdash supports the following nodes:

  • text(): Creates a raw text node, which will be inserted into the model.

  • returns(): Have the model generate a stream of text, until an end token (or an end string) is met. This stream of text will be returned as a variable.

  • arg(): Creates an argument node. When the node is executed, the argument will be converted into a string and passed into the model.

  • format_args(text): Formats arguments in a constant text string. The formatted string gets fed into the model.

  • choice(returns, choices): Forces the LM to select one stream of tokens in a choices array, and have it be returned in a variable.

  • repeat(): Repeats a subchain for a specific number of time, or until a probability condition is met.

Examples using chains are provided in the repository. A TV show generator can be constructed by concatenating nodes that generate random titles, description about the show and a basic plot summary. There is also a working implementation of using ReAct prompting to search for facts in a database, which is a chain-of-thought-like process, where the model repeatedly gives its thoughts and performs an appropriate action until it completes a task provided.

Sessions#

When calling a chain, you have the option to return the session after inference by setting the return_session parameter to true. Each session keeps track of the current state, which approximately records the tokens that have been submitted to, or generated by the language model.

returns, session = chain.call(ctx="model", args={}, return_session=True)

The session object provides a way to manage the context of a language model. Sessions can be used to continue executing prompts, submitting them to the language model.

You can execute a chain on an existing model by passing the session object into the ctx parameter. The state will not be reset, and new tokens will be injected.

chain.call(ctx=session)

Chain caching#

Chain caching is a feature of Langdash that allows you to save the state of your model between iterations of a chain of nodes. This is done by specializing a chain on a model through the cached() method:

cached_chain = chain.cached("model")

Cached chains can be called or streamed like regular chains. When a cached chain is called with the same input arguments, it will reuse the same state as the last time that chain object was passed in, thus avoiding unnecessary computations.

Cached chains can also intelligently figure out the shared state between a subset of input arguments. For example, let’s say we have a cached chain QA with the inputs: question, and context. The nodes of the chains are formatted like so:

Question: {question}
Context: {context}
Answer:

Say we call the chain QA with the following inputs:

QA.call(args={
  "question": "How many apples does John have?",
  "context": "John has 5 apples",
})

When called, the chain will store the state of the model after the question argument is injected. Since “Context: ” is a constant string, the state will be immediately before the context argument is fed into the model. This can be useful when the question stays constant, but the context changes. By using chain caching, we can improve the performance of our prompts by avoiding unnecessary computations.

Chain caching must be done explicitly, so do keep that it mind.

Text classification & search#

Classification through single-token generation#

Langdash also provides a simple way to classify text using generative models through the TokenQA class. It allows the user to prompt the language model with a prompt chain and have it perform classification by retrieving the probabilities of the next tokens. Here’s an example of how to use the TokenQA class to classify sentiment for a given text:

sentiment = TokenQA(
  prompt_chain=ld.chain(
    args={
      "query": str
    },
    nodes=[
      ld.format_args(
        """\
Classify the following text to positive or negative.
Text: {query}
Sentiment: """
      )
    ]
  ).cached("model"),
  classes={
    "positive": ["Positive", "positive"],
    "negative": ["Negative", "negative"],
  }
)

print(sentiment.query("I love you.")) # => { 'positive': 0.9123, 'negative': 0.0877 }

The TokenQA class uses a chain to interact with a language model. The prompt_chain parameter supplies the prompt that the model will receive. The classes dictionary maps specific text that would be labeled as positive or negative.

You can run the query using query() by passing a string as input. The result is a dictionary that maps the positive and negative classes to a numerical probability value.

Search through embeddings#

You can use Langdash to easily store and query the word embeddings of any supported model. In the following example, only one document is stored for simplicity, but the library can handle any arbitrary number of documents.

ld = Langdash()
ld.register_model(
  name="embedding_model",
  model=Langdash.model_from_type(type="sentence_transformers", model="all-MiniLM-L6-v2"),
)
db = EmbeddingSearch(ld.session_for_model("embedding_model"))

db.add(['example document'])
list(db.search(query, max_documents=1)) # => [(0, 'example document', 1.0)]

The EmbeddingSearch class is used to perform an embedding query on the embedding_model model. The session_for_model() method is used to create a new session associated with the model object registered earlier. The add() method is used to add a single document to the search index.

The query is a string which is passed to the search method search(). This method will return a list of tuples containing the indices of the relevant documents, its content and a float indicating the distance between the embeddings of the query and that document.

Search through generation using multi-choice Q&A#

Langdash supports document searching through providing a generative model a list of documents, and then having the model select which document fits the query the most. This can be done through the MultichoiceSearch helper class. The prompt for this style of search must be provided by the user.

As an example, say you want to provide following prompt will be provided to the generative model for search:

The following is a list of facts, which can be used to answer a query:

#1. (sentence 1)
#2. (sentence 2)
#3. ...

Which fact can be used to best answer the following query?

{query}

### Response:
The query can be answered by fact #{answer}

Using the MultichoiceSearch class, you can build a search chain like so:

def document_prompt(search, key, document):
  return f"{key+1}", f"#{key+1}. {document}\n"


db = MultichoiceSearch(
  prompt_chain=ld.chain(
    args={
      "prompts": str,
      "query": str
    },
    nodes=[
      ld.format_args(
        """\
The following is a list of facts, which can be used to answer a query:

{prompts}
Which fact can be used to best answer the following query?

{query}

### Response:
The query can be answered by fact #"""
      )
    ]
  ).cached("model"),
  document_prompt=document_prompt,
)

Here, the prompt_chain parameter is the chain that is executed every time the database is queried. The document_prompt parameter is a function that returns a tuple: containing (1) a token specifying the answer token to be selected at the end; and (2) a string which will be used to format the list of documents, a joined list of the second string will be provided to the prompt_chain as the prompts string argument.

Adding documents and querying can be done similarly to vector embedding search. This method works best for smaller documents, with the optimal being documents of around one to two sentences in length.