Text generation#

Load a model template#

Here’s an example of registering a model template to a Langdash instance. The Langdash.model_from_type method is used to load a model template with specific model type and model name. For this example, we will load the Raven model for inference.

ld = Langdash()
ld.register_model(
  name="raven",
  model=Langdash.model_from_type(type="rwkv_cpp", "/path/to/ggml-raven.bin"),
)

The register_model function takes two arguments:

name: the name of the language model to be registered
model: the model template class. In this case, the rwkv.cpp backend is used, with a path to the GGML file passed as a parameter.

When the register_model function is called, the Langdash API adds a template to the language model to its list of available language models, so that it can be used in subsequent API calls.

Here is a list of every model type currently supported by langdash:

Model	Description	Usage
`rwkv_cpp`	rwkv.cpp backend	`Langdash.model_from_type(type="rwkv_cpp", model_path, tokenizer_path=None)` model_path is the path to the binary GGML weights tokenizer_path is the path to the tokenizer (assumed to be the `20B_tokenizer.json` file in the same directory of the weights)
`llama_cpp`	llama.cpp backend through the llama-cpp-python library	`Langdash.model_from_type(type="llama_cpp", model_path, model_kwargs)` model_path is the path to the binary GGML weights model_kwargs**: keyword arguments which will be passed to the library’s Llama constructor.
`transformers`	transformers backend through the transformers library	`Langdash.model_from_type(type="transformers", model, tokenizer=None, model_kwargs)` model is the name of the transformers model, or the model instance tokenizer** is the name of the tokenizer (assumed to be the same as model_name), or the tokenizer instance model_kwargs: keyword arguments which will be passed to the model constructor.
`ctransformers`	GGML backends through the ctransformers library (Note: it is not recommended to run llama-based models through this backend)	`Langdash.model_from_type(type="ctransformers", model, tokenizer=None, model_kwargs)` model is the path to the transformers model tokenizer is `None` for the using the tokenizer embedding within the ggml weights, or a huggingface tokenizer instance model_kwargs**: keyword arguments which will be passed to the library’s constructor.

Alternatively, you can specify model config with the LLMConfig class, and pass it into the method as a parameter named config.

Create a prompt template, and specialize it for a model#

Chains form the core functionality of Langdash. They can be used as a prompt template.

instruct_chain = ld.chain(
  args={"instruction": str},
  returns={"response": str},
  nodes=[
    ld.format_args("""\
Below is an instruction that describes a task. Write a response that appropriately completes the request.

# Instruction:
{instruction}

# Response:
"""),
    ld.returns("response", "", inference_args=InferArgs(
      typical_mass=0.8,
      # more inference arguments
    )),
  ],
)

In the code above, the chain method takes in the arguments:

args: a dictionary that specifies the argument names and types.
returns: a dictionary that specifies the return (output) values
nodes: a list of prompts to be fed into the model, as well as return nodes for generation.

The first node in the list formats the instruction using the input value, replacing “{instruction}” with the instruction parameter. The second node returns the formatted response string.

An InferArgs object can be passed into the return node, which contains additional inference arguments that can be specified to customize the response. Here, typical_mass is used to infer with typical sampling (mass=0.8).

raven_instruct_chain = instruct_chain.cached("raven")

The code above creates a cached version of the instruct_chain chain using the raven model. This means it will cache the state of the model for similar parameters instead of recreating it every time the chain is called.

Perform inference#

This code below uses the cached instruct chain defined earlier to ask the model to explain what a raven is. The call method takes a dictionary called args containing the instruction.

instruction = "Explain what a raven is."
raven_instruct_chain.call(args={"instruction": instruction})

Example output (on ggml-RWKV-4-Raven-1B5-v11-Eng99%-Other1%-20230425-ctx4096-f16.bin model):

LDResult(returns={'response': 'A raven is a type of wild bird known for its piercing cry of \"Rra-ra-ra-ra.\" These birds are also known for their incredible agility and speed, often catching fish from waterways with ease. Rra-ra-ra-ra can be seen in the natural world and is commonly seen in the parklands and forests of Canada, the United States, Australia, and New Zealand.'}, prompt_tokens=32, completion_tokens=87)