langdash package#

Subpackages#

Submodules#

langdash.core module#

Classes:

Langdash()

Core Langdash instance.
class langdash.core.Langdash[source]#

Bases: object

Core Langdash instance.

Methods:

arg(arg, **kwargs)

Creates a new argument node with the specified argument.

chain(**kwargs)

Chain a list of nodes together.

choice(returns, choices, **kwargs)

Creates a new choice node with the specified choices.

format_args(text, **kwargs)

Creates a format argument node.

model_from_type(type, *args, **kwargs)

Create an instance of a builtin model with specified type name.

register_model(name, model)

Register a new language model to the Langdash instance.

repeat(**kwargs)

Creates a new repetition node that repeats a subchain.

returns(returns, end, **kwargs)

Create a new return node for the specified return value.

session_for_model(model, **kwargs)

Create a new session for a given model.

text(text, **kwargs)

Creates a raw text node.
arg(arg, **kwargs)[source]#

Creates a new argument node with the specified argument.

Parameters:

  • arg (str) – The argument.

  • padleft (str) – The padding string to use for the left side of the argument.

  • padright (str) – The padding string to use for the right side of the argument.

Returns:

The argument node.

chain(**kwargs) → LDChain[source]#

Chain a list of nodes together.

Parameters:

  • nodes (List[Union["LDNode", str]]) – A list of nodes or constant text nodes (represented by strings) to chain together.

  • args (TypeDict) – A dictionary of argument types for the chain function.

  • returns (TypeDict) – A dictionary of return value types for the chain function.

Returns:

The chain of nodes.

choice(returns, choices, **kwargs)[source]#

Creates a new choice node with the specified choices.

Parameters:

  • returns (str) – The name of the return value.

  • choices (Union[str, List[str]]) – Either the list of choice strings or the name of the argument containing the list.

  • padleft (str) – Left padding for every choice string. Defaults to empty string.

  • padright (str) – Right padding for every choice string. Defaults to empty string.

Returns:

The choice node.

format_args(text, **kwargs)[source]#

Creates a format argument node.

Parameters:

text (str) – The format text.

Returns:

The format text node.

static model_from_type(type: str, *args, **kwargs) → LLM[source]#

Create an instance of a builtin model with specified type name.

Additional arguments will be passed to the model constructor. Alternatively, a keyword argument with the name config of type LLMConfig can be passed to specify config parameters.

Parameters:

type (str) – The type of the model.

Returns:

The model.

register_model(name: str, model: LLM)[source]#

Register a new language model to the Langdash instance.

Parameters:

  • name (str) – The name of the model.

  • model (LLM) – The LLM object.

repeat(**kwargs)[source]#

Creates a new repetition node that repeats a subchain.

Parameters:

  • subchain (LDChain) – The subchain to be repeated.

  • append_source (str) – The return variable of the subchain, extracted into the parent chain’s append_target list.

  • append_target (str) – The append target variable of the parent chain.

  • end (str) – Token used to mark the end of the repetition. This token will not be injected as a prompt after the loop.

  • max_len (int) – Maximum number of repetitions. -1 means the chain will repeat until the next most likely token is end.

  • end_threshold (float) – Minimum probability of end token for the repetition to end.

Returns:

The repeat node.

returns(returns, end, **kwargs)[source]#

Create a new return node for the specified return value.

Parameters:

  • returns (str) – The name of the return value.

  • end (InferEndSequence) – Where to stop the inference. See LLMGenerationSession#infer for more details.

  • padleft (str) –

    The left padding value for the return. If the generated string starts with padleft then it will be stripped.

    Defaults to empty string.

  • infer_args (Optional[InferArgs]) – Optional inference arguments for generation. Defaults to None.

  • end_is_token (bool) – If true, then the end string will be interpreted as a token. Defaults to False.

  • logit_preprocessors (Optional[List[LogitPreprocessor]]) – Optional list of logit preprocessor functions to be called before sampling. Defaults to None.

Returns:

The return node.

session_for_model(model: str, **kwargs) → LLMSession[source]#

Create a new session for a given model.

Parameters:

  • model (str) – The name of the model to be used.

  • default_infer_args (InferArgs) – Default arguments for the inference.

  • token_healing (bool) – Whether to enable token healing. Defaults to True.

  • event_handlers (Optional[LLMGenerationSessionEvents]) – Optional event handlers for prompt events. Defaults to None.

  • initial_prompt (Optional[Union[str, int, List[int]]]) – Optional initial prompt to be injected. Defaults to None.

Returns:

The session object.

text(text, **kwargs)[source]#

Creates a raw text node.

Parameters:

  • text (str) – The raw text.

  • add_special_tokens (bool) – Whether to treat text as containing special tokens or not. Defaults to False.

Returns:

The text node.

langdash.infer module#

Classes:

InferArgs([min_new_tokens, max_new_tokens, ...])

Data class for inference arguments.

SpecialToken

class langdash.infer.InferArgs(min_new_tokens: int = 0, max_new_tokens: int = 512, temperature: float = 1.0, top_k: int = 0, top_p: float = 0.0, typical_mass: float = 0.0, max_rep_ctx: int = 64, rep_penalty: float = 1.0, rng: torch.Generator | None = None)[source]#

Bases: object

Data class for inference arguments.

Attributes:

max_new_tokens

Maximum number of new tokens to generate

max_rep_ctx

Maximum number of tokens to look back for repetition penalty.

min_new_tokens

Minimum number of new tokens to generate.

rep_penalty

Repetition penalty, applied to logits for every repeated token before sampling.

rng

The random number generator (torch.Generator) to be used for token sampling.

temperature

Temperature

top_k

Top-K parameter.

top_p

Top-P parameter.

typical_mass

Typical mass parameter.
max_new_tokens: int = 512#

Maximum number of new tokens to generate

max_rep_ctx: int = 64#

Maximum number of tokens to look back for repetition penalty.

min_new_tokens: int = 0#

Minimum number of new tokens to generate.

If this is set, the end argument passed to the session’s infer method will be interpreted as a single token. Special tokens count as an end token.

rep_penalty: float = 1.0#

Repetition penalty, applied to logits for every repeated token before sampling.

rng: torch.Generator | None = None#

The random number generator (torch.Generator) to be used for token sampling. Defaults to None for the global RNG.

temperature: float = 1.0#

Temperature

top_k: int = 0#

Top-K parameter. If set, generation uses top-K sampling method. Works with top_p.

top_p: float = 0.0#

Top-P parameter. If set, generation uses top-P sampling method.

typical_mass: float = 0.0#

Typical mass parameter. If set, generation will use typical sampling.

class langdash.infer.SpecialToken[source]#

Bases: str

langdash.llm module#

Classes:

EmbeddingLLM()

A language model class for generating embeddings.

LLM()

A language model class for inference.

LLMCapability(value[, names, module, ...])

Flag representing the capabilities of a language model.
class langdash.llm.EmbeddingLLM[source]#

Bases: LLM[T_LLMEmbeddingSession]

A language model class for generating embeddings.

Attributes:

capability

Returns the capability of the language model.
property capability: LLMCapability#

Returns the capability of the language model.

class langdash.llm.LLM[source]#

Bases: Generic[T_LLMSession]

A language model class for inference.

Attributes:

capability

Returns the capability of the language model.

Methods:

session(*args, **kwargs)

Create a new session for the given model.
property capability: LLMCapability#

Returns the capability of the language model.

session(*args, **kwargs) → T_LLMSession[source]#

Create a new session for the given model.

Parameters:

  • default_infer_args (InferArgs) – Default arguments for the inference.

  • token_healing (bool) – Whether to enable token healing. Defaults to True.

  • event_handlers (Optional[LLMGenerationSessionEvents]) – Event handlers for prompt events. Defaults to None.

Returns:

A new session object.

class langdash.llm.LLMCapability(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: Flag

Flag representing the capabilities of a language model.

langdash.llm_config module#

Classes:

LLMConfig([model, batch_size, threads, ...])

Data class used to store unified LLM config.
class langdash.llm_config.LLMConfig(model: str | None = None, batch_size: int = -1, threads: int = -1, context_length: int = -1, gpu_layers: int = 0)[source]#

Bases: object

Data class used to store unified LLM config.

model#

The name, or the path of the model. Defaults to None.

Type:

Optional[str]

batch_size#

The number of tokens per batch size for inference. Defaults to -1 for auto.

Type:

int

threads#

The number of threads used for inference. Defaults to -1 for auto.

Type:

int

context_length#

The number of tokens used for context. Defaults to -1 for auto.

Type:

int

gpu_layers#

The number of layers to run on GPU. Defaults to 0.

Type:

int

langdash.llm_session module#

Classes:

LLMEmbeddingSession(llm, ld)

Session for a language model that outputs an embedding for raw text.

LLMGenerationSession(llm, ld[, ...])

Generation session for a language model.

LLMGenerationSessionForRawText(*args, **kwargs)

Generation session for a language model that processes raw text.

LLMSession(llm, ld)

Base class for a session for a language model.

LLMState()

A state class for a language model.
class langdash.llm_session.LLMEmbeddingSession(llm: T_LLM, ld: Langdash)[source]#

Bases: LLMSession, Generic[T_LLM]

Session for a language model that outputs an embedding for raw text.

Methods:

embed(documents)

Infer the embedding of a list of text.

Attributes:

embedding_size

Returns the embedding size of the model.
embed(documents: List[str]) → torch.Tensor[source]#

Infer the embedding of a list of text.

Parameters:

documents (List[str]) – The text to be embedded.

Returns:

The embedding vector of the list of text.

property embedding_size: int#

Returns the embedding size of the model.

class langdash.llm_session.LLMGenerationSession(llm: T_LLM, ld: Langdash, default_infer_args: InferArgs = InferArgs(min_new_tokens=0, max_new_tokens=512, temperature=1.0, top_k=0, top_p=0.0, typical_mass=0.0, max_rep_ctx=64, rep_penalty=1.0, rng=None), token_healing: bool = True, event_handlers: LLMGenerationSessionEvents | None = None, initial_prompt: str | int | List[int] | None = None)[source]#

Bases: LLMSession, Generic[T_LLM, T_LLMState]

Generation session for a language model.

Methods:

clone()

Clone the current session.

clone_state()

Clone the current state of the language model.

get_vocab()

Returns a mapping of token strings to their respective ids.

infer([args])

Infer the next tokens from the input sequence.

next_token_logits()

Returns the logits for next token.

next_token_probs()

Returns the probabilities for next token.

scratch_state()

Returns a context manager that manages setting and unloading temporary state.

set_state(state)

Set the state of the language model.

tokenize(text[, add_special_tokens])

Tokenize the given text into a list of tokens.

tokenize_special(text)

Tokenize the given text as a special token.

Attributes:

context_length

Returns the context length of the model, or zero for models that don't support one.

default_infer_args

Default inference arguments.

event_handlers

Optional list of event handlers.

token_healing

Whether token healing is enabled for this session.

tokens_used

Total number of tokens used.
clone() → LLMGenerationSession[source]#

Clone the current session.

Returns:

A new generation session with the same arguments.

clone_state() → T_LLMState | None[source]#

Clone the current state of the language model.

Returns:

The current state as an object.

property context_length: int#

Returns the context length of the model, or zero for models that don’t support one.

default_infer_args: InferArgs#

Default inference arguments.

event_handlers: LLMGenerationSessionEvents | None#

Optional list of event handlers.

get_vocab() → Dict[str | bytes, int][source]#

Returns a mapping of token strings to their respective ids.

infer(args: InferArgs | None = None, **kwargs) → Generator[Response, None, None][source]#

Infer the next tokens from the input sequence.

Parameters:

  • end (InferEndSequence) –

    The end of the output sequence.

    • If is string:

      • if end_is_token is set to True then inference will stop when the generated text ends with end.

      • otherwise, inference will stop when the token end is encountered.

    • If is int, then inference will stop at the token with ID of end.

    • If is List[str] and end_is_token set to False, then inference will stop when the generated text ends with any string in end.

    • If is List[str] (with end_is_token set to True) or List[int], then inference will stop when token matches any in the list.

    • If set to None, the output sequence will be generated until the maximum number of tokens is reached.

    Empty strings are interpreted as end of text token.

    Warning

    If end is interpreted as a token, then the token will not be injected into the session.

    Defaults to None.

  • args (Optional[InferArgs]) – Optional inference parameters. Defaults to None.

  • end_is_token (bool) – If true, then the end string will be interpreted as a token. Defaults to False.

  • logit_preprocessors (Optional[List[Type[LogitPreprocessor]]]) – Optional list of logit preprocessor classes to be called before sampling. Defaults to None.

Returns:

Generator yielding inference events.

next_token_logits() → torch.Tensor[source]#

Returns the logits for next token.

next_token_probs() → torch.Tensor[source]#

Returns the probabilities for next token.

scratch_state() → LLMGenerationSessionStateManager[source]#

Returns a context manager that manages setting and unloading temporary state.

set_state(state: T_LLMState | None)[source]#

Set the state of the language model.

Parameters:

state (Optional[T_LLMState]) – The state of the language model, or None to clear the state.

token_healing: bool#

Whether token healing is enabled for this session.

tokenize(text: str, add_special_tokens: bool = False) → List[int][source]#

Tokenize the given text into a list of tokens.

Parameters:

  • text (str) – The text to tokenize.

  • add_special_tokens (bool) – Whether to add special tokens to the output.

Returns:

The list of tokens.

tokenize_special(text: str) → int | None[source]#

Tokenize the given text as a special token.

Parameters:

text (str) – The text to tokenize.

Returns:

The token id, or None if the text doesn’t correspond to a special token.

tokens_used: int#

Total number of tokens used. Does not decrease unless session is reset.

class langdash.llm_session.LLMGenerationSessionForRawText(*args, **kwargs)[source]#

Bases: LLMGenerationSession, Generic[T_LLM, T_LLMState]

Generation session for a language model that processes raw text.

Methods:

flush_token()

Flushes the previous token into the language model if healing is enabled.

next_token_probs(*args, **kwargs)

Returns the probabilities for next token.
flush_token()[source]#

Flushes the previous token into the language model if healing is enabled.

Warning

Unexpected behavior might occur if the previous token is a “boundary” token, like the space ‘ ‘ token.

next_token_probs(*args, **kwargs) → torch.Tensor[source]#

Returns the probabilities for next token.

class langdash.llm_session.LLMSession(llm: T_LLM, ld: Langdash)[source]#

Bases: Generic[T_LLM]

Base class for a session for a language model.

Methods:

clone()

Clone the current session.
clone() → LLMSession[source]#

Clone the current session.

class langdash.llm_session.LLMState[source]#

Bases: object

A state class for a language model.

langdash.logit_preprocessors module#

Classes:

LogitPreprocessor(session)

Base class for a logit preprocessor.
class langdash.logit_preprocessors.LogitPreprocessor(session: LLMGenerationSession)[source]#

Bases: ABC

Base class for a logit preprocessor.

langdash.response module#

Classes:

RespChoice(choice, tokens_counter)

Choice response.

RespInfer(tokid, tokstr, running_infer)

Inference response.

RespInferEnd(running_infer, tokens_counter)

RespInject(tokens_counter)

Injected response.

RespReturns(key)

Returns response.

RespSubchain(key, resp_subchain)

Subchain response.

Response()

Base class for responses from language model.
class langdash.response.RespChoice(choice: int, tokens_counter: int)[source]#

Bases: Response

Choice response.

class langdash.response.RespInfer(tokid: int, tokstr: str, running_infer: str)[source]#

Bases: Response

Inference response. Will be emitted on every token generated.

tokid#

Token ID

Type:

int

tokstr#

Token string representation

Type:

str

running_infer#

Current generated string

Type:

str

class langdash.response.RespInferEnd(running_infer: str, tokens_counter: int)[source]#

Bases: Response

class langdash.response.RespInject(tokens_counter: int)[source]#

Bases: Response

Injected response. Will be emitted every time new context is injected.

tokens_counter#

Number of tokens injected to model

Type:

int

class langdash.response.RespReturns(key: str)[source]#

Bases: Response

Returns response. Acts as a header for return values.

key#

Return key

Type:

str

class langdash.response.RespSubchain(key: str, resp_subchain: Response)[source]#

Bases: Response

Subchain response.

key#

Return key

Type:

str

resp_subchain#

Response generated in subchain

Type:

langdash.response.Response

class langdash.response.Response[source]#

Bases: object

Base class for responses from language model.

langdash.sampling module#

Functions:

logits_to_probs(logits)

Converts logit tensor to probability tensor using softmax.

sample(logits, args, ctx)

Sample from a distribution of tokens specified by logits.
langdash.sampling.logits_to_probs(logits: torch.Tensor)[source]#

Converts logit tensor to probability tensor using softmax.

langdash.sampling.sample(logits: torch.Tensor, args: InferArgs, ctx: Sequence[int]) → int[source]#

Sample from a distribution of tokens specified by logits.

Parameters:

  • logits (torch.FloatTensor) – Logits to sample from.

  • args (InferArgs) – Sampling arguments.

  • ctx (Sequence[int]) – Sequence of tokens generated so far.

Returns:

The token sampled.

Module contents#