langdash.models package#

Submodules#

langdash.models.ctransformers module#

Classes:

CTransformersModel([model_path])

ctransformers model.

CTransformersSession(*args, **kwargs)

Session for ctransformers model.
class langdash.models.ctransformers.CTransformersModel(model_path: str | None = None, **model_kwargs)[source]#

Bases: LLM[CTransformersSession]

ctransformers model.

Classes:

Session

alias of CTransformersSession

Methods:

session(**kwargs)

Create a new session for the given model.
Session#

alias of CTransformersSession Methods:

clone_state()

Clone the current state of the language model.

flush_token(*a, **k)

Flushes the previous token into the language model if healing is enabled.

get_vocab()

Returns a mapping of token strings to their respective ids.

next_token_logits()

Returns the logits for next token.

next_token_probs()

Returns the probabilities for next token.

set_state(state)

Set the state of the language model.

tokenize(text[, add_special_tokens])

Tokenize the given text into a list of tokens.

tokenize_special(text)

Tokenize the given text as a special token.

Attributes:

context_length

Returns the context length of the model, or zero for models that don't support one.
session(**kwargs)[source]#

Create a new session for the given model.

Parameters:

  • default_infer_args (InferArgs) – Default arguments for the inference.

  • token_healing (bool) – Whether to enable token healing. Defaults to True.

  • event_handlers (Optional[LLMGenerationSessionEvents]) – Event handlers for prompt events. Defaults to None.

Returns:

A new session object.

class langdash.models.ctransformers.CTransformersSession(*args, **kwargs)[source]#

Bases: TensorBasedInferWithSessionMixin, LLMGenerationSessionForRawText[CTransformersModel, LLMState]

Session for ctransformers model.

Methods:

clone_state()

Clone the current state of the language model.

flush_token(*a, **k)

Flushes the previous token into the language model if healing is enabled.

get_vocab()

Returns a mapping of token strings to their respective ids.

next_token_logits()

Returns the logits for next token.

next_token_probs()

Returns the probabilities for next token.

set_state(state)

Set the state of the language model.

tokenize(text[, add_special_tokens])

Tokenize the given text into a list of tokens.

tokenize_special(text)

Tokenize the given text as a special token.

Attributes:

context_length

Returns the context length of the model, or zero for models that don't support one.
clone_state() → ctransformers.llm.LLMState | None[source]#

Clone the current state of the language model.

Returns:

The current state as an object.

property context_length: int#

Returns the context length of the model, or zero for models that don’t support one.

flush_token(*a, **k)[source]#

Flushes the previous token into the language model if healing is enabled.

Warning

Unexpected behavior might occur if the previous token is a “boundary” token, like the space ‘ ‘ token.

get_vocab()[source]#

Returns a mapping of token strings to their respective ids.

next_token_logits() → torch.Tensor[source]#

Returns the logits for next token.

next_token_probs() → torch.Tensor[source]#

Returns the probabilities for next token.

set_state(state: ctransformers.llm.LLMState | None)[source]#

Set the state of the language model.

Parameters:

state (Optional[T_LLMState]) – The state of the language model, or None to clear the state.

tokenize(text: str, add_special_tokens: bool = False) → List[int][source]#

Tokenize the given text into a list of tokens.

Parameters:

  • text (str) – The text to tokenize.

  • add_special_tokens (bool) – Whether to add special tokens to the output.

Returns:

The list of tokens.

tokenize_special(text: str) → int | None[source]#

Tokenize the given text as a special token.

Parameters:

text (str) – The text to tokenize.

Returns:

The token id, or None if the text doesn’t correspond to a special token.

langdash.models.llama_cpp module#

Classes:

LlamaCppModel([model_path])

llama.cpp model.

LlamaCppSession(*args, **kwargs)

Session for llama.cpp model.
class langdash.models.llama_cpp.LlamaCppModel(model_path: str | None = None, **model_kwargs)[source]#

Bases: LLM[LlamaCppSession]

llama.cpp model.

Creates a template for the Llama language model (using the llama.cpp library).

Classes:

Session

alias of LlamaCppSession

Methods:

session(**kwargs)

Create a new session for the given model.
Session#

alias of LlamaCppSession Methods:

clone_state()

Clone the current state of the language model.

flush_token(*a, **k)

Flushes the previous token into the language model if healing is enabled.

get_vocab()

Returns a mapping of token strings to their respective ids.

next_token_logits()

Returns the logits for next token.

next_token_probs()

Returns the probabilities for next token.

set_state(state)

Set the state of the language model.

tokenize(text[, add_special_tokens])

Tokenize the given text into a list of tokens.

tokenize_special(text)

Tokenize the given text as a special token.

Attributes:

context_length

Returns the context length of the model, or zero for models that don't support one.
session(**kwargs)[source]#

Create a new session for the given model.

Parameters:

  • default_infer_args (InferArgs) – Default arguments for the inference.

  • token_healing (bool) – Whether to enable token healing. Defaults to True.

  • event_handlers (Optional[LLMGenerationSessionEvents]) – Event handlers for prompt events. Defaults to None.

Returns:

A new session object.

class langdash.models.llama_cpp.LlamaCppSession(*args, **kwargs)[source]#

Bases: TensorBasedInferWithSessionMixin, LLMGenerationSessionForRawText[LlamaCppModel, LlamaState]

Session for llama.cpp model.

Methods:

clone_state()

Clone the current state of the language model.

flush_token(*a, **k)

Flushes the previous token into the language model if healing is enabled.

get_vocab()

Returns a mapping of token strings to their respective ids.

next_token_logits()

Returns the logits for next token.

next_token_probs()

Returns the probabilities for next token.

set_state(state)

Set the state of the language model.

tokenize(text[, add_special_tokens])

Tokenize the given text into a list of tokens.

tokenize_special(text)

Tokenize the given text as a special token.

Attributes:

context_length

Returns the context length of the model, or zero for models that don't support one.
clone_state() → llama_cpp.LlamaState | None[source]#

Clone the current state of the language model.

Returns:

The current state as an object.

property context_length: int#

Returns the context length of the model, or zero for models that don’t support one.

flush_token(*a, **k)[source]#

Flushes the previous token into the language model if healing is enabled.

Warning

Unexpected behavior might occur if the previous token is a “boundary” token, like the space ‘ ‘ token.

get_vocab()[source]#

Returns a mapping of token strings to their respective ids.

next_token_logits() → torch.Tensor[source]#

Returns the logits for next token.

next_token_probs() → torch.Tensor[source]#

Returns the probabilities for next token.

set_state(state: llama_cpp.LlamaState | None)[source]#

Set the state of the language model.

Parameters:

state (Optional[T_LLMState]) – The state of the language model, or None to clear the state.

tokenize(text: str, add_special_tokens: bool = False) → List[int][source]#

Tokenize the given text into a list of tokens.

Parameters:

  • text (str) – The text to tokenize.

  • add_special_tokens (bool) – Whether to add special tokens to the output.

Returns:

The list of tokens.

tokenize_special(text: str) → int | None[source]#

Tokenize the given text as a special token.

Parameters:

text (str) – The text to tokenize.

Returns:

The token id, or None if the text doesn’t correspond to a special token.

langdash.models.rwkv_cpp module#

langdash.models.sentence_transformers module#

Classes:

SentenceTransformersModel(model_name)

sentence_transformers embedding model.

SentenceTransformersSession(*args, **kwargs)

Session for sentence_transformers embedding model.
class langdash.models.sentence_transformers.SentenceTransformersModel(model_name: str)[source]#

Bases: EmbeddingLLM[SentenceTransformersSession]

sentence_transformers embedding model.

Classes:

Session

alias of SentenceTransformersSession
Session#

alias of SentenceTransformersSession Methods:

embed(documents)

Infer the embedding of a list of text.

Attributes:

embedding_size

Returns the embedding size of the model.
class langdash.models.sentence_transformers.SentenceTransformersSession(*args, **kwargs)[source]#

Bases: LLMEmbeddingSession[SentenceTransformersModel]

Session for sentence_transformers embedding model.

Methods:

embed(documents)

Infer the embedding of a list of text.

Attributes:

embedding_size

Returns the embedding size of the model.
embed(documents: List[str]) → torch.Tensor[source]#

Infer the embedding of a list of text.

Parameters:

documents (List[str]) – The text to be embedded.

Returns:

The embedding vector of the list of text.

property embedding_size: int#

Returns the embedding size of the model.

langdash.models.transformers module#

Classes:

TransformersModel(model[, tokenizer])

transformers model.

TransformersSession(*args, **kwargs)

Session for transformers model.

TransformersState([_logits, _state, _next_token])

class langdash.models.transformers.TransformersModel(model: str | transformers.PreTrainedModel, tokenizer: str | transformers.PreTrainedTokenizer | None = None, **model_kwargs)[source]#

Bases: LLM[TransformersSession]

transformers model.

Creates a template for a language model powered by the transformers library.

Parameters:

  • model (Union[str, transformers.PreTrainedModel]) – The name of the model, or the model class itself.

  • tokenizer (Optional[Union[str, transformers.PreTrainedTokenizer]]) – The name of the tokenizer, or the tokenizer class itself. Defaults to None. If not set, the name of the model will be used to load the tokenizer.

Classes:

Session

alias of TransformersSession
Session#

alias of TransformersSession Methods:

clone_state()

Clone the current state of the language model.

get_vocab()

Returns a mapping of token strings to their respective ids.

next_token_logits()

Returns the logits for next token.

next_token_probs()

Returns the probabilities for next token.

set_state(state)

Set the state of the language model.

tokenize(text[, add_special_tokens])

Tokenize the given text into a list of tokens.

Attributes:

context_length

Returns the context length of the model, or zero for models that don't support one.
class langdash.models.transformers.TransformersSession(*args, **kwargs)[source]#

Bases: TensorBasedInferMixin, LLMGenerationSessionForRawText[TransformersModel, TransformersState]

Session for transformers model.

Methods:

clone_state()

Clone the current state of the language model.

get_vocab()

Returns a mapping of token strings to their respective ids.

next_token_logits()

Returns the logits for next token.

next_token_probs()

Returns the probabilities for next token.

set_state(state)

Set the state of the language model.

tokenize(text[, add_special_tokens])

Tokenize the given text into a list of tokens.

Attributes:

context_length

Returns the context length of the model, or zero for models that don't support one.
clone_state() → TransformersState | None[source]#

Clone the current state of the language model.

Returns:

The current state as an object.

property context_length: int#

Returns the context length of the model, or zero for models that don’t support one.

get_vocab()[source]#

Returns a mapping of token strings to their respective ids.

next_token_logits() → torch.Tensor[source]#

Returns the logits for next token.

next_token_probs() → torch.Tensor[source]#

Returns the probabilities for next token.

set_state(state: TransformersState | None)[source]#

Set the state of the language model.

Parameters:

state (Optional[T_LLMState]) – The state of the language model, or None to clear the state.

tokenize(text: str, add_special_tokens: bool = False) → List[int][source]#

Tokenize the given text into a list of tokens.

Parameters:

  • text (str) – The text to tokenize.

  • add_special_tokens (bool) – Whether to add special tokens to the output.

Returns:

The list of tokens.

class langdash.models.transformers.TransformersState(_logits: torch.Tensor | None = None, _state: Any = None, _next_token: Tuple[int, str] | None = None)[source]#

Bases: LLMState

Module contents#