Deploy Inference

High-performance inference engine for deployed Frankenstein models.

Provides FrankensteinInference for loading quantized or standard deployment artifacts and running MLM predictions, masked token prediction, batch inference, and performance benchmarking.

class src.deploy.inference.FrankensteinInference(model_dir: str, device: str = 'auto', use_half_precision: bool = False)

Bases: object

Optimized inference engine for Frankenstein deployed models.

Loads deployment artifacts (config, quantized or standard weights, optional tokenizer) and provides methods for single/batch prediction, masked token prediction, and performance benchmarking.

model_dir

Path to the deployment directory.

device

Resolved PyTorch device string.

use_half_precision

Whether FP16 inference is enabled (CUDA only).

config

Loaded FrankensteinModelConfig.

model

Loaded FrankensteinEncoder in eval mode.

tokenizer

Optional SpanishSPMTokenizer instance.

__init__(model_dir: str, device: str = 'auto', use_half_precision: bool = False)

Initialize the inference engine.

Parameters:

• model_dir – Directory containing deployment artifacts (config.json, model*.pt, optional tokenizer.model).

• device – Device string ("auto", "cuda", "mps", or "cpu").

• use_half_precision – If True, use FP16 for faster inference (CUDA only).

batch_predict(texts: List[str], batch_size: int = 8, max_length: int = 512) → List[torch.Tensor]

Process multiple texts in batches.

Parameters:

• texts – List of input texts

• batch_size – Batch size for processing

• max_length – Maximum sequence length

Returns:

List of prediction tensors

benchmark(batch_size: int = 1, seq_length: int = 512, num_runs: int = 10)

Benchmark inference performance.

Parameters:

• batch_size – Batch size to test

• seq_length – Sequence length to test

• num_runs – Number of runs for averaging

predict(inputs: str | List[str] | torch.Tensor, max_length: int = 512, return_logits: bool = False) → torch.Tensor | List[torch.Tensor]

Run inference on input(s).

Parameters:

• inputs – Text string(s) or token tensor

• max_length – Maximum sequence length

• return_logits – If True, return raw logits; else return probabilities

Returns:

Model predictions (logits or probabilities)

predict_masked(text: str, mask_token: str = '[MASK]') → List[tuple]

Predict masked tokens (MLM task).

Parameters:

• text – Text with [MASK] tokens

• mask_token – Mask token string

Returns:

List of (position, top_predictions) tuples

src.deploy.inference.interactive_mode(engine: FrankensteinInference)

Interactive inference mode.

src.deploy.inference.main(argv=None)