SPM Spa RedPajama35 Tokenizer

SentencePiece tokenizer for Spanish text trained on RedPajama.

Provides SpanishSPMTokenizer, a BPE SentencePiece tokenizer trained on the erickfmm/red_pajama_es_hq_35 dataset with special tokens [CLS], [SEP], [MASK], and [PAD].

class src.tokenizer.spm_spa_redpajama35.SpanishSPMTokenizer(vocab_size: int = 50000, model_path: str = None)

Bases: object

Spanish SentencePiece BPE tokenizer trained on RedPajama data.

Supports training a new tokenizer from the streaming dataset, loading a pre-trained model file, and encoding text with special tokens ([CLS], [SEP], [PAD]) and configurable max length.

vocab_size

Vocabulary size (may be updated after loading).

model_path

Path to the .model file, or None.

sp_model

The underlying SentencePieceProcessor, or None.

vocab

Token-to-ID mapping dictionary.

inverse_vocab

ID-to-token mapping dictionary.

storage_manager

StorageManager for temp file tracking.

__init__(vocab_size: int = 50000, model_path: str = None)

Initialize the tokenizer.

Parameters:

• vocab_size – Target vocabulary size for training.

• model_path – Optional path to an existing .model file. If provided and exists, the model is loaded immediately.

encode(text: str, max_length: int = 512) → Dict[str, List[int]]

Encode text with special tokens and padding/truncation.

Prepends [CLS], appends [SEP], and pads with [PAD] to reach max_length. Truncates if the tokenized sequence exceeds max_length.

Parameters:

• text – Raw input text string.

• max_length – Maximum sequence length including special tokens.

Returns:

Dictionary with "input_ids" and "attention_mask" lists, both of length max_length.

Raises:

RuntimeError – If the tokenizer has not been initialized (call train() or load() first).

load(model_path: str)

Load an existing SentencePiece model from disk.

Parameters:

model_path – Path to the .model file.

Raises:

FileNotFoundError – If the model file does not exist.

prepare_training_data(max_samples: int = 50000000, target_ram_gb: float = 100.0) → str

Prepare training data from erickfmm/red_pajama_es_hq_35

Parameters:

• max_samples – Maximum number of text samples to collect (default: 50M)

• target_ram_gb – Target RAM usage in GB (default: 100GB)

The actual limit will be determined by whichever comes first: max_samples or estimated RAM usage approaching target_ram_gb

train(model_prefix: str = 'es_spm_50k', max_training_samples: int = 50000000, target_ram_gb: float = 100.0)

Train SentencePiece tokenizer with maximum data utilization

Parameters:

• model_prefix – Prefix for output model files

• max_training_samples – Maximum samples to collect from dataset

• target_ram_gb – Target RAM usage (default: 100GB)