emb_diversity.utility package

Submodules

emb_diversity.utility.project_root.find(marker_files=None)

Find project root by looking for marker files.

Validation helpers for measure functions.

Checks that depend on measure-specific arguments (e.g. the distance metric) live here; they cannot run in the shared input validation of to_numeric_array, which never sees those arguments.

emb_diversity.utility.validate.ensure_cosine_defined(X, metric)

Raise if metric is cosine and X contains all-zero rows.

Cosine distance divides by the row norms, so an all-zero row would silently turn every distance involving it into nan.

Parameters:

• X (ndarray) – Data matrix of shape (n_samples, n_features).

• metric – The metric name or callable the caller will compute with; only the string "cosine" triggers the check.

Raises:

ValueError – If metric == "cosine" and X has at least one all-zero row.

Return type:

None

emb_diversity.utility.validate.warn_on_zero_norm_rows(X, measure)

Warn if X contains all-zero rows before cosine-kernel normalization.

The cosine-similarity kernels (kernel_type="cs" with normalize=True) divide each row by its L2 norm. A zero vector has no direction, so cosine similarity is undefined for it. The measure still proceeds — it clips the zero norm to a tiny positive value, so the zero row stays zero and contributes as if near-orthogonal to every other point — but its contribution to the score is not meaningful, so this warns.

Parameters:

• X (ndarray) – Data matrix of shape (n_samples, n_features).

• measure (str) – Name of the calling measure, used in the warning message.

Return type:

None

Module contents

emb_diversity.utility.cached_encode(texts, encode_fn, model_name, cache_dir=PosixPath('.cache/embeddings'), max_workers=8, key_suffixes=None)

Wrapper that adds disk caching to any encode function.

Parameters:

• texts (Sequence[str]) – Input texts to encode.

• encode_fn (Callable[[List[str]], List[List[float]]]) – Function that takes a list of strings and returns List[List[float]].

• model_name (str) – Used to namespace the cache (different models get different cache folders).

• cache_dir (Path) – Root cache directory.

• max_workers (int) – Threads for parallel I/O.

• key_suffixes (Sequence[str] | None) – Optional per-text suffix folded into each cache key, aligned positionally with texts. Distinguishes embeddings of the same text produced under different settings (e.g. truncation vs chunking). When None, every suffix is empty.

Return type:

List[List[float]]

Returns:

List of embedding vectors as lists of floats.

emb_diversity.utility.clear_cache(model_name=None, cache_dir=PosixPath('.cache/embeddings'))

Clear cached embeddings. If model_name given, only that model.

Return type:

None