Getting Started
Installation
Optional Dependencies
Platform-specific Instructions
Dependencies
Core
Optional
LLMs
Seamless Chat with LLMs
Authentication
LLM Inference
API Request Format
StreamProcessor
Authentication
Data Labeling using LLMs
Chunking
Fixed-size chunking
Semantic Chunking
Recursive chunking
MLMs
Train Text Classifiers
Using the Classifier
Pretrain MLMs
Tokenizer and Tokenization
Initialize an MLM and Train
Further Pretrain MLMs
Model pretrained using Langformers
Existing Model from HuggingFace
Embeddings
Embed Sentences
Textual Similarity
Semantic Search
Loading an Existing Database
Rerank Sentences
Mimick a Pretrained Model
Library Reference
Tasks
tasks
Classifiers
HuggingFaceClassifier
LoadClassifier
Embedders
HuggingFaceEmbedder
Generators
OllamaGenerator
HuggingFaceGenerator
StreamProcessor
Labellers
HuggingFaceDataLabeller
OllamaDataLabeller
Mimickers
EmbeddingMimicker
MLMs
MLMTokenizerDatasetCreator
HuggingFaceMLMCreator
Rerankers
CrossEncoder
Searchers
FaissSearcher
ChromaDBSearcher
PineconeSearcher
Development
Changelog
v0.5.0 (2024-05-04)
v0.4.0 (2024-04-17)
v0.3.1 (2024-04-16)
v0.3.0 (2024-04-14)
v0.2.0 (2024-04-10)
v0.1.0 (2024-04-08)
License
Contributing
Setting Up the Project
Ways to Contribute
Pull Request Guidelines
Documentation
License
langformers
Index
Edit on GitHub
Index
_
|
A
|
C
|
E
|
F
|
G
|
H
|
L
|
M
|
O
|
P
|
Q
|
R
|
S
|
T
_
__init__() (langformers.classifiers.LoadClassifier method)
(langformers.embedders.HuggingFaceEmbedder method)
(langformers.generators.HuggingFaceGenerator method)
(langformers.generators.OllamaGenerator method)
(langformers.generators.StreamProcessor method)
(langformers.HuggingFaceClassifier method)
(langformers.labellers.HuggingFaceDataLabeller method)
(langformers.labellers.OllamaDataLabeller method)
(langformers.mimickers.EmbeddingMimicker method)
(langformers.mlms.HuggingFaceMLMCreator method)
(langformers.mlms.MLMTokenizerDatasetCreator method)
(langformers.rerankers.CrossEncoder method)
(langformers.searchers.ChromaDBSearcher method)
(langformers.searchers.FaissSearcher method)
(langformers.searchers.PineconeSearcher method)
A
add() (langformers.searchers.ChromaDBSearcher method)
(langformers.searchers.FaissSearcher method)
(langformers.searchers.PineconeSearcher method)
C
ChromaDBSearcher (class in langformers.searchers)
classify() (langformers.classifiers.LoadClassifier method)
count() (langformers.searchers.ChromaDBSearcher method)
(langformers.searchers.FaissSearcher method)
(langformers.searchers.PineconeSearcher method)
create_chunker() (langformers.tasks static method)
create_classifier() (langformers.tasks static method)
create_dataset() (langformers.HuggingFaceClassifier method)
create_embedder() (langformers.tasks static method)
create_generator() (langformers.tasks static method)
create_labeller() (langformers.tasks static method)
create_mimicker() (langformers.tasks static method)
create_mlm() (langformers.tasks static method)
create_reranker() (langformers.tasks static method)
create_searcher() (langformers.tasks static method)
create_tokenizer() (langformers.tasks static method)
CrossEncoder (class in langformers.rerankers)
E
embed() (langformers.embedders.HuggingFaceEmbedder method)
EmbeddingMimicker (class in langformers.mimickers)
F
FaissSearcher (class in langformers.searchers)
G
generate() (langformers.generators.HuggingFaceGenerator method)
(langformers.generators.OllamaGenerator method)
H
HuggingFaceClassifier (class in langformers)
HuggingFaceDataLabeller (class in langformers.labellers)
HuggingFaceEmbedder (class in langformers.embedders)
HuggingFaceGenerator (class in langformers.generators)
HuggingFaceMLMCreator (class in langformers.mlms)
L
label() (langformers.labellers.HuggingFaceDataLabeller method)
(langformers.labellers.OllamaDataLabeller method)
load_classifier() (langformers.tasks static method)
LoadClassifier (class in langformers.classifiers)
M
MLMTokenizerDatasetCreator (class in langformers.mlms)
O
OllamaDataLabeller (class in langformers.labellers)
OllamaGenerator (class in langformers.generators)
P
PineconeSearcher (class in langformers.searchers)
pipeline() (langformers.labellers.HuggingFaceDataLabeller method)
(langformers.labellers.OllamaDataLabeller method)
process() (langformers.generators.StreamProcessor method)
Q
query() (langformers.searchers.ChromaDBSearcher method)
(langformers.searchers.FaissSearcher method)
(langformers.searchers.PineconeSearcher method)
R
rank() (langformers.rerankers.CrossEncoder method)
run() (langformers.generators.HuggingFaceGenerator method)
(langformers.generators.OllamaGenerator method)
S
setup_routes() (langformers.generators.HuggingFaceGenerator method)
(langformers.generators.OllamaGenerator method)
similarity() (langformers.embedders.HuggingFaceEmbedder method)
StreamProcessor (class in langformers.generators)
T
tasks (class in langformers)
train() (langformers.HuggingFaceClassifier method)
(langformers.mimickers.EmbeddingMimicker method)
(langformers.mlms.HuggingFaceMLMCreator method)
(langformers.mlms.MLMTokenizerDatasetCreator method)