Evidently Cheat Sheet

01Install & importPython 3.10+

pip install evidently
Core library. Also on conda-forge: conda install -c conda-forge evidently.
pip install evidently[llm]
Adds the extra dependencies for LLM judges and semantic evals. Other extras: spark, s3, gcs, fsspec, sql.
pip install tracely
Sister package for OpenTelemetry-based tracing of a live LLM app.
from evidently import Dataset, DataDefinition, Report
The three objects you need every single time. All top-level since 0.7.
from evidently.presets import *
Pre-built Report templates. Note: evidently.presets, not the old evidently.metric_preset.
from evidently.metrics import *
Individual dataset- and column-level metrics.
from evidently.descriptors import *
Row-level text / LLM evaluators.
from evidently.tests import *
The condition operators: eq gt gte lt lte not_eq is_in not_in.

02Dataset — wrap the DataFrame★the entry point

eval_data = Dataset.from_pandas(df, data_definition=DataDefinition())
An empty DataDefinition() auto-maps columns by dtype and by name. Good enough to start.
Dataset.from_pandas(df, data_definition=schema, descriptors=[...])
Compute row-level scores at the same time you build the Dataset.
eval_data.add_descriptors(descriptors=[...])
Or add them later to an existing Dataset. Same result, two styles.
eval_data.as_dataframe()
Get the frame back — original columns plus every descriptor column. Your main debugging view.
report.run(df_raw, None)
You can pass a bare DataFrame for simple summary and drift checks, but the docs recommend always building the Dataset explicitly.

03DataDefinition — column typesreplaces ColumnMapping

DataDefinition(numerical_columns=["age", "salary"])
Explicit beats automatic — stops numeric columns with few unique values being read as categorical.
DataDefinition(categorical_columns=["dept"])
Default catch-all: everything non-numeric and non-datetime.
DataDefinition(text_columns=["question", "answer"])
No automatic mapping exists for text. Required for text drift detection.
DataDefinition(datetime_columns=["joined"])
A column type. You can have many. Ignored in drift calculations.
# a column you leave out of an explicit mapping is ignored everywhere
Excluding columns from the mapping silently drops them from all evaluations.

04Roles: id, timestamp, descriptorstype vs role

DataDefinition(id_column="Id", timestamp="Date")
Auto-mapped only from columns literally named id and timestamp. Both are ignored in drift.
# datetime_columns = a TYPE (many allowed)
timestamp is a role — exactly one, used as the plot index. Different things.
DataDefinition(numerical_descriptors=["user_rating"])
Tell Evidently that a column you computed elsewhere should be treated as a descriptor.
DataDefinition(categorical_descriptors=["model_type"])
Descriptors get picked up by TextEvals() and plotted as descriptors in the UI.
# descriptors you generate are mapped automatically
You only need the two options above for externally computed scores.

05Mapping ML taskstarget & prediction

DataDefinition(regression=[Regression(target="y_true", prediction="y_pred")])
Defaults are target / prediction. It is a list — several regressions in one table are allowed.
DataDefinition(classification=[BinaryClassification(target="y", prediction_labels="pred")])
Binary defaults: prediction_probas="prediction", pos_label=1.
MulticlassClassification(target=..., prediction_labels=..., prediction_probas=["0","1","2"])
Probability column names must match the class labels exactly; target/prediction values should be strings.
DataDefinition(ranking=[Recsys()])
Defaults user_id, item_id, target, prediction. Prediction is a score by default, or a rank.
labels={"0": "legit", "1": "fraud"}
Display only — renames classes in the report, changes nothing in the maths.

06Current vs reference★order matters

my_eval = report.run(current_data, reference_data)
Current first. The dataset you are evaluating, then the baseline you compare it against.
my_eval = report.run(current_data=cur, reference_data=ref)
Keywords work too, and are worth using until the order is muscle memory.
my_eval = report.run(eval_data, None)
Single dataset. Perfectly normal — only drift genuinely requires two.
# reference unlocks three things
Side-by-side comparison, drift detection, and auto-generated test conditions derived from the baseline.
# both datasets must share an identical data definition
Build them from the same schema object.

07The CLI & the local UIoptional dashboard

evidently ui
Serves the dashboard over the workspace folder in the current directory. Opens on localhost:8000.
evidently ui --workspace ./workspace --port 8080
Point at a specific workspace, move off the default port.
evidently ui --demo-projects all
Launches pre-populated demo projects — the fastest way to see what the UI does.
uv run --with evidently evidently ui --demo-projects all
Same thing with no install, if you have uv.
ws = Workspace.create("evidently_ui_workspace")
from evidently.ui.workspace import Workspace. Backends: filesystem, any SQL database, or S3-compatible storage via fsspec.
ws.add_run(project.id, my_eval, include_data=True)
Log a snapshot. include_data=True ships the scored rows too, so you can sort and browse them.

08Report — the core object★config, then run

report = Report([DataSummaryPreset()])
A plain positional list. The Report holds only configuration — no data yet.
my_eval = report.run(cur, ref)
Running returns a separate result object (a snapshot). The Report itself is reusable.
Report([DataDriftPreset(), DataSummaryPreset(), ColumnCount()])
Mix presets and individual metrics freely in one Report.
DataDriftPreset(columns=["target", "prediction"])
Limit any preset to a subset of columns.
from evidently import compare compare(my_eval_1, my_eval_2, my_eval_3)
Line several snapshots up side by side as a DataFrame.
from evidently.metrics.group_by import GroupBy GroupBy(MaxValue(column="Salary"), "Department")
Compute a metric per segment. Auto-generated test conditions are not available with GroupBy.

09The five Presets★that is the whole list

DataSummaryPreset()
Descriptive stats and a profile for every column. The general-purpose starting point.
DataDriftPreset()
Distribution shift for all columns. Requires a reference dataset.
TextEvals()
Summarises every descriptor you computed. Compute descriptors first or it has nothing to show.
ClassificationPreset()
Quality metrics plus confusion matrix and curves. Optional probas_threshold.
RegressionPreset()
Error metrics with error distribution and normality plots.
# gone: DataQualityPreset, TargetDriftPreset, RecsysPreset, DataStabilityTestPreset...0.7
The old preset zoo was consolidated into these five. Ranking metrics still exist — as individual metrics, not a preset.

10Column-level Metricscolumn= required

ValueStats(column="age")
Small preset: unique / missing / min / max / mean / std / quantiles at once. Adapts to the column type.
MinValue() MaxValue() MeanValue() StdValue() MedianValue()
One number each. Result key: value.
QuantileValue(column="age", quantile=0.95)
Defaults to 0.5 if you omit the quantile.
MissingValueCount(column="age")
Returns both count and share — which matters when you write tests.
InRangeValueCount(column="age", left=18, right=80)
Also OutRangeValueCount. Both need left and right.
InListValueCount(column="city", values=["NY", "LON"])
Also OutListValueCount. Category allow-lists and deny-lists.
CategoryCount(column="city", category="NY")
Use categories=["a","b"] for a joint share. Returns count and share.
UniqueValueCount(column="city")
Result is a dict of count, share.

11Dataset-level Metricsno column needed

DatasetStats()
Small preset: rows, columns by type, missing values, empty columns.
RowCount() ColumnCount()
The two simplest metrics in the library, and the two most useful in a pipeline guard.
DuplicatedRowCount() DuplicatedColumnsCount()
Default test with no reference: fails if there is even one.
EmptyRowsCount() EmptyColumnsCount()
Same zero-tolerance default.
ConstantColumnsCount() AlmostConstantColumnsCount()
"Almost constant" means 95% identical values — a good dead-feature detector.
DatasetMissingValueCount() ColumnsWithMissingValuesCount()
Whole-table missingness, plus a per-column breakdown in the visual.
Accuracy() Precision() Recall() F1Score() RocAUC() LogLoss()
Classification. Most require you to pick at least one visual: conf_matrix, pr_curve, pr_table, roc_curve.
MAE() RMSE() MAPE() R2Score() MeanError() AbsMaxError()
Regression. MAE, MAPE and MeanError return mean and std — test them with mean_tests / std_tests.
NDCG(k=10) MAP() MRR() HitRate() PrecisionTopK() RecallTopK()
Ranking and RAG retrieval. k is required on all of them.

12Metric generatorsstop writing loops

from evidently.generators import ColumnMetricGenerator
Applies one metric across many columns.
ColumnMetricGenerator(ValueDrift)
Pass the metric class, not an instance. Applies to every column.
ColumnMetricGenerator(ValueDrift, columns=["a", "b"], metric_kwargs={"method": "psi"})
metric_kwargs forwards parameters to each generated metric.
ColumnMetricGenerator(UniqueValueCount, column_types='cat')
Filter by type: 'num', 'cat', or 'all'.
ColumnMetricGenerator(MinValue, column_types='num', metric_kwargs={"tests": [gt(0)]})
Generates the same Test across every numerical column.

13Descriptors — the row level★one score per row

descriptors=[Sentiment("answer", alias="Sentiment")]
First positional argument is the column. Descriptors add a new column to the Dataset.
TextLength("answer", alias="Length")
Always set an alias. It becomes the column header and the label in every chart; auto-generated names get long fast.
SemanticSimilarity(columns=["question", "answer"])
Multi-column descriptors take columns= instead of a single positional column.
eval_dataset.as_dataframe()
Inspect the scores. Sort by a descriptor to find the bad rows — this is the real workflow.
report = Report([TextEvals()])
Then summarise. TextEvals runs ValueStats over each descriptor.
TextEvals(columns=["Sentiment", "Length"])
Restrict the summary to specific descriptors for a lighter report.
# descriptors are computed once, reused many times
That decoupling is the point: score rows once, then aggregate or test them repeatedly with no recomputation.

14Deterministic descriptorsfree & instant

TextLength() WordCount() SentenceCount()
Absolute counts. Length is in symbols.
OOVWordsPercentage() NonLetterCharacterPercentage()
Both 0–100 scores. OOV needs NLTK dictionaries downloaded.
Contains(items=["chatgpt"], mode="any")
Substring match. DoesNotContain is the inverse. Options: mode, case_sensitive.
IncludesWords(words_list=["sorry", "apologize"])
Vocabulary words only, with optional lemmatize. ExcludesWords is the inverse — good for profanity lists.
ItemMatch(["Answer", "Expected_items"])
Per-row expected lists held in a second column. Also ItemNoMatch, WordMatch, WordNoMatch.
ExactMatch(columns=["answer", "target"])
Two-column equality. The cheapest possible regression check.
RegExp(reg_exp=r"^I") BeginsWith(prefix=...) EndsWith(suffix=...)
Pattern matching, all returning True/False per row.
IsValidJSON() IsValidPython() IsValidSQL() ContainsLink()
Syntax validation without executing anything. Essential for tool-calling and code-gen output.
JSONSchemaMatch(expected_schema={"name": str}, validate_types=True)
Keys present and non-null; exact_match=True also forbids extra keys. JSONMatch compares against a reference column.
CustomDescriptor(func=my_fn) CustomColumnsDescriptor(func=my_fn)
Your own Python. The first can take and return multiple columns; the second is single-column.

15ML-based descriptorslocal models, no API key

Sentiment("answer")
NLTK word-based model. Returns −1 (negative) to 1 (positive). Needs vader_lexicon.
SemanticSimilarity(columns=["question", "answer"])
Cosine similarity via all-MiniLM-L6-v2. Read it as 0 different, 0.5 unrelated, 1 identical — 0.5 is the floor in practice, not the midpoint.
BERTScore(columns=["Answer", "Target"])
Token-embedding F1. Options: model, tfidf_weighted.
HuggingFaceToxicity("answer", toxic_label="hate")
Probability from roberta-hate-speech-dynabench-r4. Scale 0–1.
HuggingFace("answer", ...)
Bring any HF classifier you like as an evaluator.
nltk.download('words'); nltk.download('wordnet')
Plus omw-1.4 and vader_lexicon. Do this once or the NLTK-backed descriptors fail.

16LLM judges — built in★needs an API key

DeclineLLMEval("answer", alias="Denials")
Returns the label DECLINE or OK. Detects refusals.
ToxicityLLMEval() BiasLLMEval() NegativityLLMEval() PIILLMEval()
Binary label plus reasoning. Labels are TOXICITY/OK, BIAS/OK, NEGATIVE/POSITIVE, PII/OK.
CorrectnessLLMEval("Response", target_output="Target")
CORRECT/INCORRECT against a ground-truth column.
FaithfulnessLLMEval("Response", context="Context")
The hallucination check: FAITHFUL/UNFAITHFUL. Runs on the response, takes the context as a parameter.
ContextQualityLLMEval("Context", question="Question")
Retrieval-side: does the context even contain the answer? VALID/INVALID.
CompletenessLLMEval("Response", context="Context")
Did the answer use everything the context offered? COMPLETE/INCOMPLETE.
ContextRelevance("Question", "Context", aggregation_method="hit")
Scores a list of chunks. method="semantic_similarity" (default, no LLM) or "llm"; aggregate by mean or hit.
provider="anthropic", model="claude-3-5-sonnet-20240620"
Default is OpenAI gpt-4o-mini. Routed through litellm, so 50+ providers work — match their naming exactly.
include_reasoning=False, include_score=True
Trim the explanation or add a numeric score. The score column is named "Alias score".

17LLM judges — your own criteriatemplates, not raw prompts

from evidently.llm.templates import BinaryClassificationPromptTemplate
You write only the grading criteria. Evidently assembles the full prompt, output format and parsing.
BinaryClassificationPromptTemplate(criteria=..., target_category="CONCISE", non_target_category="VERBOSE")
Both category names are required. Which one is "target" only matters when you later measure the judge itself.
uncertainty="unknown"
Where to put ambiguous cases. Options: unknown (default), target, non_target.
pre_messages=[("system", "You are a judge which evaluates text.")]
Set the evaluator's role and context.
print(my_template.get_template())
Preview the assembled prompt before you spend tokens on it.
LLMEval("response", template=my_template, provider="openai", model="gpt-4o-mini", alias=...)
Apply the template as a descriptor like any other.
additional_columns={"context": "Context"}
Multi-column judges: reference {context} as a placeholder inside criteria, then map it here. Do not mention the primary column in the prompt.
MulticlassClassificationPromptTemplate(category_criteria={...})
A dict of category name to definition. Keep the rubric short — few, sharply defined classes grade far more reliably.
from evidently.utils.llm.wrapper import AnthropicOptions options=AnthropicOptions(api_key=..., rpm_limit=50)
Pass keys and rate limits inline instead of via environment variables.

18Tests = Metric + condition★no separate object

MinValue(column="Age", tests=[gte(18)])
That is the whole idea. Add tests= to any metric and it becomes a Test.
report = Report([MissingValueCount(column="Age", tests=[eq(0)])])
Results appear as an extra tab in the same HTML report. There is no TestSuite class any more.
MinValue(column="Age", tests=[gte(17), lte(19)])
Several conditions on one metric become several separate Tests.
MissingValueCount(column="Age", share_tests=[lte(0.1)])
Metrics returning count and share: tests checks the absolute count, share_tests the proportion.
MAE(mean_tests=[lte(5)], std_tests=[lte(2)])
Multi-output metrics get one test parameter per output.
OutRangeValueCount(column="Age", left=18, right=80, tests=[eq(0)])
Some metrics need their own parameters before a condition makes sense.
# four outcomes: Pass · Fail · Warning · Error
Warning = condition missed but marked non-critical. Error = the check itself blew up.

19Condition operatorsevidently.tests

eq(val) not_eq(val)
Equal / not equal.
gt(val) gte(val) lt(val) lte(val)
Strict and inclusive comparisons. These four cover most real checks.
is_in([18, 21, 30]) not_in([16, 17])
Membership against a list of allowed or forbidden values.
gte(Reference(relative=0.1))
Condition expressed against the reference dataset — here, within ±10% of the baseline. No hard-coded thresholds.
gte(Reference(absolute=5))
Same idea with an absolute tolerance.
eq(0, is_critical=False)
Downgrades a failure to a yellow Warning, which does not fire alerts.
share_tests=[eq(0, is_critical=False), lte(0.1, is_critical=True)]
Layered severity: warn on any missing values, fail past 10%. A tidy alert-fatigue fix.
from evidently.future.tests import Referencedoc bug
Two docs pages still show evidently.future. On 0.7+ drop the .future: from evidently.tests import Reference.

20Auto-generated conditionszero-config testing

Report([DataSummaryPreset()], include_tests=True)
Turns any preset into a test suite with sensible defaults. Default is False.
report.run(cur, ref) # conditions derived from ref
With a reference, defaults become relative: missing values within ±10% of baseline, accuracy within 20%, and so on.
report.run(cur, None) # heuristic conditions
Without one, fixed heuristics apply: expect zero missing values; beat a dummy model. Min/max/mean have no heuristic at all.
Report([MinValue(column="Age")], include_tests=True)
Works on hand-picked metrics too, not just presets.
MissingValueCount(column="Age", tests=[])
An empty list opts this metric out of auto-generated tests while the rest keep them.
Report([RowCount(tests=[gt(10)]), MissingValueCount(column="Age")], include_tests=True)
Mix freely — your explicit conditions override the defaults where you set them.
# classification defaults compare against a dummy model
Accuracy, precision, recall and F1 fail if they cannot beat the built-in baseline. RocAUC fails at ≤ 0.5.

21Row-level tests & TestSummarypass/fail per row

TextLength("answer", alias="Length", tests=[lte(100, alias="Under 100")])
Descriptors take conditions too. Each adds a True/False column.
DeclineLLMEval("answer", alias="Denials", tests=[eq("OK", column="Denials")])
Judges return several columns (label, reasoning), so you must name which one to test with column=.
TestSummary(success_all=True, alias="Test result")
Collapses every row's tests into one verdict column.
TestSummary(success_any=..., success_count=..., success_rate=..., score=...)
Other aggregations: any passed, how many, what share, or a weighted score.
score_weights={"Under 100": 0.1, "Sentiment ok": 0.9}
Weight the score by test alias — hence always aliasing your tests.
# TestSummary only sees tests listed BEFORE it
Put it last in the descriptor list or it silently misses checks.
ColumnTest("Feedback", eq("Positive"))
Applies a row-level condition to any pre-existing column, descriptor or not.
CategoryCount(column="Test result", category=False, share_tests=[lte(0.1)])
The bridge back up: "no more than 10% of rows may fail" as a dataset-level Test.

22Drift — how the test is chosen★size & type decide

DataDriftPreset() DriftedColumnsCount() ValueDrift(column=...)
All three require a reference and share one default algorithm.
# reference ≤ 1000 rows → statistical tests, drift if p < 0.05
Numerical with >5 unique values: K–S. Categorical or ≤5 unique: chi-squared. Binary (≤2): Z-test on proportions.
# reference > 1000 rows → distances, drift if ≥ 0.1
Numerical: Wasserstein. Categorical: Jensen–Shannon. The switch is automatic and easy to miss.
# text → domain classifier, drift score = ROC AUC
Trains a model to tell current from reference. >1000 rows: drift if ROC AUC > 0.55. ≤1000: compared against a random classifier's 95th percentile to avoid false positives.
DriftedColumnsCount() # fails when 50% of columns drift
Dataset-level drift is just a rule on top of per-column results.
DataDriftPreset(drift_share=0.7)
Move that bar. Default 0.5.
# an empty column in either dataset raises an error
And drift does not react to a rise in nulls — test missing values separately.

23Drift — picking your own20+ methods

ValueDrift(column="Salary", method="psi")
Column level. Drift conditions use method/threshold, not the usual gt/lt test API.
DataDriftPreset(cat_method="psi", cat_threshold=0.3)
Also num_method/num_threshold, text_method/text_threshold, and per_column_method taking a dict.
method="psi" | "jensenshannon" | "kl_div" | "hellinger"
Distance and divergence measures — work on both numerical and categorical. Default threshold 0.1, drift when ≥.
method="ks" | "anderson" | "cramer_von_mises" | "mannw" | "t_test" | "es" | "empirical_mmd"
Numerical-only hypothesis tests. Default threshold 0.05, drift when p < threshold.
method="chisquare" | "z" | "fisher_exact" | "g-test" | "TVD"
Categorical-only tests.
method="wasserstein" | "ed"
Numerical-only distances (normed Wasserstein, energy distance).
method="perc_text_content_drift" | "abs_text_content_drift"
The two text options: percentile-based (default ≤1000 rows) or absolute ROC AUC threshold (default >1000).
from evidently.legacy.calculations.stattests import register_stattest, StatTest
Register a custom method. Your function takes (reference, current, feature_type, threshold) and returns (score, drift_detected).
stattest="psi" # legacy namedoc bug
The methods table still says "pass them using stattest". In the current API the parameter is method.

24Getting results outfive destinations

my_eval
Just name the object in Jupyter or Colab and it renders inline. There is no .show() any more.
my_eval.save_html("report.html")
Self-contained interactive file — the shareable artefact.
my_eval.json() · my_eval.save_json("report.json")
JSON is also the snapshot format the dashboard reads.
my_eval.dict()
Python dict for pipeline logic — pull one number out and branch on it. Was as_dict() in the old API.
eval_dataset.as_dataframe()
Row-level scores, not the aggregate. The two exports answer different questions.
ws.add_run(project.id, my_eval, include_data=False)
Log to a workspace to track results over time.

25Tags, metadata & timestampsfor tracking runs

Report([...], tags=["classification", "production"])
Tags on the Report config. Filterable, and usable to scope dashboard panels.
report.run(cur, ref, tags=["prompt_v1"])
Tags on the individual run — the right place for prompt and hyperparameter sweeps.
Report([...], metadata={"deployment": "shadow"})
Free-form key/value pairs.
Report([...], model_id=..., reference_id=..., batch_size=..., dataset_id=...)
Four built-in metadata fields with reserved names.
report.run(cur, ref, timestamp=datetime(2026, 1, 29))
Defaults to datetime.now(). Override it to backdate a run — essential when labels arrive late.

26The API the internet still teaches★all dead on 0.7+

from evidently import ColumnMappinggone
→ DataDefinition, passed into a Dataset. Maps by type and role, and supports multiple targets.
from evidently.report import Reportgone
→ from evidently import Report.
from evidently.metric_preset import DataDriftPresetgone
→ from evidently.presets import DataDriftPreset.
from evidently.test_suite import TestSuitegone
Reports and Tests were merged. Tests are now a tab on a Report, not a separate object and file.
Report(metrics=[DataDriftPreset()])gone
→ Report([DataDriftPreset()]). Positional list, no keyword.
report.run(reference_data=ref, current_data=cur, column_mapping=cm)gone
The one that bites hardest. → report.run(cur, ref) — current first, and column_mapping no longer exists.
report.show() · report.as_dict()gone
→ call the object to render; .dict() for the dictionary. save_html() survived unchanged.
TargetDriftPreset() DataQualityPreset() RecsysPreset()gone
Folded into the five current presets and individual metrics.
TestNumberOfMissingValues() TestShareOfDriftedColumns()gone
The whole named-Test catalogue is replaced by metric + condition.
from evidently.future import Report0.6 only
The transition-period namespace. On 0.7+ it is just evidently.
pip install evidently==0.6.7
If you inherit a codebase on the old API and cannot port it yet, pin here — the last version before the break. Old docs live at docs-old.evidentlyai.com.

27When something looks wrongincluding in the docs

# the LLM quickstart still walks you through Evidently Cloud signup
The self-hosting page states Evidently Cloud is no longer available as a SaaS product, and app.evidently.cloud now serves a shutdown notice. Ignore CloudWorkspace and self-host instead.
ws.add_report(...) # stale
Appears in the descriptors guide. The current method is ws.add_run(...).
# docs changelog stops at 0.7.11 (Jul 2025); PyPI ships 0.7.21 (Mar 2026)
Read the GitHub releases, not the changelog page, when you need to know what changed.
# descriptor scores appear but TextEvals() shows nothing
You summarised the wrong object. Pass the Dataset you added descriptors to, not the original DataFrame.
# text drift silently skipped
text_columns has no automatic mapping. Unmapped text is treated as categorical.
# evaluation takes forever
Everything is processed in memory. Sample large datasets — random or stratified — before drift detection.
# LLM judge bill larger than expected
One API call per row per judge. Prototype on a slice, and prefer SemanticSimilarity or keyword descriptors where they suffice.
# sweeping numbers, no clear cause
Check the reference row count. Crossing 1000 rows silently switches drift from p-values to distances.

evidently✓✗

Setup & the data layer

Scoring rows, summarising datasets

Tests, drift, output — and the legacy minefield

Four things worth seeing

How the drift test gets chosen

The argument order that bites

One Report, two tabs

Which RAG descriptor watches which edge

Worth memorizing