2025-08-08

Build Machine Learning Models in Microsoft Fabric

Ship ML faster on Microsoft Fabric. This guide shows how to go from Lakehouse data to production models—without the spreadsheet chaos. You’ll get a practical blueprint for curated data layers, reproducible notebooks, MLflow tracking, governed model registry, and one-click batch/real-time serving.…

Fabric ML, End to End — A Practical Field Guide

Architecture at a glance

Data: OneLake → Lakehouse (bronze/silver/gold tables with Delta).
Compute: Fabric Notebooks (Spark + Python/R), preloaded DS stack.
Tracking: MLflow runs + artifacts + metrics (built into Fabric).
Registry & Governance: Versioned models, approvals, lineage.
Serving: Managed batch jobs (scheduled) or real-time endpoints (HTTP).
Consumers: Power BI, Power Apps, Power Automate, API clients.

Lakehouse: tame inputs, keep control

Layer your tables
- Bronze: raw ingests (schema-on-read, immutable).
- Silver: cleaned/typed joins (PII minimized, QA checks).
- Gold: feature-ready aggregates (documented contracts).
Access by intent
- Workspace RBAC + table ACLs; give DS read on silver/gold, write on features.
Data quality guardrails
- Expectations: not-null, ranges, referential checks; write failed rows to _quarantine.
Feature store lite
- Keep reusable features as gold tables (e.g., feat_customer_rolling_90d), with a YAML data dictionary.

Notebooks without the pain

Use Fabric kernels (pandas, PySpark, scikit-learn, PyTorch preinstalled).
Project scaffolding
/ml /notebooks 01_explore.ipynb 02_features.ipynb 03_train.ipynb 04_evaluate.ipynb /src data.py # read/write helpers features.py model.py /conf env.json # paths, table names
Repro in code, not clicks
- Read from Lakehouse via Spark SQL; write features as Delta.
- Seed small dev sample tables for fast iteration.

Fabric notebook → Lakehouse snippet (PySpark)

from pyspark.sql import functions as F

# Read curated data
sales = spark.read.table("lakehouse.gold.sales_txn")
tickets = spark.read.table("lakehouse.silver.support_tickets")

# Build features
feat = (sales.groupBy("customer_id")
              .agg(F.sum("amount").alias("amt_90d"),
                   F.countDistinct("order_id").alias("orders_90d"))
        .join(tickets.groupBy("customer_id")
                     .agg(F.count("*").alias("tickets_90d")), "customer_id", "left")
        .fillna({"tickets_90d":0}))

# Persist features
(feat.write.mode("overwrite")
     .format("delta")
     .saveAsTable("lakehouse.gold.feat_customer_90d"))

Track everything with MLflow (built-in)

Why: Run-to-run comparability, auditability, and painless rollbacks.
Log parameters, metrics, artifacts

import mlflow, mlflow.sklearn
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score

mlflow.set_experiment("/ml/experiments/churn")

with mlflow.start_run():
    X_train, X_val, y_train, y_val = ...
    clf = RandomForestClassifier(n_estimators=400, max_depth=12, random_state=42)
    clf.fit(X_train, y_train)
    pred = clf.predict_proba(X_val)[:,1]
    auc = roc_auc_score(y_val, pred)

    mlflow.log_param("n_estimators", 400)
    mlflow.log_param("max_depth", 12)
    mlflow.log_metric("val_auc", auc)
    mlflow.sklearn.log_model(clf, artifact_path="model", registered_model_name="churn_rf_v1")

Governance tips
- Tag runs with data_window, feature_set, code_commit.
- Pin training tables to snapshot versions (Delta) and log them (e.g., sales_txn@v123).

Registry, approvals, and lineage

Promote best runs → Registered Model with semantic versioning (1.0.0, 1.1.0-rc1).
Require two-step approval: DS owner → Business owner.
Capture model card metadata: purpose, data scope, fairness notes, monitoring SLOs.

Serving options (pick by workload)

Batch scoring (cheap, reliable)

Use scheduled Fabric job or Notebook to write predictions back to Lakehouse:

# Load prod model
model = mlflow.sklearn.load_model("models:/churn_rf_v1/Production")

batch = spark.read.table("lakehouse.gold.scoring_batch_2025_11")
pred = model.predict_proba(batch.toPandas())[:,1]
out = batch.withColumn("churn_score", F.array(*[F.lit(float(s)) for s in pred])) \
           .withColumn("scored_at", F.current_timestamp())
(out.write.mode("overwrite")
    .format("delta")
    .saveAsTable("lakehouse.gold.scored_churn_latest"))

Consumers: Power BI dataset points to scored_churn_latest; Power Automate pulls rows > threshold.

Real-time endpoint (interactive apps)

Deploy registered model as managed endpoint (Fabric UI).
Contract: POST /score → JSON; respond with scores + version.
Wire to apps: Power Apps/Automate call the endpoint; log request IDs for traceability.

When to choose which

Batch: dashboards, nightly ranking, email campaigns.
Real-time: form validation, product recommendations, on-click risk checks.

Monitoring & retraining

Data drift: compare live feature distributions vs training baseline; alert at KL/PSI thresholds.
Performance: track AUC/MAE monthly on labeled holdouts; revalidate after schema changes.
Ops health: endpoint latency, error rate, cost per 1k scores.
Automate retrains: time-based (e.g., monthly) + event-based (drift or perf breach) → new run → staged → canary.

Security & compliance

Least privilege: DS read on silver/gold; write only to /ml and feature tables.
Secrets: Managed identities/Key Vault—no creds in notebooks.
PII minimization: hash or tokenize; keep PII in silver, never in features/gold unless essential.
Auditability: MLflow lineage + approval records + deployment logs = one evidence trail.

What to measure (prove value fast)

Model power: AUC/PR-AUC (classification), MAE/RMSE (regression); lift@k, precision@k for business ranking.
Business impact: incremental revenue/savings per 1k predictions; conversion or churn deltas on A/B.
Velocity: time from data ready → first prod score; deploys per quarter.
Reliability: SLOs met, rollback MTTR, drift detection time.
Cost: $ per 1k batch scores; endpoint $/RPS; training GPU/CPU hours.

14-day rollout plan

Day 1–2 – Lakehouse hygiene: bronze/silver/gold, ACLs, quality checks.
Day 3 – Notebook scaffolding + data helpers; create dev sample tables.
Day 4–5 – Feature set v1; baseline model; log with MLflow.
Day 6 – Evaluation suite + model card; register v0.1.
Day 7 – Batch scoring job → scored_* table; Power BI quick win.
Day 8 – Monitoring MVP: drift checks + metric logging.
Day 9 – Approval workflow + promotion to Production.
Day 10 – (If needed) Real-time endpoint; throttle + logging.
Day 11–12 – A/B or backtest; define business KPIs; dashboard.
Day 13 – Cost & SLO guardrails; autoscaling; canary plan.
Day 14 – Runbook docs; handoff; backlog next features.

Common pitfalls (and quick fixes)

Notebook roulette: lock env to Fabric kernel; keep code in /src; save runs to MLflow only.
“Works on sample, fails on full.” Test on 5% + schema validations; avoid .toPandas() for big sets.
Ghost models in prod: require registry promotion; block ad-hoc file deploys.
Cost blowups: default to batch; cap endpoint concurrency; cache derived features.
Unexplainable changes: log dataset snapshots + code commit; require model card deltas for promotions.

Handy snippets

Read Delta with version pinning

sales_v = spark.read.option("versionAsOf", 123).table("lakehouse.gold.sales_txn")

Feature importance (sklearn)

import pandas as pd
fi = pd.Series(clf.feature_importances_, index=feature_cols).sort_values(ascending=False)
mlflow.log_artifact(fi.head(30).to_frame("importance").to_csv("fi.csv", index=True))

Simple drift (PSI)

import numpy as np
def psi(expected, actual, bins=10):
    e, b = np.histogram(expected, bins=bins, range=(np.min(expected), np.max(expected)))
    a, _ = np.histogram(actual,   bins=b,     range=(np.min(expected), np.max(expected)))
    e = np.clip(e / e.sum(), 1e-6, None); a = np.clip(a / a.sum(), 1e-6, None)
    return np.sum((a - e) * np.log(a / e))

If you want, drop your current Lakehouse table names and a target use case (e.g., churn, lead score, forecast). I’ll map a minimal feature set, a baseline model choice, and the exact batch/real-time wiring to ship your first Fabric deployment quickly.

Spotify · YouTube

Host: Mirko Peters