NannyML monitors machine-learning models after deployment and estimates performance when ground-truth labels are delayed or unavailable. It detects concept drift and both multivariate and univariate data drift, then links drift alerts to changes in model performance. Data-quality checks and ranked alerts can help teams investigate features associated with performance issues. Users can connect model performance to monetary or other business outcomes. NannyML Cloud supports webhooks for retraining actions and an SDK to automate monitoring-data ingestion. Cloud is available through Azure and AWS marketplaces as SaaS or a managed application; in the managed option, the application and required infrastructure are provisioned within the customer's cloud subscription. The Azure managed-application documentation describes encrypted traffic and disks, but says automated backups were still being implemented and advises customers to safeguard model inputs and outputs elsewhere. The open-source library is self-managed and installs with pip or conda. Direct local dataset uploads through the Cloud UI must be smaller than 100 MB. The listed Starter plan costs 399.00 USD per month and includes 2 models, 10 M predictions, email support, and a 30-day free trial.
Who it is for
NannyML suits teams responsible for monitoring deployed machine-learning models, especially when ground truth is delayed or absent. Its open-source library is an option for teams prepared to manage their own installation.
What is good
- Estimates model performance without immediate ground-truth labels.
- Detects concept, multivariate, and univariate drift.
- Links drift alerts to model performance changes.
- Cloud supports webhooks and automated data ingestion.
- Open-source library installs with pip or conda.
What to know first
- Cloud UI local uploads must be smaller than 100 MB.
- Azure managed-application automated backups were still being implemented.
- Starter plan includes 2 models.
Verdict
NannyML combines drift monitoring, performance estimation, and data-quality checks for deployed models. Teams considering the Azure managed application should account for the documented backup limitation and safeguard inputs and outputs elsewhere.
NannyML plans and pricing
All plansCompared on model monitoring software
- Free plan
- Yes
- Paid from
- $99/mo
- Drift monitoring
- Yes
- Model performance metrics
- Yes
- Data quality checks
- Yes
- Alert channels
- email, Slack, webhook
- Included model limit
- 2 models
