I was initially very reluctant about testing.
Couple of gateways into testing:
def f1score(y_true, y_pred): """F1 score is given by this formula. F1 = 2 * (precision * recall) / (precision + recall) """ y_true = set(y_true) y_pred = set(y_pred) precision = sum([1 for i in y_pred if i in y_true]) / len(y_pred) recall = sum([1 for i in y_true if i in y_pred]) / len(y_true) if precision + recall == 0: return 0.0 else: return (2 * precision * recall) / (precision + recall)
import pytest assert f1score([1, 2, 3], [2, 3]) == 0.8 assert f1score(['None'], [2, 'None']) == pytest.approx(2/3) assert f1score([4, 5, 6, 7], [2, 4, 8, 9]) == 0.25 assert f1score([1, 2, 3, 4], ) == 0.4
First the concept of concept drift.
You have a nice model, trained well, evaluating correctly.
Then the environment changes.
One way to avoid concept drift is to retrain models regularly, before they get stale.
You want this to be low cost operation.
Stealing ideas from continuous deployment/delivery
These dashboards not just for software engineers!
This idea scales well. You can also log:
Each job is a container.
Each container should do one thing and do it well. Connect together like Lego.
Components can be parts of data pipeline, presentation of results, or parts of the model.
Data scientists love notebooks (jupyter)
A great experience is watching an engineer hate them
An even better experience is watching an engineer say: "they are a great tool for collaboration and experimentation, but wouldn't use them for much else"
Being data aware. Analysis and stats.
Dealing with uncertainty.
Knowing about the domain applications of our work.
If a data scientist who knows engineering is awesome, then an engineer who knows data science is too. Together we build super cool data driven products that scale.