image.jpg

Armaghan Khan

Product Manager

Model evaluation is one of the most important prerequisites prior to shipping an ML model. If you don’t properly evaluate your model, performance regressions might appear in production and have a direct and negative impact on your business. For example, if a faulty pedestrian detection model is deployed onto an autonomous vehicle, it can potentially cause an accident. In its most basic form, model evaluation is very straightforward, but as we discussed in our <a href="https://scale.com/blog/find-best-model" rel="noopener noreferrer" target="_blank">previous blog</a>, approaches that are too simple aren’t really reliable. Therefore, most ML teams aspire to maintain a mature model evaluation pipeline in order to systematically understand model performance. A mature evaluation pipeline covers the entire lifecycle of ML development:<ol><li>Model training and re-training</li><li>Model testing and validation</li><li>Model performance monitoring in production&nbsp;&nbsp;</li></ol>To cover all these aspects, ML teams need sophisticated tooling, which is why we assembled a list of 4 MLOps tools that we believe will help you build a great model evaluation pipeline:Weights &amp; Biases<img src="https://lh5.googleusercontent.com/FT9Rlbx977RDuvofnHrtTOxQPjRAjHtncmZQJZel9sDeCRiRYgdle8ubOonzVkVGgkxhxuv_7J5oSyXSSMpU2yZa-Vkv4zYxXibg2Z-nB42bjTd3hl4O8PaZc_Vpluk7S8Wsw5M8GpxNOw4mep2TOiqYX08B0haUhu-_YqmI9YXes0WfSIZAJV1KGg"><a href="https://wandb.ai/site/experiment-tracking" rel="noopener noreferrer" target="_blank">Source</a>Weights &amp; Biases (W&amp;B, for short) is a great tool for logging information and analyzing performance during model training. It is designed to track standard training metrics like loss, training dataset (“train”) accuracy, validation (“val”) dataset accuracy, GPU utilization, and more.&nbsp; W&amp;B will also log your raw data like images or point clouds along with their matching predictions like bounding boxes, cuboids, and segmentation masks. Moreover, W&amp;B supports experiment tracking. Users can log training run configurations, including sets of hyperparameters, and use these to reproduce the experiment later. Already widely adopted in the ML community, W&amp;B is a more powerful cloud-hosted alternative to TensorBoard, especially if teams want to converge on a collaborative tool.COCO Eval API<img src="https://lh5.googleusercontent.com/SXczlMWaBJoU7IofJKyyrJoP1yYnfo_m7VvAY55w_eZMU-tvP9Bz5Q69U87f18Qv_6gvZ7FGWLjdnH2PZqfJm1bp9BxwyqQvuas3FT-hxgZkBwExjg_cifCN8noswnenhVnDsPjo1P8inay5wUrFAMHhFhvTNiCaITChXgcP8-5DCF-a-g036X6mjA">While there are a lot of tools available to log training and evaluation metrics, COCO’s eval API is among the handful that provide these metric definitions and implementation out of the box. Released as a toolkit for interacting with the MSCOCO dataset, the COCO eval API has now become the standard tool for many ML developers working on computer vision use cases like object detection. The API supports the calculation of IoU, precision, and recall on bounding boxes, segmentation masks, and keypoints. It also allows users to aggregate these scores—both per item and per class. Although very bare bones, the COCO eval API is a handy tool for obtaining standard metrics.Scale Validate<img src="https://lh6.googleusercontent.com/H5OAAYhwy544Pj2iFx8yuDCeKxPJ7phxvazeRPgDgYI5s8tUTHra66cuqLzgxHTcCT6qOJuKG8Wd3DQTnkP754VhCekZMV2wqQXZxgUfp2oTWNIhwnGtPIp4a4g5gUphE41hhAJ_X3DaJU-hx7bNbJ9I3LfALBWJg7nrFGslKQkcN0-i5G11EjvPsA">While metric computation and logging are essential, they are not sufficient to uncover the true performance of the model prior to deployment for reasons such as data inequality and comparison complexities (<a href="https://scale.com/blog/find-best-model" rel="noopener noreferrer" target="_blank">you can learn more here</a>). ML teams frequently tackle these challenges using Scale Validate, which provides a structured framework for testing and comparing ML models. Validate allows you to:<ol><li>Compare models on a rich set of pre-computed metrics e.g. precision, recall, confusion matrix, IoU or mAP.</li><li>Find underperforming slices through automatic clustering based on image embeddings and metric scores&nbsp;&nbsp;</li><li>Set up tests on mission-critical scenarios to track progress on metrics of choice over time and catch regressions.</li></ol>Through these strategies, Validate helps dissect obscure aggregate metrics into more granular insights and ensures that the model being shipped performs better not just on the aggregate but also on the more critical and rare long-tail situations.&nbsp;Why Labs<img src="https://lh5.googleusercontent.com/AO7znH2bIj4K__AVI3X-NGBdBqbpMhWXE_tUQ3Q-3XM_n4Xybl-1DG7vsn2BTWbsPJpn3LqQgMif9Smq-GOsl3tD5HyZjRmurRj8JgErcIVp2W1IC_ZBO54wyiJfPhg8rJfzsAaZG18PUXy7XEmDOtn-7W7lWp-XXQSm7nFzYzeDjlV7zOtwYva1PA">After a model has been deployed to production, it is crucial to keep monitoring its performance to ensure that the model performs as expected. WhyLabs is an observability platform designed to monitor ML model performance degradation. Users can create monitors to target a particular performance metric—typically, precision, recall or F1 score. Once the monitors are set, WhyLabs tracks the metrics relative to baselines set by the user. These baselines can be based on a dynamic range, like an average metric value over a historic time interval or a static fixed value. Using these monitors, users can also set corresponding actions if the value crosses certain thresholds, like alert messages. WhyLabs can be deployed in production cloud environments including AWS and GCP, or on-premises.&nbsp;Great AI applications require dependable ML models and an ML model’s dependability can’t be ascertained without foolproof evaluation. Using these tools, ML teams can build a reliable evaluation pipeline to uncover true model performance and avoid any regressions in production. <a href="https://dashboard.scale.com/nucleus/scenario-tests?utm_source=blogpost&amp;utm_medium=website&amp;utm_campaign=top_eval_tools" rel="noopener noreferrer" target="_blank">Get started with Scale Validate</a> today to uncover and fix regressions over critical scenarios.

Model evaluation is one of the most important prerequisites prior to shipping an ML model.

Top 4 Tools to Build Your ML Model Evaluation Pipeline

Any ML developer who has trained an ML model multiple times e.g. with different hyperparameters, and tried to figure out which iteration is better, is already familiar with the basics of model comparison. Before her model can deploy to production, an ML developer needs to be confident that it would perform better than the previous model. However, this is easier said than done. Comparing models with the intent of figuring out which one is better is a complex task. Most ML developers take a rather rudimentary approach, which creates a significant risk of shipping the wrong model.In this blog, we will go through the core challenges associated with model comparison and discuss strategies to mitigate them. Before we do, though let’s take a quick look at the basic approach adopted my most ML developers:<h3 class="ql-align-justify">The basic approach</h3>Let’s say we have recently trained two versions of an object detection model with a different learning rate and now want to find out which one is better. Here’s what that process looks like for most ML engineers:<ol><li class="ql-align-justify">Choose a standard aggregate metric e.g. mAP (mean average precision), <a href="https://en.wikipedia.org/wiki/F-score" rel="noopener noreferrer" target="_blank">F-1</a> score, precision or recall&nbsp;</li><li class="ql-align-justify">Apply <a href="https://towardsdatascience.com/non-maximum-suppression-nms-93ce178e177c" rel="noopener noreferrer" target="_blank">non-max suppression</a> to reduce the number of candidate predictions, ideally leaving the best one accessible and viewable to the user</li><li class="ql-align-justify">Match each of the resulting predictions with ground truth using intersection over union</li><li class="ql-align-justify">Set minimum IoU and confidence scores thresholds to obtain class confusions</li><li class="ql-align-justify">Using these confusions—True Positives (TPs) &amp; False Positives (FPs)—calculate each model’s aggregate metric&nbsp;&nbsp;</li><li class="ql-align-justify">The candidate model with the highest aggregate metric score is best and the one you’ll want to deploy&nbsp;&nbsp;</li></ol> <img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F17406fbd379046e08c2882ab8590acb4"> <img src="https://lh5.googleusercontent.com/AYGlxcGUVLCh48j-UXIyixhm5qtp4hZTrGtycJQm6ejAuV-Qx3z_ibyj_nWS-ACIEsTmLJ4hS2xBAoQy1Xgjrndgzoxud3o2wVT3gO7c4NYBrAgpq1tYASskbRH_0oZtzTTr-NHC1Bf_lS62mulaY_u3Snf2KrxtQ03wIFubYFLhbVku1hPB9sqOZA">While such a one-dimensional and aggregate comparison is helpful thanks to its simplicity, on its own it’s definitely not sufficient to identify the best model. An aggregate approach obfuscates performance over critical rare cases by assuming all data is equally important. Finding the best model is a complex multidimensional process and more often than not relies on the developer’s subjective opinion. Over the past years, we have worked closely with many ML teams on this topic and have identified strategies to systematically find the best model. It all starts with finding the right metric: <h3 class="ql-align-justify">💡Deciding on the right metric&nbsp;</h3>More often than not, ML engineers choose the standard metrics—for example mAP (referenced above) for object detection. However, metric selection is one of the most important decisions when evaluating and comparing models. The metric must capture the nature of the application and the business expectations from the model. For example, if you are working on a cancer detection problem, you would want to detect as many cancer cases as possible even if it means in a few scenarios, you classify a healthy person as having cancer. So “Recall” would be a great metric. However, if you are performing email spam detection then it’s acceptable to miss a few spam emails but not okay to classify a legit email as spam. So “Precision” would be the metric of choice. Typically, you’ll need to make a trade-off between precision and recall. In these situations, tools like the PR (precision-recall) curve or <a href="https://en.wikipedia.org/wiki/F-score" rel="noopener noreferrer" target="_blank">F-scores</a> can help better capture model performance.&nbsp; To get a feel for these metrics, try computing a sample metric on a slice of data in <a href="https://dashboard.scale.com/nucleus/scenario-tests?utm_source=blogpost&amp;utm_medium=website&amp;utm_campaign=best_model" rel="noopener noreferrer" target="_blank">Scale Validate</a>.<a href="https://scaleai.wistia.com/medias/9sp4zf58r3" rel="noopener noreferrer" target="_blank">https://scaleai.wistia.com/medias/9sp4zf58r3</a> <h3 class="ql-align-justify">🔎 Finding the optimal operating point</h3>Operating point is just another name for input controls which determine confusion calculations e.g. for object detection the operating point is a combination of minimum IoU and Confidence thresholds, if the thresholds are 0.5 then the evaluation logic will only consider objects with &gt;0.5 confidence and &gt;0.5 IoU. So you would want to ensure that these thresholds are set such that your model is giving the lowest number of confusions and the highest performance e.g. the highest F1 score. However, the extent to which you can reduce or increase these thresholds depends on your application. For example, you don’t want a low IoU threshold for object detection because that would mean large errors in location &amp; size of the detected objects. Therefore, you need to find the application constraints first and then find the optimal point for each candidate model within those constraints. To find the optimal point you can traverse over each sensible range of these input controls and look at the resulting performance metric. Once you have the optimal operating points you can do an apples to apples comparison of the aggregate scores. Try varying the IoU &amp; Confidence thresholds of this <a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?explorer_display=image&amp;display=insights&amp;faceted_open=1&amp;filtered_model_run=run_bwmaprbx3nhg040ghndg" rel="noopener noreferrer" target="_blank">sample model on Scale Nucleus</a> and see the effects on the confusion matrix and PR curve.&nbsp;<a href="https://scaleai.wistia.com/medias/x2yaorc7f0" rel="noopener noreferrer" target="_blank">https://scaleai.wistia.com/medias/x2yaorc7f0</a> <h3 class="ql-align-justify">🧮 Complexity of multiple metrics</h3>In most instances it is hard to capture model performance with a single metric. For example, for obstacle detection you don’t just want to detect an object but also be very accurate with the location and size. Therefore, on top of an accuracy metric such as recall, you would also want to track average IoU to check for the tightness of the predictions being produced. The challenge of doing model comparison with multiple metrics is that no model might perform the best on all metrics. To tackle this problem, the simplest method is to do a weighted aggregation of these metrics and obtain a single score. This is exactly what the F-scores do for the metrics of precision &amp; recall. If you want to assign equal importance to both metrics you can compute F1 score, otherwise you can use F0.5 or F2 scores to weigh one metric more than the other. You can read more about F-beta scores <a href="https://machinelearningmastery.com/fbeta-measure-for-machine-learning/" rel="noopener noreferrer" target="_blank">here</a>. Try <a href="https://dashboard.scale.com/nucleus/model-comparison?analyze_dataset_id=ds_bwkezj6g5c4g05gqp1eg&amp;analyze_base_model_id=run_camszmp8nd200ch71170&amp;analyze_compare_model_id=run_cabt1b5e3xf005r21sxg" rel="noopener noreferrer" target="_blank">model comparison over multiple metrics</a> on Scale Validate.<a href="https://scaleai.wistia.com/medias/g5ie6sy2bl" rel="noopener noreferrer" target="_blank">https://scaleai.wistia.com/medias/g5ie6sy2bl</a> <h3 class="ql-align-justify">📊 Dealing with data inequality&nbsp;</h3>Real world data is never equally distributed or equally relevant. Some subsets or slices are just more important than others for the model in question. And more often than not these high importance subsets are underrepresented and hence small in size. The way aggregate metrics are calculated, they assume that all data is equally important and obscure performance on rare important samples. For example, a pedestrian detection model can have an mAP of 0.9. But that 0.9 can be very misleading if the examples it fails on are the ones where pedestrians are on a crosswalk! Similarly when comparing two models, the aggregate performance might go up but that doesn’t mean that the new model is better. To address this it is important to first identify these longtail situations and curate small subsets which act as data unit tests similar to unit tests for software. Then models can be compared with regards to how well they perform over these unit tests in addition to the aggregate metric. For example, model B is only better than model A if it improves on the aggregate and doesn’t regress on any of the data unit tests. Read more about data unit tests <a href="https://scale.com/blog/testing-models-validate" rel="noopener noreferrer" target="_blank">here</a> and try to compare models on specific unit tests in Scale Validate.<a href="https://scaleai.wistia.com/medias/0kekfm8stb" rel="noopener noreferrer" target="_blank">https://scaleai.wistia.com/medias/0kekfm8stb</a> <h3 class="ql-align-justify">To summarize:&nbsp;&nbsp;</h3><ol><li class="ql-align-justify">Choose metrics which are representative for your application</li><li class="ql-align-justify">Carefully analyze and define thresholds in your models</li><li class="ql-align-justify">Ensure that critical subsets of data have an impact on the evaluation score</li></ol>We have built Scale Validate with these strategies in mind. You can try it for free on your data or an open source dataset <a href="http://scale.com/validate?utm=n" rel="noopener noreferrer" target="_blank">here</a>.

 Model comparison can be hard—here is how you can find your best model!

Finding Your Best Model Is Tricky—Here's How to Do It!

If code quality is what sets apart good software systems from bad ones, data quality is what distinguishes great AI systems from the ones that will simply never make it to production. While most of the research in machine learning (ML) continues to focus on models, industrial players building AI systems are focusing more and more on the data. The number of companies adopting a <a href="https://datacentricai.org/" rel="noopener noreferrer" target="_blank">data-centric approach to AI</a> development (understood as systematically engineering datasets used for AI) is growing rapidly across industries and levels of company maturity. According to former Tesla Autopilot lead Andrej Karpathy, engineers at Tesla lose sleep over the dual challenges of structuring their datasets effectively and handling the edge cases that might come up.<img src="https://lh4.googleusercontent.com/bGi4IgNBP_6iLqhMvrAwXJFXRzwRHiQt5iMhU1jvUgHC44K-eJFb5uO_tPueneP6pZu9fkxsWnWUsMhITkK6I2acFC52sC3K_Kzhhk0ogXqNPFHfZD5V7HDx1bQhliZIX-UYR_KKBZjNlkTlswb28I0">Graphic adapted from Andrej Karpathy’s slide deck [Train AI 2018]But why is using the right data so central for building out high-performing ML models? The answer lies in unequal data distribution and low-quality ground truth (annotations).<a href="https://dashboard.scale.com/nucleus" rel="noopener noreferrer" target="_blank">Try Scale Nucleus</a><h3>Not all data is created equal</h3>The core hypothesis behind data-centric AI is that <a href="https://scale.com/blog/improve-data-annotation" rel="noopener noreferrer" target="_blank">you can achieve better ML model performance by training your model on higher-quality data</a>. In an ideal world, your model performs equally well in all kinds of situations. For example, in autonomous driving, you would want a model detecting pedestrians to work equally well, irrespective of the weather, visual conditions, how fast the pedestrian is moving, how occluded they are, et cetera. Most likely however, your model will perform much worse on cases that are more rare—for example, a baby stroller swiftly emerging from behind a parked car in inclement weather.&nbsp;The point of failure here is that the model has been trained on data that was recorded during regular traffic conditions. As a result, the representation of these rare scenarios (as a portion of the entire training dataset) is much lower compared to common scenarios. Below is another example of two highway scenarios, whereas lane detection will be significantly more difficult in the right hand picture compared to the left.<img src="https://lh5.googleusercontent.com/Nr-B0CIcVGPiYOS9BRizyV96w9k_tLKcdLpaDpYzoZeTfDB2LOY5F5--BKhH70lyDC9ZBvqxoL2ks2x1C_qIfoO6cBaScI-SkyFTGUCv3GU5NjkT8GpsF788mQpm-bl1yqnVMabnW8umPNOajNgySlE">What can we do about this? We need to increase the prevalence of these rare cases in our labeled training data.<img src="https://lh4.googleusercontent.com/8VXzOph0ZPD3peHbzOCCOgX1Ecx3TbroIKlFlD0-il7MTqFkLVHz7HBwRaYwAINnOMzTZnussN0dS-ivxzk4Flb5VX37a-uAvlXhMuuVKXHBpvakXAkUFkMjQuW_lmxsdpwynICGXNNz_pVCPtR6bX0">Accuracy improvements with data selection via active learning versus random sampling given a fixed labeling budget.Source: <a href="https://arxiv.org/abs/2111.12880" rel="noopener noreferrer" target="_blank">Emam et. al., 2021. Active Learning at the ImageNet Scale</a>In ML research, this approach is often referred to as active learning. Many papers (e.g. <a href="https://arxiv.org/abs/2111.12880" rel="noopener noreferrer" target="_blank">Active Learning at ImageNet Scale</a>) have demonstrated the effectiveness of this approach, showing significant performance improvements by picking more difficult (most often underrepresented) training data instead of random sampling. At Scale AI, this is also what we observe with our customers, especially those with advanced ML teams and models deployed in production. Companies that are the best at picking the right data for labeling—and eventually training—have the highest performing models. Unfortunately, identifying the highest impact data is difficult, and most ML practitioners perform this task manually, even when wrangling very large datasets.<h3>Labeling errors are breaking your model</h3>To make things worse, almost all datasets include labeling errors. Labeling errors are more likely to occur on edge cases. This makes sense, because images that have been seen 1,000 times are easier to label correctly compared to an image seen only once. Labeling errors are also most destructive for your model performance if they happen on edge cases, because you have a limited amount of these samples in the first place. So if the labels on these difficult samples are wrong, model performance will never improve. Instead, it’s more likely to quickly regress.<img src="https://lh4.googleusercontent.com/rFWziHErcB0S1tlTzQQDw2OWyhFgaCHQJumi-rYeMg0Wtmql4_Xr31QXo8bZrBo_xt_Nhk5_XflX7BEW4cOaJmoqS0bVcrPviFCzeKQC8wEAQbytWbb-FsJghbtm0tnqREVjq_NJdoplyM3u6_UEFxs">Example of a typical l labeling error on the public Berkeley Deep Drive dataset: class confusion between car (the annotation)&nbsp;and bus (the prediction).&nbsp;As a result, finding and cleaning bad labels (generally, but especially for rare cases) is probably one of the most impactful activities you can do to meaningfully improve model performance. Many teams do this already. Unfortunately, though, doing this consistently, especially on large datasets while keeping track of issues and measuring progress, is often a very manual and isolated process. Many times, teams will use Jupyter notebooks to find bad labels and keep track of them in spreadsheets.Doing things manually will only work for so long, and it will not get your model into production anytime soon. To address this we have built Scale Nucleus, a data management tool that helps to decide what data to label and make sure that these labels are consistently of high quality.In the rest of this article, we’ll walk through common strategies to mitigate the problems outlined above, starting with unequal data distribution.<h3>Use similarity search to get from one failure case to many</h3>One of the most common ways to build good training data is to look at a single sample that is particularly difficult for the model (e.g. based on pure luck, a field report, domain knowledge or other data points) and try to find more of the same.<img src="https://lh3.googleusercontent.com/JKZZC8hjToji6ebsTT9Tm6iAk-nY3gX-BPs4j-Ojc76bp_J3BXcT-kU1vZJ9njE2f0KYt3opfz9n8cQW2bf5_DMRB40XtXkvBfRbkw7P6Y-tVpKTCU4mz7-Z9jFFh5soMJ6t79KOoyxZDMzojbi_FRk">In Nucleus, you can do this by selecting a few images of interest in your dataset and running a similarity search. For example, you might know that perception performance is bad on blurry images. In this case, you’d select a single blurry image and run a similarity search.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F3c1101b17e1d4d95a3030dec3104f835">However, a single similarity search might yield mixed results; not everything is equally relevant. Furthermore, you might want to codify the issue you were looking for into a “tag” and consistently find all images that have this issue. In our example, you might want to make sure every image in your dataset can be identified as blurry or not blurry going forward.One way to address this is using a Nucleus Autotag. Autotag is a way of teaching Nucleus about a new concept, by interactively selecting more and more specific examples of a case, for example blurry images, in a few refinement steps. In just a few minutes, the platform will “learn” a concept like blurriness and enable you to search your data using this attribute going forward. In the background, we are training a binary classifier, which assigns and updates a similarity score for the concept you are searching for across your whole dataset. You can learn <a href="https://nucleus.scale.com/docs/autotag" rel="noopener noreferrer" target="_blank">more about Autotag here</a>.<a href="https://scaleai.wistia.com/medias/gd6yfivwwk" rel="noopener noreferrer" target="_blank">Here</a><h3>Find rare edge cases with natural language search</h3>Many times, an ML engineer might know (e.g. based on domain expertise) what kind of edge case data is difficult for their model, but does not immediately have an example of this at hand. Edge cases are often also environment conditions or situations, for which you have neither annotations and nor does your model predict them—yet. Examples might be specific lighting conditions, occlusions, or novel objects types.&nbsp;<img src="https://lh6.googleusercontent.com/ZVoRKE_yYTYVkKMTLeXN1vuh-3ExukAYnhTIPL9vtW55Bee0ePoKU5tycAWXJwytidezHuu_f5jvbInP5-P2GLYlqUPmTS-WM0ocfqEVwfJr-G1lNSmUIIHtPhwYnu3f4ke3tGXqoutxCCkXzBOhPmw">To start, you might run a Natural Language Query to find these examples. For example, you might want to find <a href="https://dashboard.scale.com/nucleus/ds_bwk18y3e696g05rk8gyg?explorer_display=image&amp;display=similarity_text&amp;similar_text_query=a+photo+of+a+pizza+box" rel="noopener noreferrer" target="_blank">all frames of pizza boxes</a> in the famous ImageNet dataset. It is important to note that this style of query works on data without any annotations, model predictions or metadata because the samples you are trying to find are edge cases which are not yet detected by your computer vision model.One way of solving the “no labels challenge” is to use precomputed embeddings that are matched against a natural language query, e.g. using OpenAI’s <a href="https://github.com/openai/CLIP" rel="noopener noreferrer" target="_blank">CLIP</a> model. Nucleus generates CLIP embeddings automatically for all images in order to support natural language search.<a href="https://scaleai.wistia.com/medias/uan3x6c3ia" rel="noopener noreferrer" target="_blank">Natural Language Search</a>Once you’ve built a more balanced dataset and enriched it with annotations, now you’re ready to ensure these labels are consistently of high quality, especially for edge cases.<h3>Review computer vision labels</h3>Now that you have a set of image annotations, you’ll want to verify that they are of high quality before using them as training data, especially for rare edge cases. This can either be done image by image, or in batches as long as the data annotations can be easily inspected and compared against the actual image content.<img src="https://lh3.googleusercontent.com/AdVeuzXqgQzQKvjjMaP4j2Dco5iGD7CLrs2FXOriHzr0I48c4J732dkeZ6Ynx_TAPlculOjwyWnWKB0HinRuzaXG--Ays5RULi1ptcMF8DmPqxKZodGnFc-gDnx98xNsZjfyzFM995-yG2wMUZkPcWM"><img src="https://lh4.googleusercontent.com/GYD7xBOcas6YKhtCKoHZWwxib2KGMRJXXRN-X_hwGzek_UO37C_gQiatduhSZKO5sKXPd09li3pDYA64kMqeN3viU8IjhYCtMq5AkNsbK-xOg7AnN5gd9KSdQqy0XF5EhGtoN_yCrxt8gOl-QYRtUwA">To improve label quality in the next iteration of labeling, it makes sense to mark each individual frame and bounding box as accepted or rejected and later query against these attributes to make sure all items are reviewed. If you encounter a particular issue, such as a badly fitted bounding box, a good option is to add the relevant images to a “slice”, which is a curated subset of data to share with colleagues or to return to at a later point.<img src="https://lh4.googleusercontent.com/UtXwA46VnC7onh-V4MrgPgF8cFHJZ3wxDeL4NJEb8PGoX8atpZsRSRMR9uIN29g5W4uVrjGxHve6mZHk734ZBAQpvY86X-nE9JaKN832GKyY3d4_oUubJR4L-DkVF5997tBqog57l1aNYKDfPTnAu7g"><h3>Find mistakes in computer vision labels</h3>Instead of going through all your data manually, it is often advisable to compare the labels you are reviewing against the predictions of your most recent computer vision model. There are several so-called model-assisted methods to find and clean labels such as sort-by-loss, sort-by-confidence where model and ground truth disagree, or confident learning (<a href="https://jair.org/index.php/jair/article/view/12125" rel="noopener noreferrer" target="_blank">paper</a>). It is in fact often surprising how effective methods such as sort-by-confidence are to improve label quality. ML practitioners do this by computing common error metrics such as the aggregated IOU, a confusion matrix, or a precision-recall (PR) curve (e.g. in a Jupyter notebook) and then inspecting the samples where the prediction of the model and the data annotation are least aligned.<img src="https://lh3.googleusercontent.com/me0p6swzqwP1_AN4g8O4BbGBuKNb0LqJLgqr240I39VTN2a-jaHLYSC-VUn5fJAFkhf6z9ClYSg9OKZs5TC-UEJzEA1tzg5IwMoiH-2Auf0ECZOJhUwxT-uIIhgGh0TNaWfkXHW0KsI2A6hrv7D8vmo">However, using a GUI this process can be 10x faster, because error metrics are pre-computed automatically and can be directly explored with interactive charts. Clicking into any graph automatically queries and surfaces all the samples associated with that data point. For example, clicking into the cell for “bus versus truck” confusions in the matrix displayed below queries your dataset for all samples where the data label is a truck but the computer vision model predicts a bus.<img src="https://lh4.googleusercontent.com/9GiOAPH8IoxI6QiMfYa7Ia7rOPcvwDTZuuMaDkPTG7-176VwjOM5cjZmLUGNXEBvwtpCclB3Ryp3e8vQV6BHMpEagh14TbEc_L90i7vuKb7KX04Xkvxt_Pjap7Ta7LShMBKcRzYfVDG35ecCdaEdjJE"><img src="https://lh4.googleusercontent.com/XFAtHxplLOV_1YHFQJv6Bb80wU2qce2hu-20b-D-BzkIIJ0n8xl6iNAgdxMuUse3ckJdEcbGUEEMzC1qhyNAdzls6lFQiU8P6JkPLZd1KxuRY_Iwuy6zS_ZQZMvVvtfMUZqeQX-6RI-A6-sbYMWVXrE">From just a single glance, we can see that these are buses that are incorrectly labeled as trucks. These image annotations need to be corrected to avoid bad model performance for both bus and truck detections.<h3>Find missing labels</h3>Finally, it is critical to ensure that no labels are missing in your annotated image data. A missing label can lead to a performance regression of your model, especially if the object with the missing label is easily detected by your model.<img src="https://lh4.googleusercontent.com/IlPV_qOBq4bmtcTAEP119G257XAKc3W5titad6vm2VF0rZ25lIGQhALm2PcLwlB1QUS8K3bTKf17Lcqf24zed-f-mPhLtPEJqk9V9m-eF6Bnz6D7ueyoeSqs10ra1YO_PHqXwFmtf9LO30pJbSQul2Y">In the above example, all cars are annotated in the image, but the single white vehicle is just clearly omitted from any detections. A perception model for vehicles will easily detect this car. However, if it is re-trained with the shown bounding boxes as ground truth, it learns to disregard white cars, which would likely be a dangerous failure mode.<img src="https://lh5.googleusercontent.com/4B7S6d1FE4cK_UP14Ma9iDx2wd7Rq2M1yHUInc7UqZKW6VXOoNUFmr_FNlsZFTiwSPlBPa-k21oGgGEegKF6AM3X5X5QZcjzpT__NNMMhLXw71IWjNrhIT8kJamypgptKEC2308JFeNFuUYpRkOi9P4">The confusion matrix is again a good starting point to find these cases and make sure they are corrected. We can find the row with “no ground truth” in the bottom and query for examples where the model is currently still predicting an object.<img src="https://lh4.googleusercontent.com/F5tkfSRWH0rvhyDN4okzsTBgsse8qDz9KwiNe6FD-5IFbDgactKLJyEPDTZpllDriNkf4ghppUwDV6rhkU826ijXHLRQqaKsxwJ7pFY4BY2Hkda5VN2mwRJC8Ko4Ow0s0uyA15jENUHUwK-xEC2kkfE"><img src="https://lh3.googleusercontent.com/vexNOiYxXuY_0_P8Ani_ao266gUZbJFRaYxSNsuisQbZbYVgbRzlP93fXzdB7J9jREzqYnGdBAEXZ9xWaoZcNdiKfxFf9AMDIXYks2TDL1e_8FgzlBh8YXTOsqeylMdR2CgrhhQB6VcTHWjlshkNmNY">With this query, we can swiftly find all cases that are obvious misses and make sure that the data annotations are corrected for these images.<h3>Dealing with long tail data distributions: the TL;DR</h3>We talked about two common problems when training high-performance computer vision models: unequally distributed data with rare edge cases and bad image annotations.Both research and industry agree that picking out the right data to annotate leads to much better model performance, because it corrects for the naturally unequal data distribution encountered in the real world. The model can learn how to deal with edge cases by being trained on more data that contains them. Secondly, almost all datasets contain annotation errors, while computer vision models work on the assumption that all annotations are correct. This causes the models to perform worse, in practice. This problem is most pronounced for rare edge cases, as these are harder to annotate and they have a worse impact if they are annotated incorrectly.There are proven methods to cope with these problems. Good strategies to pick high quality data include starting from difficult images and finding more of the same or running semantic search for situations that are known to be problematic. Second, we have learned that reviewing the quality of data annotations is essential. Using error metrics that compare annotations with current model predictions (e.g. with a confusion matrix) is a good way to quickly surface the lowest quality data labels or even missing annotations.Getting the edge cases right is considered to be the “holy grail” of building high performance computer vision models, and with the right techniques, you don’t have to lose sleep over them any longer.<h3>Ready to try Nucleus yourself?</h3> <ul><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?explorer_display=image&amp;display=similarity_text&amp;similar_text_query=a+photo+of+a+woman+with+a+baby+stroller" rel="noopener noreferrer" target="_blank">Find edge cases from scratch with Natural Language Search</a></li><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?selected_items=di_bwkhpa19xvgg05rxd73g&amp;item_grid_scroll_pos=785.5556030273438&amp;explorer_display=image&amp;display=similarity_image" rel="noopener noreferrer" target="_blank">Find similar images from a single sample</a></li><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?selected_items=di_bwkhpa19xvgg05rxd73g&amp;item_grid_scroll_pos=785.5556030273438&amp;explorer_display=patch&amp;q=annotations.count+%3E+0+&amp;faceted_open=1&amp;filtered_model_run=run_bwmaprbx3nhg040ghndg&amp;selected_model_runs=run_bwmaprbx3nhg040ghndg&amp;prediction_labels=bus&amp;order=Desc&amp;metric=nucleus_confidence&amp;annotation_labels=truck" rel="noopener noreferrer" target="_blank">Find mistakes in computer vision labels (i.e. class confusions)</a></li><li><a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?selected_items=di_bwkhpa19xvgg05rxd73g&amp;item_grid_scroll_pos=785.5556030273438&amp;explorer_display=patch&amp;q=annotations.count+%3E+0+&amp;faceted_open=1&amp;filtered_model_run=run_bwmaprbx3nhg040ghndg&amp;explorer_patch_display=false_positive&amp;iou_extent=0%2C0&amp;selected_model_runs=run_bwmaprbx3nhg040ghndg&amp;prediction_labels=car&amp;order=Desc&amp;metric=nucleus_confidence" rel="noopener noreferrer" target="_blank">Find missing image annotations</a></li></ul>

 A guide for ML engineers losing sleep over edge cases

Don't lose sleep over long tail data distributions.

How to Handle Long Tail Data Distributions in Machine Learning

Nucleus is the centralized, scalable, and collaborative control center for your machine learning (ML) model’s training data. Nucleus helps you update missing or erroneous labels, and identify model failure cases so that you can direct your data curation efforts with the express goal of training better models.With simple integrations for model integration, testing, validation, and deployment, Nucleus can serve as your one stop shop to continuously iterate on your dataset for superior model quality, even in the face of new edge cases, model drift, and unexpected changes in environmental variables. Nucleus helps you tackle the long tail, the one certain path to training models that can address new scenarios and eliminate failure modes.Read on to learn the essentials—how Nucleus can help you manage your dataset efficiently.<h2>The basics:</h2><h3>Addressing model failures:</h3><ul><li>Where is my model currently failing and how can I fix these issues with the next round of labeling data?</li></ul>Once you’ve uploaded ground truth labels and at least one set of predictions for your dataset, you can click into the “Objects” view in Nucleus to identify False Positives, as shown in the below video. You can also identify other simple failure modes like False Negatives or low Intersection-over-Union (IoU) scores for image data.<a href="https://scaleai.wistia.com/medias/511bisntyb" rel="noopener noreferrer" target="_blank">this video</a><h3>Addressing label quality issues:</h3><ul><li>How can I easily check and improve the quality of existing labels?</li></ul>With your ground truth labels already in Nucleus along with one set of model predictions, the IoU slider and the “sort by” drop down menu lets you sort classifications by confidence from low to high. You can then choose if you want to label or fix the lowest-confidence images, with Rapid or if you’re an enterprise customer, your labeling pipeline.<a href="https://scaleai.wistia.com/medias/bgmcf03b9u" rel="noopener noreferrer" target="_blank">this video</a><h2>Breaking down your data in new ways:</h2><h3>Finding images without labels:</h3><ul><li>How can I find images with certain properties for which no ground truth label is available?</li></ul>If no label exists for the query you’ve conceptualized, you can search your dataset with simple, natural language text. Using CLIP embeddings under the hood, Natural Language Search lets you search for images with English phrases, instead of the structured query language you use for labels and metadata.&nbsp;<a href="https://scaleai.wistia.com/medias/uan3x6c3ia" rel="noopener noreferrer" target="_blank">this video</a><h3>Finding similar images:</h3><ul><li>I’ve identified an example image: how can I find all similar images in my dataset?</li></ul>If no label is present for a class of images, but you’ve found a sample image, you can use Object or Image Autotag to build up a Slice of your dataset that consists of similar images. A step-by-step process will show you how to provide positive and negative examples, in multiple rounds, to fine-tune your search for like images. When you proceed to query your Autotag, you can set a threshold to broaden or narrow the scope of your search.<a href="https://scaleai.wistia.com/medias/uuavyfeapq" rel="noopener noreferrer" target="_blank">this video</a><h2>Debug your models:</h2><h3>Find classification failures:</h3><ul><li>How can I find examples of classes in which my model and my labels disagree?</li></ul>In Objects mode, once you select a model (predictions that you uploaded, or generated from a model in our Model Zoo), you can find a Confusion Matrix on the Charts page. This grid lets you find color-highlighted cases in which the labels and model disagree on a classification. You can click on any box to filter your entire dataset on just those examples, to further explore where your model fails and where it succeeds.<a href="https://scaleai.wistia.com/medias/ic8y27jymv" rel="noopener noreferrer" target="_blank">this video</a><h3>Sorting and filtering both failures and successes:</h3><ul><li>How can I find the best and worst classification examples?</li></ul>In Objects mode, once you select a model (predictions that you uploaded, or generated from a model in our Model Zoo), you can sort all of your data objects by confidence or intersection-over-union (IOU), or bound your images with specific minimum and maximum thresholds for these score values. You can also filter by specific types of confusions such as False Positives and False Negatives.<a href="https://scaleai.wistia.com/medias/c2f14217wn" rel="noopener noreferrer" target="_blank">this video</a><h3>Avoiding regressions:</h3><ul><li>How can I ensure I'm not breaking anything that worked with a previous model or dataset?</li></ul>As you improve and expand your dataset, then proceed to train new models, you’ll want to ensure that your new models don’t fail on cases that your old models solved successfully. Through Scale Validate, Nucleus helps you track metrics such as mean average precision (mAP), precision, and recall on specific scenarios in your dataset. Or, you can establish and write your own metrics to track over time. This process ensures that every label addition and dataset modification you make only yields improving outcomes rather than regressions. You can think of this as a continuous integration (CI) or regression detection process for your models.<a href="https://scaleai.wistia.com/medias/ak5lqayith" rel="noopener noreferrer" target="_blank">this video</a> Ready to try Nucleus? <a href="https://dashboard.scale.com/nucleus" rel="noopener noreferrer" target="_blank">Click Here</a> to set up your first project.

Nucleus facilitates data curation and dataset management.

Armaghan Khan

Top 4 Tools to Build Your ML Model Evaluation Pipeline

Finding Your Best Model Is Tricky—Here's How to Do It!

How to Handle Long Tail Data Distributions in Machine Learning

Nucleus Simplified—Dataset Management for Everyone