User permissions required: 'View Sources' AND 'Review and label'
Please Note: Previously, the 'Teach' function, when filtered to reviewed verbatims, would show verbatims where the platform thought that the selected label may have either been misapplied, or missed. 'Check label' and 'Missed label' split these into two separate views, with 'Check label' displaying verbatims with the label potentially misapplied, and 'Missed label' showing verbatims that may be missing the selected label.
Introduction to using 'Check label' and 'Missed label'
Using the 'Check label' and 'Missed label' training modes is the part of the Refine phase where you try to identify any inconsistencies or missed labels in the verbatims that have already been reviewed. This is different to the 'Teach label' step, which focuses on unreviewed verbatims that have predictions made by the platform, rather than assigned labels.
'Check label' shows you verbatims where the platform thinks the selected label may have been misapplied - i.e. it potentially should not have been applied.
'Check label' mode with the 'Claim > Reason for Loss > Fire' label selected
'Missed label' shows you verbatims that the platform thinks may be missing the selected label - i.e. it potentially should have been applied but wasn't. Here the selected label will typically appear as a suggestion, as shown in the image below.
'Missed label' mode with the 'Payment > Error' label selected
The suggestions from the platform in either mode are not necessarily correct, these are just the instances where the platform is unsure based on the training that's been completed so far. You can choose to ignore them if you disagree with the platform's suggestions after reviewing them.
Using these training modes is a very effective way of finding occurrences where the user may have not been consistent in applying labels. By using them you are able to correct these occasions and therefore improve the performance of the label.
When to use 'Check label' and 'Missed label'?
The simplest answer of when to use either training mode is when they are one of the 'Recommended actions' in the Model Rating section or specific label view in the Validation page (see here).
As a rule of thumb, any label that has a significant number of pinned examples but has low average precision (which can be indicated by red label warnings in the Validation page or in the label filter bars) will likely benefit from some corrective training in either 'Check label' and 'Missed label' mode.
When validating the performance of a model, the platform will determine whether it thinks a label has often been applied incorrectly, or where it thinks it's been regularly missed, and will prioritise whichever corrective action it thinks would be most beneficial for improving a label's performance.
'Missed label' is also a very useful tool if you've added a new label to an existing taxonomy with lots of reviewed examples. Once you've provided some initial examples for the new label concept, 'Missed label' can quickly help you identify any examples in the previously reviewed verbatims where it should also apply. See here for more detail.
How to use 'Check label' and 'Missed label':
To reach either of these training modes, there are two main options:
- If it is a recommended action in Validation for a label, the action card acts as a link that takes you directly to that training mode for the selected label
- Alternatively, you can select either training mode from the dropdown menu at the top of the page in Explore, and then select a label to sort by (see image above for example)
Please Note: You must first select a label before either 'Check label' or 'Missed label' will appear in the dropdown menu. Both of these modes also disable the ability to filter between reviewed and unreviewed verbatims, as they are exclusively for reviewed verbatims)
In each mode, the platform will show you up to 20 examples per page of reviewed verbatims where it thinks the selected label may have been applied incorrectly ('Check label') or may be missing the selected label ('Missed label').
'Check label'
In 'Check label' review each of the examples on the page to confirm that they are genuine examples of the selected label. If they are, move on without taking action. If they are not, remove the label (by clicking the 'X' when hovering over it) and ensure you apply the correct label(s) instead.
Review as many pages of reviewed verbatims as necessary to identify any inconsistencies in the reviewed set and improve the model's understanding of the label. Correcting labels added in error can have a major impact on the performance of a label, by ensuring that the model has correct and consistent examples from which to make predictions for that label.
'Missed label'
In 'Missed label' review each of the examples on the page to see whether the selected label has in fact been missed. If it has, click the label suggestion (as shown in the image above) to apply the label. If it has not, ignore the suggestion and move on.
Just because the platform is 'suggesting' a label on a reviewed verbatim, does not mean the model considers it to be a prediction, nor will it count towards any statistics on the number of labels in a dataset. If a suggestion is wrong, you can just ignore it.
Review as many pages of reviewed verbatims as necessary to identify any examples in the reviewed set that should have the selected label but do not. Partially labelled verbatims can be very detrimental to the model's ability to predict a label, as when you do not apply a label to a verbatim, you essentially tell the model 'this is not an example of this label concept'. If it is in fact a correct example, this can be very confusing for the model, particularly if there are other very similar examples that DO have the label applied.
Adding labels that have been missed can therefore have a major impact on the performance of a label, by ensuring that the model has correct and consistent examples from which to make predictions for that label.
Once the model has had time to retrain after your corrective training in these modes, you can check back in Validation to see the positive impact your actions have had on the Model Rating and the performance of the specific labels you've trained.
Previous: Why might a label have low average precision? | Next: Training using 'Teach label' (Refine)