This work has proposed a novel data engine for binary classification tasks. The method uses two different verification methods — "Expensive Verification" and "Model Prediction" — to obtain both high reliability and fast run-time. By using the Wilson-Score method, confidence bounds on the model prediction can be obtained. By selecting the expensive verification if the confidence is out of the set confidence bounds, the error-rate of the system is controlled.
Related Work
First we give an overview of methods for prediction of failures in systems similar to ours. Then methods for calculating confidence in binary classification are reviewed and as we use dimensionality reduction to pre-process the data, we report on related work in these areas.
Failure Prediction in Assembly Processes
As disruptions in manufacturing are very costly, detection of failures has been studied extensively. One approach to avoid faults is to predict anomalies using sensor data.
A method similar to ours is presented, where also force data is recorded during an insertion operation and number of classifiers for error prediction are tested. They show good classification results on the UCI benchmark dataset and a Naive Bayesian classifier is found to work best. However, they don't use any kind of confidence bounds for the classifiers.
Decision trees have also been used for the classification of errors using force data. Similar to the aforementioned method, the approach is tested on the UCI benchmark dataset. We do not use the UCI benchmark dataset as the sampling rate and dataset size are much smaller than in our approach.
These methods all rely on supervised learning of collected data. Thus an experimental set-up separate from the running assembly system is required to collect the training data which is a costly procedure.
Another solution is to collect data when the system fails. But, in many cases this is very costly as it will lead to many system failures. However, this can be added to the system as an additional feature. Thus even though faults are to be avoided, if they occur the recorded data can be used to improve the system. We add this as an additional feature to our system, where any system failures are logged and then labeled to improve the system.
By using the Wilson-Score method for classification our system is able to provide a bound on the maximum frequency of failures allowing for a more reliable process.
Confidence Bounds for Binary Classification
There exists several different methods for quantifying the uncertainty in binary classification. As mentioned, the k-NN classifier provides a simple but also inaccurate way to estimate the confidence, since it does not take the sampling uncertainty of the neighboring points into consideration.
A more principled method for uncertainty estimation is Gaussian Process Classification. While this methodology performs well, it is a computationally expensive method, so approximate methods are required for large data sets, such as those that will be encountered in a running system.
Dimensionality Reduction
Dimensionality reduction techniques offer several advantages when dealing with high dimensional data, including the removal of insignificant or redundant information, reduced training time, improved model performance, mitigation of overfitting, easier data visualization, and enhanced prediction accuracy, as demonstrated by prior work, where the authors explored how these techniques could affect force estimation in robotic assisted surgery using neural networks.
Recent and popular dimensionality reduction techniques include Uniform Manifold Approximation and Projection (UMAP), a manifold learning technique which preserves local neighborhood relationships while also maintaining, to some extent, global structure, and has been shown to outperform several other nonlinear dimensionality reduction methods in these aspects. For our work, we adopt UMAP, as it is particularly effective for visualizing and interpreting large, high-dimensional datasets.
Method
The core of our method is an online data engine, which is able to use both a machine learning model and an expensive verification method, for an insertion quality classification use case. This allows the system to minimize incorrect classifications while maintaining efficiency.
The following section depicts in detail the individual components of the system. We begin with the data collection process ("Insert into the Fixture" and "Obtain Data"), followed by the dimensionality reduction technique ("UMAP"). We then perform the adopted machine learning model ("Radius Neighbors Classification") and apply the "Wilson Score" method for computing confidence bounds.
Finally, we show how these components are integrated into the complete data engine (below "Model Prediction"): when the machine learning model is still too uncertain to perform a prediction (i.e., the prediction is not within the required confidence bounds), the "Expensive Verification" is relied upon.
The purpose of the expensive verification step is to ensure the correct labeling of the robotic operation. In this work, it is realized as a single operation named height check and described in more details in the following section.
Each time the expensive verification is used, the system collects training data to further improve the machine learning model ("Store in Database"). Depending on the result of the prediction, the system executes accordingly and generates further data for improving the model. The confidence bounds of the system guarantee a limit on the failure rate while reducing the number of required expensive verifications.
In the sections below, we now describe these steps in detail.
Data Collection
In general, the expensive verification can be a series of operations, but they need to provide with absolute certainty the perfect label.
Dimensionality Reduction
We first reduce the high-dimensional feature space using UMAP to obtain a two-dimensional representation.
Firstly, UMAP constructs a high-dimensional neighbor graph that captures local structure by connecting each point to its nearest neighbors according to the chosen distance metric. This graph is then converted into a fuzzy topological representation. Subsequently, UMAP performs a low-dimensional embedding of the data, which involves mapping the data from its original high-dimensional space to a lower-dimensional space, while preserving the neighborhood relationships and global structure. The figure above shows a visualization example of our training data at the end of an experiment.
Data Engine
After the Wilson score outputs the confidence bounds, we then evaluate the lower bound value. If it exceeds the chosen threshold T, it indicates that the UMAP transformation places the sample in a region densely populated by similar points, allowing the model to predict its class with greater confidence.
In case of a prediction of an incorrect insertion, the object is returned to the bin, and the system proceeds to the next sample. Otherwise, we proceed with the standard sequence of robot actions. If a problem arises during this step (in our case, a system stop), we know that the classification previously made resulted in a false positive, leading to a system stop where the user needs to interfere to restart the process. Consequently, we can save this sample into the database, taking into account that it translates into additional runtime.
After periodic insertion iterations, the system's model can be retrained using the growing database consisting of samples for which the model exhibited low confidence, as well as false positive samples obtained from the "Store in Database" component.
The data engine has been tested only on CPU, and training time has been measured to be less than a full robotic cycle time, which is on average of approximately 80 seconds with the current dataset size.
Results: Wilson Score
For a fixed alpha and different threshold T, precision is consistently above the threshold. The threshold T directly influences the precision lower bound: higher threshold yields higher precision with fewer false positives, but at the cost of a reduced number of overall predictions and by that higher run-times.
Instead, alpha determines when the model begins to predict by directly influencing the Wilson Score bounds. Lower values of alpha delay predictions by reducing the lower bound, prompting the engine to accumulate more data before issuing confident predictions.
Ablation Study: Baseline Results Using Binomial Interval Score
We compare our results with the baseline method using Binomial Interval score. An example result can be seen in the figure below, and the results of different tested configurations are listed.
When comparing the two methods, the Wilson Score approach tends to reduce reliance on the expensive verification as confidence accumulates. This results in fewer false positives and a higher percentage of certain predictions.
Conclusion
This work has proposed a novel data engine for binary classification tasks. The method uses two different verification methods — "Expensive Verification" and "Model Prediction" — to obtain both high reliability and fast run-time. By using the Wilson-Score method, confidence bounds on the model prediction can be obtained. By selecting the expensive verification if the confidence is out of the set confidence bounds, the error-rate of the system is controlled.
Future Work
In future research, we plan to extend our approach to tasks with variation in parts, fixture, materials, and other domains that are concerned with similar problems to demonstrate the generalizability of the data engine. As long as the input data for classification is clearly defined and the expensive verification component provides reliable and accurate labeling, the proposed data engine can be systematically applied.
Frequently Asked Questions
What is the main contribution of this paper? The paper proposes a novel data engine for binary classification in robotic insertion tasks that combines a machine learning model with an expensive verification method, using Wilson-Score confidence bounds to control error rates while minimizing expensive verifications.
How does the Wilson-Score method improve classification reliability? The Wilson-Score method provides confidence bounds on model predictions, allowing the system to defer to expensive verification when confidence is low, which guarantees a limit on the failure rate while reducing the number of required expensive verifications.
What role does UMAP play in the system? UMAP is used to reduce the high-dimensional feature space to a two-dimensional representation, which preserves local neighborhood relationships and global structure, enabling effective visualization and interpretation of large, high-dimensional datasets.
How does the system handle false positive predictions? When a false positive occurs and causes a system stop, the sample is saved into the database with its label, and the model is retrained periodically using the growing database of low-confidence and false positive samples to improve future predictions.