DEEP LEARNING
Making AI production-ready
Following speaking at Embedded Vision Europe, Pierre Gutierrez, lead machine learning researcher at Scortex, writes about the challenges of deploying deep learning on the factory floor
T
he efficacy of deep learning for computer vision is now well proven. As a result, many manufacturing
companies are trying to deploy or at least run a first project using deep learning. Take the example of industrial inspection,
our task at Scortex. If a company was to start from scratch, a typical setup for a machine or deep learning proof-of-concept would be: first, the data scientist is given a dataset of, let’s say, around 10,000 annotated images. Te dataset is then split into a training set on which the model parameters are going to be learned, and an evaluation set to assess the model’s performance. A convolutional neural network, pre-trained on ImageNet, is used and gives potentially good results, for example 99 per cent recall and 99 per cent precision. Te proof-of-concept project is then
considered a success, and therefore most of the work is done, right? Wrong! Tis is because of several challenges, namely that there is a huge gap between an approximate working model and a solution running in production, and that, in factories, processes and manufactured parts will evolve and the deep learning solution must be able to cope with unavoidable changes.
Deploying deep learning A deep learning approach is a data-based solution. Tis means, firstly, the acquisition system is paramount – the phrase ‘garbage in, garbage out’ is apt here. In computer vision, the easier it is to distinguish defective from non-defective regions on an image with the human eye, the easier it will be for a computer model. While it is true that deep learning is more robust to acquisition system changes than earlier computer vision solutions, do not expect the model to detect things a trained human would not be able to
robust database will be required. In the first case, acquiring data on the factory floor is error prone – some unnecessary data will be collected, some wrongly labelled. Te user needs to be able to curate data from a specific date and time. In addition, a proper database is necessary to create clear training and evaluation sets, for class-stratified sampling for training, or for using per-part or per-day dataset balancing. A proper database also enables
performance breakdown; for example, on what part or defect does the model perform well? Finally, a database is needed for traceability: what happened on this day? Why did the model infer this part was defective while the human did not see it? In addition, a deep learning approach
‘Te proof-of-concept project is considered a success, and therefore most of the work is done, right? Wrong!’
distinguish. Indeed, the dataset the model is trained on is itself annotated by humans who must be able to spot such defects. A second consideration is that vertical
data understanding is key. Te deep learning algorithms and the ways of training them are becoming a commodity; vertical deep understanding of business and data is not. Changing model architecture can give a few more accuracy points, but the real game changer is the quality of the data. Here is an unfortunate example. After a few weeks of collecting and annotating data, you train a model that seems to perform well. You test it in production, but realise the quality specifications have been misunderstood leading to a very high defect tolerance. To remedy the problem, all the data has to be re- annotated, which costs tens of thousands of euros and leads to weeks of project delay. Storage and data governance is also
necessary. Sure, at first, a folder with images and a large JavaScript Object Notation (JSON) for labels seems okay. But pretty soon, a more
18 IMAGING AND MACHINE VISION EUROPE DECEMBER 2019/JANUARY 2020
needs a fast and robust inference pipeline. Scortex is dealing with real-time constraints, while working with high-resolution images at 10 to 20 frames per second. Tis means that an overly large off-the-shelf network architecture may not be a good fit for the application, and that the company might not be able to deploy a trained model unless there is a software engineer team making the prediction delivery a commodity. Once the user has a solution for all
these issues, they have a decent industrial prototype because handling the inputs and outputs of the system is reasonably covered. Now this is where it gets complex, from the machine learning point of view.
Te open world challenge In a factory, things tend to change and unpredictable cases will appear. As a result, the user cannot expect a model trained on a few images from a specific day on a particular setup to generalise to all possible cases forever. Tis means that models need to be maintained over time and have the ability to adapt. In the deep learning community this is called the open world challenge. Here are a few factors the user may want
the deep learning solution to be robust to, or at least be able to adapt to: • Continuous distributions drift: Te environment will change, whether you want it to or not – some camera pixels may die, the intensity of an old LED may vary, the amount of dust in the factory or on the camera lens may increase.
@imveurope |
www.imveurope.com
Nice Illustration/
Shutterstock.com
Page 1 |
Page 2 |
Page 3 |
Page 4 |
Page 5 |
Page 6 |
Page 7 |
Page 8 |
Page 9 |
Page 10 |
Page 11 |
Page 12 |
Page 13 |
Page 14 |
Page 15 |
Page 16 |
Page 17 |
Page 18 |
Page 19 |
Page 20 |
Page 21 |
Page 22 |
Page 23 |
Page 24 |
Page 25 |
Page 26 |
Page 27 |
Page 28 |
Page 29 |
Page 30 |
Page 31 |
Page 32 |
Page 33 |
Page 34 |
Page 35 |
Page 36 |
Page 37 |
Page 38 |
Page 39 |
Page 40
