pctechguide.com

  • Home
  • Guides
  • Tutorials
  • Articles
  • Reviews
  • Glossary
  • Contact

Monitoring in Machine Learning Part 2: Monitoring Techniques

We talked about the reasons that you need to monitor in machine learning in our last post. We are now clear about the main factors that can degrade the performance of a model.

So we can define monitoring as the phase of Machine Learning Operations in which we measure different performance variables of the model and compare them with reference values to determine if it continues to generate adequate predictions or if it is necessary to take actions to improve performance.

And there are several ways to perform this monitoring, some quite simple and others more sophisticated.

Monitoring through global metrics
The simplest of all is to continuously record a global metric of the model’s performance and compare it to a reference level.

For example, if we have a face detection system that at the development stage had an accuracy of 97% then we can periodically (e.g. daily) record this performance on the deployed model and if it is observed to fall below this reference level an alert could be generated indicating that we should take some action before things continue to get worse.

The drawback of monitoring using a global performance metric is that we cannot determine the reasons behind the degradation, i.e. whether the underlying problem is “data drift” or “concept drift”.

Monitoring through statistical methods
A more sophisticated way to perform monitoring is for example to obtain the statistical distribution of the input data before deployment and periodically calculate this distribution but for the data used by the deployed model, and then apply a statistical test to determine if there are significant differences between one and the other. In the case of finding differences we could conclude that the origin of the degradation is in “data drift”.

Monitoring in detail
Perfect, we already have clear the main factors that can degrade the performance of a model.

So we can define monitoring as the phase of Machine Learning Operations in which we measure different performance variables of the model and compare them with reference values to determine if it continues to generate adequate predictions or if it is necessary to take actions to improve performance.

And there are several ways to perform this monitoring, some quite simple and others more sophisticated.

Monitoring through global metrics
The simplest of all is to continuously record a global metric of the model’s performance and compare it to a reference level.

For example, if we have a face detection system that at the development stage had an accuracy of 97% then we can periodically (e.g. daily) record this performance on the deployed model and if it is observed to fall below this reference level an alert could be generated indicating that we should take some action before things continue to get worse.

The drawback of monitoring using a global performance metric is that we cannot determine the reasons behind the degradation, i.e. whether the underlying problem is “data drift” or “concept drift”.

Monitoring through statistical methods
A more sophisticated way to perform monitoring is for example to obtain the statistical distribution of the input data before deployment and periodically calculate this distribution but for the data used by the deployed model, and then apply a statistical test to determine if there are significant differences between one and the other. In the case of finding differences we could conclude that the origin of the degradation is in “data drift”.

Or we can do something similar but for the data distributions at the model output before and after deployment, so that if we find statistically significant differences we can conclude that the performance degradation is in this case due to “concept drift”.

Conclusion
Very well, in this article we have seen that after deployment it is very likely that the performance of the model begins to decline and this is precisely because both the data and the environment in which the model is in are dynamic and can continuously present variations.

So monitoring allows detecting this performance degradation, either by analyzing global metrics or by using more advanced techniques such as the use of statistical tests applied to the model’s input or output data.

But this process does not end with monitoring, because if performance degradation is confirmed, corrective actions must be taken to keep the model in production. This phase is known as model maintenance and will be discussed in a future article.

Filed Under: Articles

Latest Articles

iPhone 6s and 6s Plus Making Waves

Apple has finally released the much anticipated iPhone 6s and 6s Plus. The tech giant has made several additions and changes to its flagship model, including the new iOS 9, extended battery life, and a more powerful camera, in order to deliver a more convenient and personalized user experience. … [Read More...]

6 Tips for Social Media Success in the New year

Social media continues to grow in leaps and bounds. With every passing year, we learn new things about what works and what fails. One thing that is for sure is that social media is about building relationships. Brands need to learn and interact with their customers intimately. They also need to be … [Read More...]

A Cool Computer– Case Closed and Location, Location, Location

We have been writing a series of articles on how to keep your PC running cool and efficiently. Keeping a computer running cool is important because heat will actually slow a computer down and cause damage to the computer over the long term. Therefore, it is important to do what you can to keep the … [Read More...]

Why Cross-Chain Trading Is the Future of Crypto Investing?

The rapid growth and evolution of the cryptocurrency market have opened up exciting opportunities for investors. Within this dynamic landscape, … [Read More...]

Revolutionize Your Internet Experience with Orbi 960 – The Ultimate WiFi System

In a world where seamless connectivity is essential, slow and unreliable internet connections are a major problem. Whether you are running a business, … [Read More...]

Do You Need a VPN When Trading Cryptocurrency?

There’s no doubt that the biggest global industries in 2023 are tech-driven, while there remains a significant crossover between many of these … [Read More...]

Goodbye Bitcoin: the 3 alternative cryptocurrencies that have great upside potential, according to experts

Bitcoin has been a very lucrative investment for people that got into it early. One report from The Motley Fool pointed out that $10 of bitcoin … [Read More...]

Self-driving cars face their Achilles’ heel and may be targets of hackers

The market for self-driving cars is booming. Customers spent $22.22 billion on these autonomous vehicles in 2021 and they will likely spend more in … [Read More...]

How to avoid scams with bitcoin and other cryptocurrencies

Cryptocurrencies got a bad reputation when scams multiplied like ants on a piece of cake. Even today many people associate bitcoin and other … [Read More...]

Guides

  • Computer Communications
  • Mobile Computing
  • PC Components
  • PC Data Storage
  • PC Input-Output
  • PC Multimedia
  • Processors (CPUs)

Recent Posts

Defeating The Threats of A-Secure 2015

A-Secure 2015 is actually a fake and deceiving anti-virus program that will make you believe it’s a legitimate anti-virus protection program starting … [Read More...]

CMOS – complementary metal oxide silicon – RAM chips on motherboards

Motherboards also include a separate block of memory made from very low power consumption CMOS (complementary metal oxide silicon) RAM chips, … [Read More...]

USB 2.0 Intefaces

While USB was originally designed to replace legacy serial and parallel connections, notwithstanding the claims that they were complementary … [Read More...]

[footer_backtotop]

Copyright © 2023 About | Privacy | Contact Information | Wrtie For Us | Disclaimer | Copyright License | Authors