Choosing the Right Machine Learning Algorithm

You have data, and you have questions to answer and problems to solve. How do you go about using your data to answer those questions and solve those problems? Due to the power and popularity of big data, machine learning (ML), and artificial intelligence (AI), many organizations leap to the conclusion that choosing machine learning is always the best approach.

However, older, less sophisticated options may deliver better results, depending on the purpose. Sometimes, a spreadsheet or database program is all you need.

The following is a list of options along with suggestions of when each option may be most appropriate for any given data product:

Spreadsheet: If all you need is a table or chart to summarize the data and help to identify patterns in the data, a spreadsheet program, such as Excel, may be great.
Database: A database management system (DBMS) may be sufficient if you need to sort or filter the data, summarize the data in a report, or query the data to explore relationships among data stored in different tables.
Business intelligence (BI) software: BI software combines the power of a DBMS and spreadsheet to retrieve, analyze, transform, and report data in support of data-driven decision-making. This type of software typically features the ability to create dashboards populated with various data visualizations — tables, charts, maps, timelines — that present the data in more meaningful formats.
Machine learning: Machine learning is the best choice when you need to harness the power of predictive or descriptive statistics:

With predictive analysis, the machine can forecast future actions, behaviors, or conditions based on patterns from the past. For example, based on past trading patterns in the stock market, the machine might be able to notify you when the company’s shares are about to increase or decrease in value.
With descriptive analysis, the machine highlights patterns in the data that a human analyst may have overlooked or never thought to look for. For example, a machine can examine the molecular structure of thousands of medications to identify similarities that lead to innovations in the use of existing medications or development of new medications.

Expert systems: To build a system that automates a task currently performed by humans, an expert system may be better and easier to design and build than one that uses machine learning. For example, if you wanted to create a machine for reviewing a mortgage loan application and approving or denying it, you wouldn't need machine language. All you would need is a set of criteria for evaluating the entries on the application along with instructions on how to apply those criteria.

When you're trying to decide between machine learning and an expert system, ask the following question: Does the task require sequential reasoning or pattern matching? If it requires sequential reasoning and the task can be mapped out, go with an expert system. If it requires pattern matching, either to make accurate predictions or to help uncover hidden meaning in the data, machine learning is probably best.

Consulting Your Data Science Team

Prior to deciding which approach is the best match for the problem you're trying to solve or the question you're trying to answer, consult your data science team. Other people on the team may be able to offer valuable insights based on their unique perspectives and training. Encourage your team to ask questions, so they begin to develop an exploratory mindset. Team members should challenge one another's ideas and recommendations, so, together, the team can choose the best approach. (During this process, you may even discover that the question or problem you have identified is not the one you should be seeking to answer or solve. Instead, there may be a more compelling path to explore.)

Keep in mind that two distinctly different approaches may be effective in answering the question or solving the problem, and that a combination of approaches (an ensemble) may be the best approach. If two different approaches seem to be equally effective, opt for the easiest, most cost-effective option.

What is important is that you and your data science team carefully consider the different approaches before starting your work. Choosing the right approach and the right tools will make your job that much easier and deliver superior results.

Frequently Asked Questions

What is a machine learning algorithm?

A machine learning algorithm is a set of rules for a computer. It helps the computer learn from the provided data. They are used to predict and find patterns in data.

How do I choose the best machine learning algorithm for my project?

Choosing the best machine learning algorithm depends on several factors, including the type of machine learning problem you are addressing, the size and nature of the dataset, the number of features, and the required level of accuracy.

Using a cheat sheet or consulting with a data scientist can also help.

What are the different types of algorithms?

The main categories include supervised learning algorithms, unsupervised learning algorithms, and reinforcement learning algorithms. Each type has a different purpose, such as classification, regression, clustering, and decision-making tasks.

What is supervised learning, and where is it commonly used?

Supervised learning is a type of machine learning where the algorithm is trained on labeled data. It is commonly used for classification and regression tasks, where the goal is to predict an output variable based on input variables.

How does the number of features in a dataset affect the choice of machine learning algorithm?

The number of features, also known as dimensionality, can significantly impact the choice of algorithm.

High-dimensional datasets may benefit from reduction algorithms or feature selection techniques, while low-dimensional datasets may work well with simpler linear algorithms.

What are some common supervised learning algorithms?

Common supervised learning algorithms include:

linear regression
logistic regression
support vector machines (SVM)
decision trees
k-nearest neighbors (KNN)
random forests
neural networks.

Why is interpretability important when choosing a machine learning algorithm?

Interpretability is important because it allows stakeholders to understand how decisions are being made by the model.

For applications where understanding the decision process is important, simpler models like linear regression or decision trees are often the best option over more complex models like neural networks.

How do I know if my machine learning problem is linear or non-linear?

You can check if a machine learning problem is linear by plotting input variables against the output variable. If the relationship is a straight line, it's linear. Non-linear relationships will require algorithms that can capture complex patterns, such as neural networks or SVM with non-linear kernels.

This is my weekly newsletter that I call The Deep End because I want to go deeper than results you’ll see from searches or LLMs. Each week I’ll go deep to explain a topic that’s relevant to people who work with technology. I’ll be posting about artificial intelligence, data science, and ethics.

This newsletter is 100% human written 💪 (* aside from a quick run through grammar and spell check).