Boosting performance with sentiment analysis

Published on June 23, 2023 by Cristina Elizondo

Introduction

Data usage has been evolving and transforming sectors since its benefits have become more tangible and immediate. However, the difference between traditional data and alternative data has been more noticeable since the late ’90s/early 2000s. On the one hand, traditional data focuses on information gathered from financial reports (SEC filings), surveys and other similar sources, while alternative data is able to convert structured and unstructured data into pragmatic insights that ease the decision-making process.

Some non-traditional sources from which alternative data is gathered are credit card transaction data, mobile device data, ESG data and social sentiment data. This blog focuses on social sentiment data, as it grown exponentially in the past decade and is expected to continue growing in the next seven years at a CAGR of 14.4% (source: Polaris Market Research Analysis, 2022)^[1]. This type of alternative data has proven to deliver insights on consumers and public opinion and help monitor brand reputation, enhance customer experience and conduct research, all of which could translate into additional value for the company. The use of social media was boosted amid the pandemic, presenting a significant growth opportunity for the sentiment analytics market.

This blog aims to define and analyse the use of sentiment analysis while understanding its effect on the overall performance of a business, specifically from two perspectives: customer experience and market sentiment. We also discuss challenges to artificial intelligence (AI) models and trends in sentiment analysis.

 

Sentiment analysis – an overview

 

Sentiment analysis, sometimes known as opinion mining, uses natural language processing (NLP), computational linguistics and text analytics to identify and classify subjective opinions (source: Xu et al, 2013)^[2]. However, the way sentiment analysis works has evolved as data growth is unavoidable, meaning that initially. social sentiment was categorised manually and as datasets grew, machine learning and NLP were introduced to ease the process.

This type of analysis has transformed the way market research is conducted: businesses no longer have to follow extensive, manual and tedious processes to know how to enhance product development; they can now actually focus on how their audiences feel about a product or brand and act accordingly. However, sentiment data on its own is not enough in any sector, whether healthcare, finance or politics; it must be treated as complementing standard data sources or as an additional layer of analysis on top of conventional approaches such traditional data (source: Bondo, Borch., 2022)^[3].

Despite most conducting sentiment analysis for brand monitoring through social media and market research (source: Sharma, 2023)^[4], there is extensive evidence that proves its benefits for the financial sector, where it can provide useful text mining to process content and filter results to obtain relevant and meaningful information, leading to an informed decision-making process and better results for investors.

 

The AI tool that boosts business performance

 

Sentiment analysis narrows the distance between a perfect customer experience and product development and innovation through NLP models.

Sentiment analysis (SA) is considered to be an AI tool that enhances customer reach, sales revenue and trends, and insight on the market, increasing overall business performance. One of the most important factors of a business that SA tends to improve is customer experience, and given that consumers are no longer willing to settle for a bad or even mediocre experience, a minor error could lead to a significant opening for other companies to gain market share by outperforming their competitors in providing exactly what the customer needs at a specific point in time (source: Almada, Jeleriu, 2021)^[5]. This highlights the true significance of SA in delivering an exceptional customer experience to maintain, as long as possible, a competitive advantage, regardless of the sector.

Diagram 1 represents a cycle in which a positive relationship between SA and customer experience is built. It starts with a product or service provided by the company; subsequently, the customer buys it and comments on it on social media. This is where SA stands out from the traditional platforms of customer reviews; SA goes beyond analysing what customers say about a product on a customer review site, and uses AI to extract actual customer sentiment on social media platforms in terms of a specific product or brand. This makes it more natural and less forced, given that customers sometimes feel obliged to post a review after buying a product. SA goes the extra mile, extracting customer intent from the actual comment on social media such as Twitter, blogs, news, forums, Reddit, broadcasts or Pinterest. The intention could be positive or negative and is extracted using NLP and computational linguistics. This fourth step is crucial for achieving an enhanced customer experience.

Probably one of the most useful platforms for this type of analysis is Meltwater, along with Audiense and Klear. These three software packages provide pragmatic data relating to how customers feel about a specific product or brand, as well as information on their interests, hobbies, which places they frequent, and even personality traits – all elements that help understand exactly what the customer wants and needs, in order to achieve customer loyalty. Ultimately, SA helps transform customers’ voices into actionable data so the company can make customer-centric decisions, enabling a brand to offer an exceptional customer experience.

A study by Bain & Company and Harvard Business School shows not only a positive relationship between SA and the customer experience, but also with business profits, where ‘increasing retention rates by 5% increases profits by 25% to 95%’ (source: Reichheld, Schefter, 2000)^[6]. Additionally, Spillane (2022) found that existing customers are 50% more likely to try new products and spend 31% more, on average, than new customers (source: Spillane, 2022)^[7], generating extra value for the company in the long term.

Achieving a great customer experience is key for the financial and overall performance of a business, and in this digital age, providing a flawless user experience is what makes the difference in maintaining a strong competitive advantage. It is safe to say that companies are now investing more in technology, specifically AI. Results of a recent Accenture survey show that 63% of the organisations surveyed do not prioritise AI over other digital technologies. Accenture’s statistics are similar to McKinsey’s; McKinsey shows that more than half of the companies it surveyed are investing more than 5% of their digital budgets in AI, compared to less than 1% previously (source: Wiggers, 2023)^[8].

As social media provides large volumes of user opinion in real time, investors could exploit this information by leveraging NLP techniques such as sentiment analysis (source: Zhao et al., 2018)^[9].

An unbeatable customer experience is definitely not the only factor contributing to business performance. Its performance in the stock market, based on price predictions, is also a factor. The relationship between market sentiment and business performance relies mainly on the rise and fall of share price values and how these price fluctuations directly affect a company’s market capitalisation and subsequently, its market value (source: Investopedia, 2021) ^[10]. The higher shares are priced, the more a company is worth in terms of market value and vice versa. Diagram 2 portrays this relationship and shows in five steps how public sentiment can translate into valuable information for customers, investors and companies, enabling them to understand how market prices fluctuate over time and to make informed decisions.

For this process to be as accurate as possible, data labelling through machine learning (ML) models is essential to clean and structure data. Labelling data allows for meaningful tags and classification into groups, which in this case, the role of SA is based on categorising emotional expression as positive, negative or neutral (source: IBM). With assistance from ML, data labelling in SA also requires “human-in-the-loop (HITL)” participation (source: IBM, 2023)^[12]. One should always bear in mind, however, that these ML models will not be 100% error-free – there will be areas that these models are unable to identify; this is where HITL leverages judgment of human “data labellers” for correcting any wrongly assigned social media comment. Google FinBERT also assimilates this type of analysis as it examines sentiment in financial text, engaging the views and opinions of managers, information intermediaries and investors. This model is fine-tuned on 10,000 manually annotated sentences from analyst reports of S&P 500 firms.

Nonetheless, neither of these models is perfect – they could be areas that they do not interpret at all or interpret incorrectly (source: Yilmaz, 2022) ^[13], such as the following:

• Context-dependent errors

  • Sarcasm

  • Polarity

  • Polysemy

• Negation detection

• Multilingual data

The error margin could be far more critical: if a social media comment is assigned as positive because the data-labelling model did not understand the context of the comment, it could give the wrong idea and companies and/or investors may act against the current, generating the opposite of the desired effect.

 

Future trends

 

This year has so far has been revolutionary year in terms of technological advances to automate routine processes. One of the most widely known, controversial and probably useful innovations is large language models (LLMs). These models are a type of AI algorithm that applies deep-learning techniques to extensive datasets to understand, summarise, generate and predict new content (source: Kerner, 2023)^[14]. Text generation, translation, content summary, rewriting content and sentiment analysis are some of these models’ numerous uses; they have gained in popularity as businesses notice the benefits in the short and long term of investing in technology to boost overall performance.

The functioning of these models depends on continuous training; this process is termed supervised learning and comprises the following steps:

1. Text input and its corresponding outputs are given to the model to predict the output for a new input.

2. An optimisation algorithm is used to adjust parameters to minimise the difference between prediction and the actual output.

3. The training data is given to the model on a smaller scale (in categories).

4. The model makes predictions for each category and makes changes based on identified errors.

5. The process is repeated frequently for two main reasons: to learn relationships and identify patterns in the data.

Essentially, the future of generative AI seems promising, as it is a trending topic among venture capitalists, entrepreneurs and Fortune 500 CEOs – from Silicon Valley to Davos (source: Toews, 2023)^[15]. These models have the potential to generate not only a vast economic and financial, but also a societal, transformation.

The next steps of sentiment analysis are tied to the rise of social media. This would lead to an inflection point, where companies clearly notice the benefits of merging alternative data with non-alternative data to gain more insightful and actionable data. Metrics that highlight the increased use of sentiment analysis for business purposes due to the rise of social media include the following:

• There are 9bn social media users worldwide.

• People approximately spend 2 hours and 35 minutes a day on social media.

• 60% of the consumers surveyed consult blogs or social media reviews before deciding to buy a product or service.

• On average, social media reviews increase sales by 18%.

Source: Ruby, 2023; Ghann, 2023^{[16] [17]}

In summary, use of social media data is expected to continue increasing in the coming years; it is, therefore, the perfect time for businesses to invest in AI and ML tools that enable them to go beyond traditional brand analysis. These tools, such as sentiment analysis, will help them examine and evaluate how customers feel about a company, brand or product. Such analysis not only provides benefits from the point of view of customer experience, but also enhances financial results, enabling better predictions of stock prices and risk management.

However, it should be remembered that this analysis is not error-free and that it has several drawbacks related to the context of customer comments. Nevertheless, as AI-driven sentiment analysis continues to evolve, it would see even more innovative applications that steadily fix the errors, although it may require extensive human participation to train the model at the beginning (source: Frąckiewicz, 2023) ^[18].

References:

• [1] Polaris Market Research (March 2022). Sentiment Analytics Market. Retrieved from Sentiment Analytics Market Size Global Report, 2022 - 2030 (polarismarketresearch.com)

• [2] Xu, G., Zhou, J., Chen, S., Luo, T. (1 January 2013). Sentiment Analysis. Research Gate. Retrieved from (PDF) Sentiment Analysis (researchgate.net)

• [3] Bondo, K., Borch, C. (20 January 2022). Alternative data and sentiment analysis: Prospecting non-standard data in machine learning-driven finance. SAGE Journals. Retrieved from https://doi.org/10.1177/20539517211070701

• [4] Sharma, S. (30 January 2023). Top 10 Applications of Sentiment Analysis in Business. Analytics Vidhya. Retrieved from Top 10 Applications of Sentiment Analysis in Business (analyticsvidhya.com)

• [5] Almada, A., & Jeleriu, M. (24 June 2021). Customer experience as a competitive advantage. 3Pillar Global. Retrieved from Customer Experience as a Competitive Advantage | 3Pillar Global

• [6] Reichheld, F., Schefter, P. (7 October 2000). The Economics of E-loyalty. Harvard Business School. Retrieved from The Economics of E-Loyalty – HBS Working Knowledge – Harvard Business School

• [7] Spillane, J. (14 December 2022). Why Your Existing Customers Are Your Best Customers. Business Community. Retrieved from Why Your Existing Customers Are Your Best Customers –  Business 2 Community

• [8] Wiggers, K. (20 March 2023). Corporate investment in AI is on the rise, driven by the tech’s promise. Tech Crunch. Retrieved from Corporate investment in AI is on the rise, driven by the tech’s promise | TechCrunch

• [9] Zhao, H., Sinha, A., Huang, Z., Deng, S. (19 September 2018). Can social media sentiment affect stock market performance? London School of Economics. Retrieved from Can social media sentiment affect stock market performance? | LSE Business Review

• [10] Investopedia (9 December 2021). How does the performance of the stock market affect individual businesses? Retrieved from How does the performance of the stock market affect individual businesses? (investopedia.com)

• [11] Da Costa, T. (28 May 2020). Using Sentiment Analysis to Examine Stocks. Daily FX. Retrieved from Using Sentiment Analysis to Examine Stocks (dailyfx.com)

• [12] IBM (2023). What is Data Labelling? Retrieved from What is Data Labeling? | IBM

• [13] Yilmaz, B. (30 December2022). Top 5 Sentiment Analysis Challenges and Solutions in 2023. AI Multiple. Retrieved from Top 5 Sentiment Analysis Challenges and Solutions in 2023 (aimultiple.com)

• [14] Kerner, S. (April 2023). Large Language Models. TechTarget. Retrieved from What is a large language model (LLM)? – TechTarget Definition

• [15] Toews, R. (7 February 2023). The Next Generation Of Large Language Models. Forbes. Retrieved from The Next Generation Of Large Language Models (forbes.com)

• [16] Ruby, D. (20 March 2023). Social Media Users In The World — (2023 Demographics). Demand SAGE. Retrieved from https://www.demandsage.com/social-media-users/

• [17] Ghann, J. (2023). 59 Product Review Statistics Every Online Store Needs To Know. Fera AI. Retrieved from https://www.fera.ai/blog/posts/online-product-review-statistics-ecommerce-stores-need-

---

---