DeFine: Decision-Making with Analogical Reasoning over Factor Profiles

Abstract

LLMs are ideal for decision-making thanks to their ability to reason over long contexts. However, challenges arise when processing speech transcripts that describe complex scenarios, as they are verbose and include repetition, hedging, and vagueness. E.g., during a company's earnings call, an executive might project a positive revenue outlook to reassure investors, despite uncertainty regarding future earnings. It is crucial for LLMs to incorporate this uncertainty systematically when making decisions.

In this paper, we introduce DeFine, a modular framework that constructs probabilistic factor profiles from complex scenarios. It then integrates these profiles with analogical reasoning, leveraging insights from similar past experiences to guide LLMs in making critical decisions in new situations. Our framework separates the tasks of quantifying uncertainty and incorporating it into LLM decision-making. This approach is particularly useful in areas such as consulting and financial deliberation, where making decisions under uncertainty is vital.

Data Statistics

11,950

Transcripts

869

Companies

10,187

Avg. Tokens per Transcript

10

Avg. Q&A Pairs per Transcript

14

Avg. Transcripts per Company

12

Avg. Speakers per Transcript

2017-2024

Year Range

Our dataset includes 11,950 earnings call transcripts from 800+ companies.

DeFine Framework

An excerpt from a typical earnings call transcript and its associated factor profile.

Factor Profile Example

Select Company:

Please select a company from the dropdown above to view its factor profile.

Analogical Reasoning Prompt

Applying Analogical Reasoning to Investment Decisions

System Message

You're a financial analyst who specializes in giving investors buy or sell recommendations by thoroughly analyzing earnings call transcripts.

User Message

Here are several example company profiles. Each profile highlights key factors from an earnings call transcript and probabilities for potential outcomes based on those factors. Each profile represents a specific company and is based on its historical earnings call data. Your job is to pick the most analogous example and use its strategy to solve the initial problem.

Example Company Profile 1:
{Factor Profile 1}
Analyst recommendation: {Action 1}

Example Company Profile 2:
{Factor Profile 2}
Analyst recommendation: {Action 2}

Example Company Profile 3:
{Factor Profile 3}
Analyst recommendation: {Action 3}

Example Company Profile 4:
{Factor Profile 4}
Analyst recommendation: {Action 4}

Example Company Profile 5:
{Factor Profile 5}
Analyst recommendation: {Action 5}

Initial Problem

Based on your analysis of the earnings call for {Company Name} held on {Announcement Date}, decide on the most likely analyst recommendation for the next 30 days from these options:

Action 1: strong buy - The stock price will increase by more than 5%

Action 2: buy - The stock price will increase by 2% to 5%

Action 3: hold - The stock price is expected to remain stable, fluctuating between -2% to 2%

Action 4: sell - The stock price will decrease by 2% to 5%

Action 5: strong sell - The stock price will decrease by more than 5%

Below is the company profile summarized from {Company Name}'s earnings call on {Announcement Date} and the historical price trend probabilities judged by an analyst:

{Factor Profile Constructed Using an Earnings Call Transcript}

Solve the Initial Problem

Please respond with the analyst recommendation for this stock in JSON format, including these keys: (idx, recommendation, justification). idx is the index of the most analogous example profile, and recommendation should be one of the actions mentioned above for 30 days of trading, and justification should clearly explain your recommendation using the strategy you learned from the selected example company profile.

Essential Factors

Factor Description

Hover over any factor below to see its detailed description.

1 Economic Health

2 Market Sentiment and Investor Psychology

3 Political Events and Government Policies

4 Natural Disasters and Black Swan Events

5 Geopolitical Issues

6 Mergers and Major Acquisitions

7 Regulatory Changes and Legal Issues

8 Financial Health

9 Company Growth

10 Company Product Launches

11 Supply Chain

12 Tech Innovation

13 Historical Earnings Per Share (EPS)

14 Historical Revenue

15 Historical Stock Prices

Factor Influence Rankings

Consumer Defensive Sector

Food & beverage, household products, and grocery stores

1 Regulatory changes and legal issues (positive outlook) 0.0364

2 Natural disasters and other black swan events (major impact) 0.0360

3 Political events and government policies (major upheaval) 0.0349

4 Geopolitical issues (escalation to conflict) 0.0345

5 Supply chain (positive outlook) 0.0322

6 Tech innovation (positive outlook) 0.0317

7 Historical stock price change (bullish) 0.0316

8 Historical EPS (bullish) 0.0315

9 Financial health (positive outlook) 0.0311

Technology Sector

Apple, Microsoft, Amazon, Google, and Meta

1 Economic health (unknown or uncertain) 0.0362

2 Market sentiment and investor psychology (unknown or uncertain) 0.0350

3 Company growth (unknown or uncertain) 0.0338

4 Supply chain (unknown or uncertain) 0.0326

5 Geopolitical issues (escalation to conflict) 0.0322

6 Historical revenue (decline) 0.0319

7 Historical stock price change (bullish) 0.0318

8 Tech innovation (unknown or uncertain) 0.0315

9 Natural disasters and other black swan events (major impact) 0.0315

10 Political events and government policies (major upheaval) 0.0313

Experimental Results

Performance Comparison

System	Recall	Precision	F₁	Accuracy	Label	Recall	Precision	F₁
LLM+CoT+Trans	21.56	33.66	13.52	19.59	Strong Sell	7.32	37.50	12.24
LLM+CoT+Summ	22.77	16.17	14.12	20.61	Sell	5.56	9.09	6.90
LLM+CoT+Factors	24.38	28.58	17.26	22.32	Hold	29.84	28.24	29.02
DeLLMa	38.30	23.14	16.68	22.35	Buy	44.83	18.93	26.62
DeFine (Ours)	26.15	27.67	23.73	29.64	Strong Buy	43.22	44.56	43.88

Left: We show the accuracy and macro-averaged F-scores for various systems. Our system, DeFine, which combines factor profiles with analogical reasoning, achieves the best performance. Right: DeFine's performance across five categories: Strong Sell, Sell, Hold, Buy, and Strong Buy.

Confusion Matrix Comparison

A comparison of confusion matrices from the LLM+CoT+Trans, DeLLMa, and DeFine methods. While LLM+CoT+Trans and DeLLMa lean towards 'Buy (B),' DeFine offers more balanced outcomes across all decision categories, showing notable improvement in 'Strong Buy (SB),' 'Buy (B),' 'Hold (H),' and 'Sell (S)' decisions.

Outcome Likelihood Analysis

We analyze and plot the probability density function (PDF) of positive and negative factor outcomes for different investment decisions. Highlighted sections illustrate where the gaps between strong buy (red) and strong sell (blue) decisions are most pronounced.