This Candidate is [MASK]. Prompt-based Sentiment Extraction and Reference Letters

Slonimczyk, Fabian (2025): This Candidate is [MASK]. Prompt-based Sentiment Extraction and Reference Letters.

Abstract

I propose a relatively simple way to deploy pre-trained large language models (LLMs) in order to extract sentiment and other useful features from text data. The method, which I refer to as prompt-based sentiment extraction, offers multiple advantages over other methods used in economics and finance. In particular, it accepts the text input as is (without preprocessing) and produces a sentiment score that has a probability interpretation. Unlike other LLM-based approaches, it does not require any fine-tuning or labeled data. I apply my prompt-based strategy to a hand-collected corpus of confidential reference letters (RLs). I show that the sentiment contents of RLs are clearly reflected in job market outcomes. Candidates with higher average sentiment in their RLs perform markedly better regardless of the measure of success chosen. Moreover, I show that sentiment dispersion among letter writers negatively affects the job market candidate’s performance. I compare my sentiment extraction approach to other commonly used methods for sentiment analysis: ‘bag-of-words’ approaches, fine-tuned language models, and querying advanced chatbots. No other method can fully reproduce the results obtained by prompt-based sentiment extraction. Finally, I slightly modify the method to obtain ‘gendered’ sentiment scores (as in Eberhardt et al., 2023). I show that RLs written for female candidates emphasize ‘grindstone’ personality traits, whereas male candidates’ letters emphasize ‘standout’ traits. These gender differences negatively affect women’s job market outcomes.

• Item Type: MPRA Paper
• Language: English
• Keywords: Large language models; text data; sentiment analysis; reference letters
• Subjects: C - Mathematical and Quantitative Methods > C4 - Econometric and Statistical Methods: Special Topics > C45 - Neural Networks and Related Topics; J - Labor and Demographic Economics > J1 - Demographic Economics > J16 - Economics of Gender ; Non-labor Discrimination; M - Business Administration and Business Economics ; Marketing ; Accounting ; Personnel Economics > M5 - Personnel Economics > M51 - Firm Employment Decisions ; Promotions

References

Abel, M.; Burger, R., and Piraino, P. The Value of Reference Letters: Experimental Evidence from South Africa. American Economic Journal: Applied Economics, 12(3):40–71, 2020.

Akbik, A.; Bergmann, T.; Blythe, D.; Rasul, K.; Schweter, S., and Vollgraf, R. FLAIR: An Easy-to-Use Framework for State-of-the-Art NLP. In Ammar, W.; Louis, A., and Mostafazadeh, N., editors, Proceedings of the Conference of the Association for Computational Linguistics, pages 54–59, Jun 2019.

Algaba, A.; Ardia, D.; Bluteau, K.; Borms, S., and Boudt, K. Econometrics Meets Sentiment: An Overview of Methodology and Applications. Journal of Economic Surveys, 34(3):512–547, 2020.

Angeletos, G.-M. and La’O, J. Sentiments. Econometrica, 81(2):739–779, 2013.

Angeletos, G.-M.; Collard, F., and Dellas, H. Quantifying Confidence. Econometrica, 86(5): 1689–1726, 2018.

Ash, E. and Hansen, S. Text Algorithms in Economics. Annual Review of Economics, 15:659–688, 2023.

Athey, S. and Imbens, G. W. Machine Learning Methods That Economists Should Know About. Annual Review of Economics, 11:685–725, 2019.

Baker, S. R.; Bloom, N., and Davis, S. J. Measuring Economic Policy Uncertainty. The Quarterly Journal of Economics, 131(4):1593–1636, 2016.

Baltrunaite, A.; Casarico, A., and Rizzica, L. Women in Economics: The Role of Gendered References at Entry in the Profession. Working Paper 1438, Bank of Italy, 2024.

Battaglia, L.; Christensen, T.; Hansen, S., and Sacher, S. Inference for Regression with Variables Generated by AI or Machine Learning. arXiv 2402.15585, 2024.

Bayer, P.; Ross, S. L., and Topa, G. Place of Work and Place of Residence: Informal Hiring Networks and Labor Market Outcomes. Journal of Political Economy, 116(6):1150–96, 2008.

Brown, M.; Setren, E., and Topa, G. Do Informal Referrals Lead to Better Matches? Evidence from a Firm’s Employee Referral System. Journal of Labor Economics, 34(1):161–209, 2016.

Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Chess, B.; Clark, J.; Berner, C.; McCandlish, S.; Radford, A.; Sutskever, I., and Amodei, D. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901, 2020.

Bu, K.; Liu, Y., and Ju, X. Efficient Utilization of Pre-trained Models: A Review of Sentiment Analysis via Prompt Learning. Knowledge-based Systems, 283:111148, 2024.

Burks, S. V.; Cowgill, B.; Hoffman, M., and Housman, M. The Value of Hiring Through Employee Referrals. The Quarterly Journal of Economics, 130(2):805–839, 2015.

Caldara, D. and Iacoviello, M. Measuring Geopolitical Risk. American Economic Review, 112(4):1194–1225, 2022.

Calvo-Armengol, A. and Jackson, M. O. The Effects of Social Networks on Employment and Inequality. American Economic Review, 94(3):426–454, 2004.

Camara, O.; Jia, N., and Raffiee, J. Reputation, Competition, and Lies in Labor Market Recommendations. Management Science, 69(11):7022–7043, 2023.

Carmichael, H. L. Incentives in Academics: Why is There Tenure? Journal of Political Economy, 96(3):453–472, 1988.

Coles, P.; Cawley, J.; Levine, P. B.; Niederle, M.; Roth, A. E., and Siegfried, J. J. The Job Market for New Economists: A Market Design Perspective. Journal of Economic Perspectives, 24(4):187—-206, Dec 2010.

Dustmann, C.; Glitz, A.; Schonberg, U., and Brucker, H. Referral-based Job Search Networks. The Review of Economic Studies, 83(2):514–546, 2016.

Eberhardt, M.; Facchini, G., and Rueda, V. Gender Differences in Reference Letters: Evidence from the Economics Job Market. The Economic Journal, 133(655):2676–2708, 2023.

Ekinci, E. Employee Referrals as a Screening Device. The RAND Journal of Economics, 47(3): 688–708, 2016.

Frankel, R.; Jennings, J., and Lee, J. Disclosure Sentiment: Machine Learning vs. Dictionary Methods. Management Science, 68(7):5514–5532, 2022.

Garcia, D. Sentiment During Recessions. The Journal of Finance, 68(3):1267–1300, 2013.

Gentzkow, M.; Kelly, B., and Taddy, M. Text as Data. Journal of Economic Literature, 57(3): 535–574, 2019.

Hellerstein, J. K.; McInerney, M., and Neumark, D. Neighbors and Coworkers: The Importance of Residential Labor Market Networks. Journal of Labor Economics, 29(4):659–95, 2011.

Hochleitner, A.; Tufano, F.; Facchini, G.; Rueda, V., and Eberhardt, M. How Tinted Are Your Glasses? Gender Views, Beliefs and Recommendations in Hiring. Working Paper 11789, CESifo, 2025.

Huang, A. H.; Wang, H., and Yang, Y. FinBERT: A Large Language Model for Extracting Information from Financial Text. Contemporary Accounting Research, 40(2):806–841, 2023.

Hutto, C. and Gilbert, E. VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text. In Eighth International Conference on Weblogs and Social Media, ICWSM-14, 2014.

Izmalkov, S. and Yildiz, M. Investor Sentiments. American Economic Journal: Microeconomics, 2(1):21–38, Feb 2010.

Liu, P.; Yuan, W.; Fu, J.; Jiang, Z.; Hayashi, H., and Neubig, G. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. ACM Computing Surveys, 55(9):1–35, 2023.

Loughran, T. and McDonald, B. When Is a Liability Not a Liability? Textual Analysis, Dictionaries, and 10-Ks. The Journal of Finance, 66(1):35–65, 2011.

Ludwig, J.; Mullainathan, S., and Rambachan, A. Large Language Models: An Applied Econometric Framework. arXiv preprint 2412.07031, 2025.

Maxted, P. A Macro-finance Model with Sentiment. Review of Economic Studies, 91(1):438–475, 2024.

Meta AI, . Introducing Llama 3.1: Our Most Capable Models to Date. Blog, Jul 2024.

Min, B.; Ross, H.; Sulem, E.; Veyseh, A. P. B.; Nguyen, T. H.; Sainz, O.; Agirre, E.; Heintz, I., and Roth, D. Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey. ACM Computing Surveys, 56(2), Sep 2023.

Montgomery, J. D. Social Networks and Labor-market Outcomes: Toward an Economic Analysis. American Economic Review, 81(5):1408–1418, 1991.

Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C. L.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; Schulman, J.; Hilton, J.; Kelton, F.; Miller, L.; Simens, M.; Askell, A.; Welinder, P.; Christiano, P.; Leike, J., and Lowe, R. Training Language Models to Follow Instructions with Human Feedback. arXiv preprint 2203.02155, 2022.

Pallais, A. Inefficient Hiring in Entry-Level Labor Markets. American Economic Review, 104(11): 3565–99, 2014.

Saloner, G. Old Boy Networks as Screening Mechanisms. Journal of Labor Economics, 3(3): 255–267, 1985.

Sanh, V.; Debut, L.; Chaumond, J., and Wolf, T. DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter. arXiv preprint 1910.01108, 2020.

Schmader, T.; Whitehead, J., and Wysocki, V. A Linguistic Comparison of Letters of Recommendation for Male and Female Chemistry and Biochemistry Job Applicants. Sex Roles, 57 (7-8):509––14, 2007.

Shapiro, A. H.; Sudhof, M., and Wilson, D. J. Measuring News Sentiment. Journal of Econometrics, 228(2):221–243, 2022.

Simon, C. J. and Warner, J. T. Matchmaker, Matchmaker: The Effect of Old Boy Networks on Job Match Quality, Earnings, and Tenure. Journal of Labor Economics, 10(3):306–330, 1992.

Stock, J. and Yogo, M. Asymptotic Distributions of Instrumental Variables Statistics with Many Instruments. In Andrews, D. W. K. and Stock, J. H., editors, Identification and Inference for Econometric Models: Essays in Honor of Thomas Rothenberg, chapter 6, pages 109–120. Cambridge University Press, 2005.

Sun, T.-X.; Liu, X.-Y.; Qiu, X.-P., and Huang, X.-J. Paradigm Shift in Natural Language Processing. Machine Intelligence Research, 19(3):169–183, 2022.

Ter Ellen, S.; Larsen, V. H., and Thorsrud, L. A. Narrative Monetary Policy Surprises and the Media. Journal of Money, Credit and Banking, 54(5):1525–1549, 2022.

Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozi`ere, B.; Goyal, N.; Hambro, E.; Azhar, F.; Rodriguez, A.; Joulin, A.; Grave, E., and Lample, G. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint 2302.13971, 2023.

Trix, F. and Psenka, C. Exploring the Color of Glass: Letters of Recommendation for Female and Male Faculty. Discourse and Society, 14:191–220, 2003.

Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L., and Polosukhin, I. Attention is All You Need. In Advances in Neural Information Processing Systems, volume 30 of NIPS-17, 2017.

Zhao, W. X.; Zhou, K.; Li, J.; Tang, T.; Wang, X.; Hou, Y.; Min, Y.; Zhang, B.; Zhang, J.; Dong, Z.; Du, Y.; Yang, C.; Chen, Y.; Chen, Z.; Jiang, J.; Ren, R.; Li, Y.; Tang, X.; Liu, Z.; Liu, P.; Nie, J.-Y., and Wen, J.-R. A Survey of Large Language Models. arXiv 2303.18223, 2023.