GPT-4o and executable text mining in marketing: A reproducibilityoriented comparison with a KNIME workflow

Abstract

This paper presents a methodological comparison focused on reproducibility between GPT-4o (via the ChatGPT interface) and an executable text mining workflow in KNIME, in the context of market research. Rather than treating the analytical statements generated by the chatbot as computed results, we distinguish between model-generated statements and executable results, and validate the quantitative statements against a documented KNIME process (Term Frequency-Inverse Document Frequency [ TF-IDF], k-means clustering and lexicon-based sentiment analysis). Using 9,795 online reviews of a fresh food retailer as an illustrative case, we compare the two approaches in terms of pre-processing, descriptive term analysis, clustering and sentiment classification. The comparison shows partial agreement at the level of general themes, but substantial discrepancies in pre-processing-dependent results and sentiment distributions. These findings suggest that GPT-4o can assist in workflow articulation and preliminary interpretation, but that analytical claims generated by the chatbot should not be treated as reproducible results without external, executable validation. The study provides comparative evidence for the prudent use of large language model (LLM)-assisted workflows in marketing text analysis. This article is also included in The Business & Management Collection which can be accessed at http://hstalks.com/business.