TorfsBot Or Not? Evaluating User Perception on Imitative Text Generation

Mimicking an individual's writing style using automated text generation approaches presents a challenge in determining the optimal level of imitation versus exaggeration. This study investigates the believability and user interaction of a text generator employing Markov models and dynamic templates in comparison to source data. Using the TorfsBot Twitterbot, which emulates the writing style of the Belgian professor Rik Torfs, we developed a secondary Twitterbot, "TorfsBot or Not?”, that conducted daily polls asking users to identify whether a tweet originated from Rik Torfs or TorfsBot. The study collected 43K votes from approximately 500 polls. The findings reveal that participants correctly identified the source of the tweets 71% of the time, with majority votes inaccurately attributing the source in 14% of the polls. Furthermore, we observed a positive correlation between the number of interactions on the source tweet and the perceived believability that the tweet originated from Rik Torfs. Our results suggest that even relatively simple text generation models can approximately replicate a target's writing style, and that a closer approximation to the source material may positively influence user engagement.