Caught in a Vicious Cycle Obstacles and Opportunities for Trust and Safety Teams in the Games Industry

Key Findings:

This report aims to expand the games industry’s understanding of content moderation and the ways it can improve. Game companies cannot control users’ actions, but they can influence their behavior and experience to a large degree by deciding which types of user behavior are prohibited and welcome in their games. Companies can create more-positive environments by not solely relying on punitive measures such as bans, which have limited efficacy.

Since 2019, ADL has conducted an annual survey of hate, harassment, and extremism in online multiplayer games. The 2022 survey shows that hate continues unabated. More than four out of five adults (86%) ages 18-45 experienced harassment. In addition, more than three out of five young people ages 10-17 experienced harassment.

For this report, however, ADL decided to focus not on users’ experiences but on the challenges faced by trust and safety employees in the games industry when it comes to moderating hate and harassment. ADL wanted to determine whether game companies prioritize content policy enforcement and whether they give staff adequate support. Building on our previous work, the ADL Center for Technology & Society analyzed the policy guidelines of 12 games and interviewed a number of trust and safety experts in the games and technology industries. 

Recommendations:

(1) Assign more resources to understaffed and overwhelmed trust and safety teams in game companies. Teams are bogged down by institutional challenges, explaining the value of content moderation to skeptical executives and securing bigger budgets to hire more staff and expand their work.

(2) Make content moderation a priority in the creation and design of a game. Trust and safety experts say content moderation should be central from a game’s conception to its discontinuation.

(3) Focus content moderation on the toxic 1%. Networks, not individuals, spread toxicity. Game companies should identify clusters of users who disproportionately exhibit bad behavior instead of trying to catch and punish every rule-breaking individual.               

(4) Build community resilience. Positive content moderation tools work. Use social engineering strategies such as endorsement systems to incentivize positive play.

(5) Use player reform strategies. Most players respond better to warnings than punitive measures.

(6) Provide consistent feedback. When a player sends a report, send an automated thank you message. When a determination is made, tell the reporting player what action was taken. This not only shows players that the team is listening, but it also models positive behavior.  

(7) Avoid jargon and legalese in policy guidelines. These documents should be concise and easy for players to read. Every game should have a Code of Conduct and Terms of Service.