Blog post by Dr. Ayşe Gizem Yaşar, CREATe Fellow
Grok and X have recently come under fire as Grok reportedly produced and disseminated thousands of sexually explicit images of children and women on X in response to user requests. Many of these images were non-consensually altered photos of real people. Regulators around the world are increasingly called upon to prohibit the AI bot from generating and distributing sexually explicit content. In response to the incident, the European Commission reiterated that X must comply with DSA, but it hasn’t yet confirmed as of 14 January whether it opened any new DSA investigations on the matter.
Whether the DSA can be leveraged to prohibit the chatbot’s generation in addition to the dissemination of sexually explicit content partially depends on a rather technical question: is Grok a “related system” under Article 34 DSA?
What happened?
Grok is a chatbot developed by xAI which runs on foundation models also developed by xAI. The chatbot is provided both as a standalone service on the Grok website and app, and as an integrated chatbot on the X platform. On X, Grok has a dedicated account, @grok, that users can engage with by tagging the account on their posts. The platform also features a private Grok chatbot. Users can directly post the chatbot’s outputs on X.
Since early 2026, Grok and X have come under fire as the @grok account began posting sexualised photos of real people in response to user requests. Though the controversy began this year, the chatbot has had a “spicy mode” for months which reportedly allows paid subscribers on X or Grok to generate sexually explicit images. X initially responded to the public backlash by just warning the users: “Anyone using or prompting Grok to make illegal content will suffer the same consequences as if they upload illegal content.” Understandably the announcement further fanned the flames. “X blames users for Grok-generated CSAM; no fixes announced” reported ArsTechnica. As the backlash grew, X took a step back and admitted to “lapses in safeguards” of xAI. As of 14 January, X has only announced some changes to the @grok account on X “to prevent the Grok account from allowing the editing of images of real people in revealing clothing”. So far it hasn’t announced any changes to the chatbot or the model features that allow users to create sexually explicit images or alter photos in sexually explicit ways.
Systemic risk mitigation under the DSA
Within the EU’s digital rulebook, AI safety is clearly the focus of the AI Act, not the DSA. The latter primarily concerns digital intermediaries and questions of intermediary liability and accountability in relation to third-party content on their services. But the DSA also comes into the picture as a form of AI governance when AI-generated content is shared across large platforms, which brings us to the systemic risk provisions of the DSA.(1) These provisions apply only to very large online platforms (VLOPs) and very large online search engines (VLOSEs). Platforms and search engines are designated as VLOP and VLOSE when they exceed the threshold of “45 million monthly active recipients of the service in the European Union” (Article 33(1)).
Article 34(1) requires the providers of VLOPs and VLOSEs to “diligently identify, analyse and assess any systemic risks in the Union stemming from the design or functioning of their service and its related systems, including algorithmic systems, or from the use made of their services.” These systemic risks include (i) dissemination of illegal content, (ii) negative effects for the exercise of fundamental rights, (iii) negative effects on civic discourse and electoral processes, and public security, and (iv) negative effects in relation to gender-based violence, the protection of public health and minors and serious negative consequences to the person’s physical and mental well-being.
The kind of sexual material that Grok generated probably falls under several of these risk categories. Child sexual abuse material (CSAM) is illegal in the EU. Some of the content is also likely to have negative effects in relation to gender-based violence and the protection of minors. But the question I seek to answer here is not whether the Grok-generated content falls under one of these risk categories. It seems clear that X must have put in place measures to prevent the dissemination of much, if not all, of that content on the platform under Article 35 which governs the mitigation of risks. But how can we situate Grok the chatbot and the AI model behind within this regulatory landscape? Should the DSA’s systemic risk mitigation measures extend beyond the platform (X) to the chatbot (Grok)? The answer to this question hinges on whether Grok is a “related system” under Article 34(1).
Is Grok a “related system”?
The DSA’s systemic risk measures apply only to designated VLOPs and VLOSEs. At the time of writing, Grok is neither.(2) But it will still be caught under these measures to the extent it is a “related system” of the X platform which is a designated VLOP as Article 34(1) clearly covers both VLOPs/VLOSEs and their related systems.
The DSA does not define “related system”. Article 34 simply refers to “related systems, including algorithmic systems”. Recital 84 confirms that “When assessing such systemic risks, providers of very large online platforms and of very large online search engines should focus on the systems or other elements that may contribute to the risks, including all the algorithmic systems that may be relevant, in particular their recommender systems and advertising systems, paying attention to the related data collection and use practices.” As such, the DSA seems to cast the net rather wide in terms of what constitutes a “related system” that should be included in the systemic risk assessment. Generative AI bots are obviously “algorithmic systems”.
The “algorithmic systems” mentioned in the DSA pertain to the pre-generative AI world, with a focus on recommender systems (eg. recitals 84 and 88, Article 35(1)(d)) and they are merely examples. The DSA “does not exhaustively enumerate parts of the service that count as relevant risk surfaces” (Husovec, 2024)(3). As such, GPAI models can be seen as a novel risk surface that has emerged on VLOPs and VLOSEs with the evolution of the technological landscape. Recital 118 of the AI Act which addresses the interplay between the two regulations in relation to systemic risk management may provide a further clue. The recital indicates that so long as general-purpose AI (GPAI) systems or models are embedded into designated VLOPs or VLOSEs, they are subject to the risk-management framework provided for in the DSA.
Against this background, in the absence of a limiting definition of “related system”, Grok may well be considered a related system. Grok is embedded into X from the users’ perspective and probably also the provider’s perspective. On the user side, Grok is embedded seamlessly on the X platform where users may simply click on a tab and start engaging with the bot. Furthermore, users can directly post Grok-generated images on X. The existence of an additional @grok account gives users another means through which to engage with the chatbot on the platform. On the provider side, xAI – the developer of Grok – acquired X back in March 2025, and Musk himself announced at the time that “xAI and X’s futures are intertwined” and that “Today, we officially take the step to combine the data, models, compute, distribution and talent.” The overall picture is one of seamless integration.
If Grok the chatbot is a related system, then X’s risk assessment should logically extend to how the chatbot functions – including the model that generates the images – and how it contributes to systemic risks on the X platform. If the chatbot creates or exacerbates the systemic risks, which it arguably does, then X’s risk mitigation will need to extend to the model and the chatbot and include safeguards at the AI model and system levels, in addition to any changes to the X platform itself. Under Article 35(1)(d), mitigation measures include testing and adapting algorithmic systems.
All that said, there remains uncertainty as to what “related system” means and how it should be interpreted within the broader spirit of the DSA. The regulation was primarily designed with intermediaries and third-party content in mind. Content generated by the platforms themselves does not sit comfortably within this framework. Furthermore, GPAI models like those powering Grok are also subject to AI Act’s systemic risk management rules so long as they are classified as “GPAI model with systemic risk” under Article 51 of the AI Act. Recital 118 AI Act establishes a presumption of compliance with AI Act’s systemic risk provisions in cases where a GPAI with systemic risk is “embedded into” a VLOP or VLOSE “unless significant systemic risks not covered by [the DSA] emerge and are identified in such models.” This recital shows a clear intention to avoid enforcement duplication. But there also remains uncertainty in practice as to how the risk management frameworks of the two regulations will interact, not the least because “systemic risk” is framed differently in the DSA and the AI Act.(4)
This blog post was written on 14 January 2026 and concerns ongoing events. Content is based on current available data and may be subject to change as the situation evolves. The views and opinions expressed in this blog post are solely those of the author and do not reflect the official policy or position of any organisation.
Featured image credit: Reihaneh Golpayegani / https://betterimagesofai.org / https://creativecommons.org/licenses/by/4.0/
Footnotes
(1) The DSA can be seen as a tool of AI governance beyond its systemic risk provisions/VLOP and VLOSE obligations. For example, Article 25 which prohibits dark patters implicates AI system design on online platforms.
(2) There are rumours that the Commission may start designating generative AI chatbots, in particular ChatGPT, as VLOSE if they satisfy the Article 33 threshold, but there hasn’t been any designation so far.
(3) Martin Husovec (2024) Principles of the Digital Services Act, Oxford University Press, Chapter 15 Section 15.2.
(4) Commission Staff Working Document Accompanying the document “Report from the Commission to the European Parliament, the Council and the European Economic and Social Committee on the application of Article 33 of Regulation (EU) 2022/2065 and the interaction of that Regulation with other legal acts”, SWD/2025/368 final, pp. 162-163.