A Rota-Baxter Leibniz algebra is a Leibniz algebra $(\mathfrak{g},[~,~]_{\mathfrak{g}})$ equipped with a Rota-Baxter operator $T : \mathfrak{g} \rightarrow \mathfrak{g}$. We define representation and dual representation of Rota-Baxter Leibniz algebras. Next, we define a cohomology theory of Rota-Baxter Leibniz algebras. We also study the infinitesimal and formal deformation theory of Rota-Baxter Leibniz algebras and show that our cohomology is deformation cohomology. Moreover, We define an abelian extension of Rota-Baxter Leibniz algebras and show that equivalence classes of such extensions are related to the cohomology groups.

• 16S80 - Deformations of associative rings
• 17A32 - Leibniz algebras
• 17A36 - Automorphisms, derivations, other operators (nonassociative rings and algebras)
• 17B38 - Yang-Baxter equations and Rota-Baxter operators
• 17B56 - Cohomology of Lie (super)algebras