Ytukyg

I search a book where it is explained how the convolution between two distributions $T, S in mathcal{D}'(Omega)$ is defined.

Thank you in advance.

My recommendation is:

Walter Rudin, Functional Analysis. (Chapter 6).

Note that the convolution of $u,vinmathscr D'(mathbb R^n)$ is definable only if one of the has compact support. (See Definition 6.36, page 175.)

I believe this is even better than a book. https://mathoverflow.net/questions/5892/what-is-convolution-intuitively

You may not understand all of the answers but together they tell a great story. Near the bottom there is a link to an MIT open courseware lecture that motivates the convolution through Laplace transforms.

I advice you to have a look at the classical textbook of Vladimirov [1]: the convolution of distributions is defined in section 4 of chapter 2 (pp. 50-74), and several applications are given. As in Rudin's book, it is clearly stated that for the convolution $Sast T$ to exists in $mathscr{D}^prime(mathbb{R}^n)$ if $S$ and $T$ satisfy particular restrictions, for example $Tinmathscr{E}^prime(mathbb{R}^n)$ or $Sinmathscr{E}^prime(mathbb{R}^n)$: however, the Author explores other situations where the product is defined, by using mainly the method of integral transforms. The whole text is filled with many applications of the concept of convolution, dealing for example with topics as convolution algebras and their relations with algebras of holomorphic functions of several complex variables, not easily found in other introductory books on the subject (basically the whole chapter 5 is dedicated to this).

[1] Vladimirov, V. S. (2002), Methods of the theory of generalized functions, Analytical Methods and Special Functions, 6, London–New York: Taylor & Francis, pp. XII+353, ISBN 0-415-27356-0, MR 2012831, Zbl 1078.46029.