with N 4. We are concerned

Houston Journal of Mathematics c 6 University of Houston Volume 3, No. 4, 6 THE EFFECT OF THE OMAIN TOPOLOGY ON THE NUMBER OF POSITIVE SOLUTIONS OF AN ELLIPTIC SYSTEM INVOLVING CRITICAL SOBOLEV EXPONENTS PIGONG HAN Communicated by Giles Auchmuty Abstract. In this paper, we consider the irichlet problem for an elliptic system of two equations involving the critical Sobolev exponents. By means of the variational method, we study the effect of the domain topology on the number of positive solutions, prove the existence of at least cat () positive solutions. 1. Introduction Let be a smooth bounded domain in R N with the problem with N 4. We are concerned (1.1) u = v = α α + β uα 1 v β + λu in, β α + β uα v β 1 + µv in, u, v in, u = v = on, where λ >, µ > are parameters, α > 1, β > 1 satisfying α + β =, denotes the critical Sobolev exponent, that is, = N N. efinition 1. A pair of nonnegative functions (u, v) H 1 () H 1 () is said to be a weak solution of problem (1.1) if Mathematics Subject Classification. 35J6, 35J5, 35B33. Key words phrases. Elliptic system, Energy functional, (P.S.) c condition, Critical point, Critical Sobolev exponent. 141

14 P. HAN (1.) ( u ϕ 1 + v ϕ λuϕ 1 µvϕ )dx α α + β u α 1 v β ϕ 1 dx β u α v β 1 ϕ dx α + β = ϕ = (ϕ 1, ϕ ) H 1 () H 1 (). (1.3) The corresponding energy functional of problem (1.1) is defined by J λ,µ (u, v) = 1 α + β ( u + v λu µv )dx u α +v β +dx (u, v) H 1 () H 1 (), where u + = max{u, }. It is well known that the nontrivial solutions of problem (1.1) are equivalent to the nonzero critical points of J λ,µ in H 1 () H 1 (). Moreover, every weak solution of problem (1.1) is classical (see Remark 4 in [1]). In a recent paper, C. O. Alves et al [1] considered problem (1.1) generalized the results in [4] to the case of (1.1). There seems no progress on problem (1.1) since then. In this paper, we are interested in the effect of the domain topology on the number of positive solutions of problem (1.1). N N Let α = β =, λ = µ u = v, then problem (1.1) reduces to the scalar semilinear elliptic problem: u = u N+ N + λu in, (1.4) u in, u = on. O. Rey [9] studied the effect of the topology of the domain on the existence of solutions of problem (1.4) for N 5, proved that problem (1.4) has at least cat () distinct positive solutions for λ > small, where cat () denotes the Ljusternik-Schnirelman category of in itself ( see [11] for the definition). For N 4, M. Lazzo [8] obtained the same result. Relevant papers on this matter are [, 5, 6, 7, 11] the references therein. Set (1.5) S = inf u,v H 1 ()\{} ( u + v )dx ( u α v β dx ),

ELLIPTIC SYSTEM INVOLVING CRITICAL SOBOLEV EXPONENTS 143 then (see [1]) (1.6) S = (( ) β ( ) α ) α β + S, β α where S is the best Sobolev constant defined by S = u dx ( u dx ), inf u H 1 ()\{} which is achieved if only if = R N by U(x) = N (N(N )) 4 (1 + x ) N In the present paper, we first establish the concentration-compactness principle for elliptic systems; then by the variational method the category theory, we prove that problem (1.1) has at least cat () positive solutions for λ, µ > small. Let λ 1 be the first eigenvalue of the operator with zero irichlet boundary conditions. We state our main result as the following: 4mm Theorem 1.1. If N 4, then there exists < λ < λ 1 such that for any λ, µ (, λ ), problem (1.1) has at least cat () positive solutions. Throughout this paper, we denote the norm of the Banach space X by X, the positive constants (possibly different) by C... Proof of the main result Before giving the proof of Theorem 1.1, we introduce some notations preliminary lemmas. Lemma.1. Let R N (possibly unbounded) u n u, v n v weakly in H 1 (); u n u, v n v a.e in. Then lim u n u α v n v β dx n (.1) = lim u n α v n β dx u α v β dx. n

144 P. HAN Proof. It is not difficult to see that u n α v n β dx (.) where = α +β = α 1 1 1 u n u α v n v β dx u n tu α (u n tu)u v n β dxdt u n u α v n tv β (v n tv)vdxdt f n udxdt + β 1 g n vdxdt, f n = u n tu α (u n tu) v n β, g n = u n u α v n tv β (v n tv), t [, 1], since moreover, f n (1 t) α 1 u α u v β g n a. e on (, 1), ( C, ( C, 1 1 f n 1 1 g n we conclude that 1 dxdt ) α 1 ( u n tu 1 dxdt 1 dxdt f n (1 t) α 1 u α u v β Hence ) α ( u n u 1 dxdt 1 1 ) β v n 1 dxdt ) β 1 v n tv 1 dxdt g n weakly in L 1 ( (, 1)). (.3) α α 1 1 f n udxdt (1 t) α 1 u α v β dxdt = u α v β dx,

ELLIPTIC SYSTEM INVOLVING CRITICAL SOBOLEV EXPONENTS 145 (.4) β 1 g n vdxdt. Inserting (.3) (.4) into (.), we obtain (.1). To proceed, we need to generalize the concentration-compactness principle (see [3, 11]) to the case of systems. Lemma.. Let {(u n, v n )} 1, (R N ) 1, (R N ) be a sequence such that u n u, v n v weakly in 1, (R N ); u n u, v n v a.e on R N, (u n u) + (v n v) µ weakly in the sense of measures, define (.5) (.6) Then u n u α v n v β ν weakly in the sense of measures, µ = lim lim sup ( u n + v n )dx, R x R ν = lim R lim sup u n α v n β dx. x R (.7) (.8) = lim sup ( u n + v n )dx R N R N ( u + v )dx + µ + µ, lim sup u n α v n β dx = R N u α v β dx + ν + ν, R N (.9) ν S 1 µ, (.1) ν Moreover, if u v ν single point. S 1 µ. = S 1 µ, then µ ν concentrate at a

146 P. HAN Proof. Set w 1n = u n u, w n = v n v, then w 1n, w n weakly in 1, (R N ); w 1n, w n a.e on R N, w 1n + w n µ weakly in the sense of measures, w 1n α w n β ν weakly in the sense of measures. For any nonnegative function h C (R N ), by Lemma.1, we have lim h w 1n α w n β dx = lim h u n α v n β dx h u α v β dx. R N R N R N Hence we obtain (.11) u n + v n u + v + µ weakly in the sense of measures (.1) u n α v n β u α v β + ν weakly in the sense of measures. (a) (.13) For any h C (R N ), we infer that ( S 1 hw 1n α hw n β dx R N ) R N ( (hw 1n ) + (hw n ) )dx. Since w 1n w n strongly in L loc, we easily obtain from (.13) that (.14) x >R ( R N h d ν ) S 1 h d µ R N which implies (.9). (b) Since lim sup w 1n dx = lim sup u n dx u dx x >R x >R lim sup x >R w n dx = lim sup v n dx v dx, x >R x >R

ELLIPTIC SYSTEM INVOLVING CRITICAL SOBOLEV EXPONENTS 147 we deduce that (.15) x >R lim R lim sup ( w 1n + w n )dx = µ. x >R By Lemma.1, we have lim sup w 1n α w n β dx = lim sup So (.16) lim R x >R lim sup w 1n α w n β dx = ν. x >R u n α v n β dx u α v β dx. x >R Let R >, ψ R C (R N ) be such that ψ R (x) = for x < R; ψ R (x) = 1 for x > R + 1 ψ R (x) 1 on R N. Then we have ( R N ψ R w 1n α ψ R w n β dx ) S 1 ( (ψ R w 1n ) + (ψ R w n ) )dx. R N Since w 1n w n strongly in L loc, we infer that (.17) Observe that ( lim sup S 1 lim sup ψ R w 1n α ψ R w n β dx R N ) R N ( w 1n + w n )ψ Rdx. (.18) x >R+1 w in dx w in ψrdx R N w in dx, i = 1, x >R (.19) w 1n α w n β dx x >R+1 Thus, from (.15)-(.19), we get (.1). ψ R w 1n α ψ R w n β dx R N w 1n α w n β dx. x >R

148 P. HAN (c) From (.11) (.1), we deduce that lim sup ( u n + v n )dx R N = lim sup ψ R ( u n + v n )dx R N + lim sup (1 ψ R )( u n + v n )dx R N = lim sup ψ R ( u n + v n )dx R N + (1 ψ R )( u + v )dx + (1 ψ R )d µ. R N R N By the dominated convergence theorem, we obtain that lim sup ( u n + v n )dx = R N ( u + v )dx + µ + µ, R N lim R which is (.7). Similarly we get (.8). (d) Let u v ν we derive that for any h C (R N ) h d ν S R N = S 1 µ µ. By Hölder inequality (.14), N We deduce that ν = S µ N µ. Hence, from (.14) we obtain for any h C (R N ) µ N ( so for each open set Q R N, R N h d µ ) ( µ(r N )) N ( µ(q)) R N h d µ. R N h d µ, µ(q), which is equivalent to µ(r N ) µ(q). This proves that µ is concentrated at a single point. Suppose that (A) X is a Banach space, I C 1 (X, R), I d = {z X I(z) d}, ψ C (X, R), V = {z X ψ(z) = 1}, for every z V, ψ (z).

ELLIPTIC SYSTEM INVOLVING CRITICAL SOBOLEV EXPONENTS 149 We denote the tangent space of V at z by T z V = {y X ψ (z), y = } the norm of the derivative of I V at z V by (I V ) (z) := sup I (z), y. y T z V, y =1 The functional I V is said to satisfy the (P.S.) c condition if any sequence {z n } V such that I V (z n ) c, (I V ) (z n ) contains a convergent subsequence. In the sequel, we take X = H 1 () H 1 (), I V (u, v) = I(u, v) = (u + ) α (v + ) β dx, ψ(u, v) = ( u + v λu µv )dx λ, µ (, λ 1 ), V = { (u, v) H 1 () H 1 () ψ(u, v) = 1 }. Obviously, the assumption (A) is satisfied for our choices. Lemma.3. I satisfies (P.S.) c condition for any c (S N, ) on V. Proof. Let {(u n, v n )} V satisfy I(u n, v n ) c, I (u n, v n ). Then, there exists a sequence {t n } R such that as n N α(u n ) α 1 + (v n ) β + + β(u n ) α +(v n ) β 1 + t n ( un v n λu n µv n ) strongly in H 1 () H 1 (). So (α + β)i(u n, v n ) t n, then t n ()c >.

15 P. HAN efine w 1n = ( ) N 4 4t n u n, w n = ( ) N 4 4t n v n, we obtain 1 ( w 1n + w n λw1n µw n)dx (w 1n ) α α + β +(w n ) β +dx = 1 ( ) N α + β ( u n + v n λu n µv 4t n)dx n ( ) N α + β (u n ) α α + β 4t +(v n ) β +dx n 1 N ( 1 c ) N w 1n w n λw 1n µw n α α + β (w 1n) α 1 + (w n ) β + β α + β (w 1n) α +(w n ) β 1 + strongly in H 1 () H 1 (). So {(w 1n, w n )} X is a (P.S.) c sequence of J λ,µ, which is defined in (1.3). It follows easily that (w 1n, w n ) X C. Thus, up to a subsequence, we may assume that (w 1n, w n ) (w 1, w ) weakly in H 1 () H 1 (); (w 1n, w n ) (w 1, w ) a.e on. Then (w 1, w ) is a weak solution of problem (1.1) (.) = J λ,µ (w 1, w ) ( 1 1 ) α + β ( w1 + w λw1 µw ) dx. Setting w 1n = w 1n w 1, w n = w n w, we have w in dx = w in dx w i dx + o(1), i = 1,.

ELLIPTIC SYSTEM INVOLVING CRITICAL SOBOLEV EXPONENTS 151 By Lemma.1, we also have ( w 1n ) α +( w n ) β +dx = (w 1n ) α +(w n ) β +dx (w 1 ) α +(w ) β +dx + o(1). Hence ( w 1n + w n )dx ( w 1n ) α +( w n ) β +dx as n. We may assume that as n ( w 1n + w n )dx a, ( w 1n ) α +( w n ) β +dx a, where a is a nonnegative number. If a =, the proof is complete. Assume a >, since ( ) S ( w 1n ) α +( w n ) β +dx ( w 1n + w n )dx, we get S ( a (.1) ) a, then a ( S On the other h, by (.) we have ) N. a a α + β = 1 α + β ( w 1n + w n )dx = J λ,µ ( w 1n, w n ) + o(1) ( w 1n ) α +( w n ) β +dx + o(1) = J λ,µ (w 1n, w n ) J λ,µ (w 1, w ) + o(1) J λ,µ (w 1n, w n ) + o(1) which contradicts (.1). = 1 ( 1 N c < N ( S ) N ) N, The following lemma follows from [1].

15 P. HAN Lemma.4. Let N 4, λ, µ (, λ 1 ). Then m(λ, µ, ) := sup (u,v) V Moreover, m(λ, µ, ) is achieved on V. efine H(u, v) = then we have the following I(u, v) > S N N. x ( u + v λu µv ) dx (u, v) V, Lemma.5. If {(u n, v n )} V satisfies lim (u n) α +(v n ) β +dx = S N N, then (.) lim dist(h(u n, v n ), ) =. Proof. Since {(u n, v n )} V, it is easy to verify that (u n, v n ) is bounded in H 1 () H 1 (). Thus, up to a subsequence, we may assume that (u n ) + u (v n ) + v weakly in H 1 (), ((u n ) + u) + ((v n ) + v) µ weakly in the sense of measures, (u n ) + u α (v n ) + v β ν weakly in the sense of measures. Since is bounded, Lemma. implies (.3) (.4) (.5) 1 = ( u + v )dx + µ, S N N = S ν u α v β dx + ν, µ.

ELLIPTIC SYSTEM INVOLVING CRITICAL SOBOLEV EXPONENTS 153 If ( u + v )dx µ =, we deduce that ( ) 1 = ( u + v )dx + µ > ( S = S = 1, ( u + v )dx N S N ) u α v β dx + S ν + µ which is a contradiction. Thus, ( u + v )dx = or µ =. If µ =, from (.3)- (.5), we get ( u + v )dx = 1 u α v β dx = S N N. Then R ( u + v )dx ( R u α v dx) β = S, which means that S is achieved by (u, v). It is impossible since S cannot be attained on any bounded domain. Hence, (.6) ( u + v )dx = µ = 1. Then, u v on, so lim n we easily have ν single point x. Thus H(u n, v n ) = ( λu n + µv n) dx =. From (.3), (.4), = 1 = S 1 µ. By Lemma., µ is concentrated at a x( u n + v n λu n µvn)dx xd µ = x. Without loss of generality, we may assume choose r > small enough such that B r (), such that + r = {x R N dist(x, ) r} r = {x dist(x, ) r}

154 P. HAN are homotopically equivalent to. efine m(λ, µ, r) := m(λ, µ, B r ()) > S N N, recall I d = {z H 1 () H 1 () I(z) d}. Then we have Lemma.6. If N 4. Then there exists < λ < λ 1 such that for any λ, µ (, λ ), (.7) cat Im(λ,µ,r) (I m(λ,µ,r) ) cat (). Proof. We first show that there exists < λ < λ 1 such that for any λ, µ (, λ ), if (u, v) I m(λ,µ,r), then H(u, v) + r. In fact, set λ (, λ 1 ), λ, µ (, λ ), from the proof s process of (.), we know that there is a positive number ɛ (independent of λ, µ) such that (u, v) V, N u α +v+dx β S N + ɛ = H(u, v) + r. Choosing λ = { ( )} λ, λ 1 1 (1 + ɛs ) (, λ1 ), for any λ, µ (, λ ) (u, v) I m(λ,µ,r), we obtain u α +v β +dx S ( ( S λ1 S ( u + v )dx ) ) ( v µv )dx ( u λu )dx + λ 1 λ 1 λ λ 1 µ ( λ 1 ( u + v λu µv )dx λ 1 max{λ, µ} ( ) < S λ1 λ 1 λ = S N N + ɛ. So that H(u, v) + r. efine γ : r I m(λ,µ,r) by (u λ,µ ( x y ), v λ,µ ( x y )) x B r (y), γ(y)(x) = x \B r (y), )

ELLIPTIC SYSTEM INVOLVING CRITICAL SOBOLEV EXPONENTS 155 where (u λ,µ ( x ), v λ,µ ( x )) is a positive function, radially symmetric about the origin, such that (see [1]) B r () B r () u α λ,µv β λ,µ dx = m(λ, µ, r) ( u λ,µ + v λ,µ λu λ,µ µv λ,µ)dx = 1. It is not difficult to check that H γ = id. Let n = cat Im(λ,µ,r) (I m(λ,µ,r) ), then there exist n closed, contractible sets {A i } (1 i n) in I m(λ,µ,r) n corresponding mappings h i C([, 1] A i, I m(λ,µ,r) ) (1 i n) such that I m(λ,µ,r) = n A i for any (u, v), (ū, v) A i i=1 h i (, (u, v)) = (u, v), h i (1, (u, v)) = h i (1, (ū, v)). Set B i = γ 1 (A i ) (1 i n). Then the sets B i are closed r = n B i. Let g i (t, x) = H(h i (t, γ(x))), then g i C([, 1] r, + r ) for any x, y r i=1 g i (, x) = H(h i (, γ(x))) = H(γ(x)) = x g i (1, x) = H(h i (1, γ(x))) = H(h i (1, γ(y))) = g i (1, y). So B i (1 i n) is contractible in r. Therefore, cat () = cat + r ( r ) n. Proof of Theorem 1.1. It is not difficult to check that m(λ, µ, r) < m(λ, µ, ). By Lemmas.3.4, I satisfies the (P.S.) c condition for any c [m(λ, µ, r), m(λ, µ, )]. Let J(u) = I(u) J d = {z H 1 () H 1 () J(z) d}. Then J is bounded below on V. Since (J V ) = (I V ) for every u V, it follows that J V satisfies the (P.S.) c condition for any c m(λ, µ, r), u is a critical point of I V if only if it is a critical point of J V. Hence, Lemma.6 Theorem 5. in [11] yields that I m(λ,µ,r) = J m(λ,µ,r) contains at least cat () critical points of I V, denoted by (u i, v i ) (1 i cat ()) satisfying

156 P. HAN u i = αt i α + β (u i) α 1 + (v i ) β + + λu i in, v i = βt i α + β (u i) α +(v i ) β 1 + + µv i in, u i = v i = on, 1 where t i = I(u i,v i ) > is a Lagrange multiplier. Since λ, µ (, λ ), by the strong maximum principle, (u i, v i ) > in, 1 i cat (). Let ũ i = t N 4 i u i, ṽ i = t N 4 i v i. Then (ũ i, ṽ i ) (1 i cat ()) is a positive solution of problem (1.1). Acknowledgements The author would like to thank the anonymous referee for carefully reading this paper suggesting many useful comments. References [1] C. O. Alves,. C. de Morais Filho M. A. S. Souto, On systems of elliptic equations involving subcritical or critical Sobolev exponents, Nonlinear Anal, TMA., 4 (), 771-787. [] V. Benci G. Cerami, The effect of the domain topology on the number of positive solutions of nonlinear elliptic problems, Arch. Rational Mech. Anal., 114 (1991), 79-93. [3] G. Bianchi, J. Chabrowski A. Szulkin, On symmetric solutions of an elliptic equation with a nonlinearity involving critical Sobolev exponent, Nonlinear Anal, TMA., 5 (1995), 41-59. [4] H. Brezis L. Nirenberg, Positive solutions of nonlinear elliptic equations involving critical Sobolev exponent, Comm. Pure Appl. Math., 36 (1983), 437-478. [5]. Cao J. Chabrowski, On the number of positive solutions for nonhomogeneous semilinear elliptic problem, Advs. iff. Equats., 1 (1996), 753-77. [6] A. Castro M. Clapp, The effect of the domain topology on the number of minimal nodal solutions of an elliptic equation at critical growth in a symmetric domain, Nonlinearity, 16 (3), 579-59. [7]. Cao, G. Li X. Zhong, A note on the number of positive solutions of some nonlinear elliptic problems, Nonlinear Anal, TMA., 7 (1996), 195-118. [8] M. Lazzo, Solutions positives multiples pour une équation elliptique non linéaire avec l exposant critique de Sobolev, C. R. Acad. Sci. Paris 314 (199), 61-64. [9] O. Rey, A multiplicity result for a variational problem with lack of compactness, Nonlinear Anal, TMA., 13 (1989), 141-149. [1] W. C. Troy, Symmetry properties in systems of semilinear elliptic equations, J. ifferential Equations, 4 (1981), 4-413. [11] M. Willem, Minimax Theorems, PNLE 4, Birkhäuser, Boston-Basel-Berlin 1996.

ELLIPTIC SYSTEM INVOLVING CRITICAL SOBOLEV EXPONENTS 157 Received March 3, 4 Revised version received ecember 8, 4 Institute of Applied Mathematics, Academy of Mathematics Systems Science, Chinese Academy of Sciences, Beijing 18, P. R. China E-mail address: pghan@amss.ac.cn