arxiv: v1 [math.ap] 22 Mar 2017

Transcript

1 CRITICAL SYSTEM INVOLVING FRACTIONAL LAPLACIAN arxiv: v1 [math.ap] Mar 017 MAODING ZHEN, JINCUN HE, AND HAOYUAN XU Abstract. In this paper, we discuss the existence of solutions for the following non-local critical systems: ( ) s u = µ 1 u u+ αγ u α u v β in, ( ) s v = µ v v + βγ u α v β v in, u,v D s ( ). By using the Nehari manifold, under proper conditions, we establish the existence nonexistence of positive least energy solution. 1. introduction Equations systems, involving laplacian have been studied for many years are applied to many subjects, such as physical, chemical materials science. Recently, a great attention has been focused on the study of equations systems, involving fractional laplacian (see [3,4,5,8,10,11, 1,15,16,18,19,] references therein). For the case of single equation, E. Colorado, A. de Pablo U. Sánchez, in [3] studied the following nonhomogeneous single equation involving fractional laplacian, {( ) s u = λu q +u N+s N s in Ω, u = 0 on Ω, proved the existence multiplicity of solutions for suitable conditions depended on s q. Furthermore in [4], they studied the following nonhomogeneous fractional equation involving critical Sobolev exponent, { ( ) s u = u u+f(x) in Ω, u = 0 on Ω, proved the existence multiplicity of solutions under appropriate conditions on the size of f. In [5], F.Tao, X.Wu studied the existence multiplicity of positive solutions for fractional schödinger equations 000 Mathematics Subject Classification. 47G0, 35J50, 35B65. Key words phrases. Fractional laplacian;nehari manifold; least energy solution. Revised on March 3,

2 M. ZHEN, J. HE, AND H. XU with critical growth. L.Caffarelli, L. Silvestre in [10] studied an extension problem related to the fractional laplacian, by given a new formulation of the fractional laplacian, this idea was used in recent articles (see [1,17] references therein). For more details about the results of single fractional equation, see [7,19]. Systems involving fractional laplacian in a bounded domain of have also been studied in recently years. In [1], Z.Chen, W.Zou have studied the positive least energy solution, phase separation for coupled systems schrödinger equation with critical exponent. Futhermore, Z.Chen, W.Zou in [0] gave the optimal constant for the existence of least energy solution of a coupled schrödinger system. By using of the idea of s-harmonic, Q.Wang in [1] studied the positive least energy solutions of fractional laplacian systems with critical exponent. X.He, M.Squassina W.Zou in [17] have studied the Nehari manifold for fractional laplacian systems involving critical nonlinearities, obtained the least energy solutions. For coupled systems involving fractional laplacian. Z.Guo W.Zou in [] showed that the positive solution has to be radially symmetric respect to some points in decays at infinity with certain rates. For the case of systems involving fractional laplacian in. X.Zhang, J.Wang in [18] studied the symmetry results for systems involving fractional laplacian. In this paper, we study the following fractional laplacian system with critical exponent. ( ) s u = µ 1 u u+ αγ u α u v β in, (1.1) ( ) s v = µ v v + βγ u α v β v in, u,v D s ( ), where, the fractional laplacian ( ) s is defined by ( ) s u(x) u(y) u(x) = C(N,s)P.V. R y x N+s)dy, N with 1 cos(ς) C(N,s) = ( dς) 1 = s π N Γ( N+s ) R ς N N+s Γ( s) s(1 s), 0 < s < 1, µ 1,µ > 0, = N N s, N > s, α,β > 1, α+β =. In this paper, by giving the range of γ the relation between N s, we get the existence nonexistence of least energy solution of (1.1). The existence of positive least energy solution of (1.1) is

3 CRITICAL SYSTEM INVOLVING FRACTIONAL LAPLACIAN 3 strongly depended on the following system. µ 1 k + αγ k α l β = 1, (1.) µ l + βγ k α l β = 1, k,l > 0. Our main results are the following: Theorem 1.1. If γ < 0, then A = s N (µ N s s 1 +µ N s s )S N s s A is not attained. Theorem 1.. If s N 4s, α,β > (1.3) 0 < γ or N > 4s, 1 < α,β < (1.4) γ 4Ns (N s) min{µ 1 α (α )β µ, β β (α )α }, β 4Ns (N s) max{µ 1 α ( β) β µ, α β ( α) α }, β then A = s N (k 0 +l 0 )S N s s, A is attained by ( k 0 U ε,y, l 0 U ε,y ), where (k 0,l 0 ) satisfies (1.) k 0 = min{k,(k,l) satisfies (1.)}. Theorem 1.3. Assume, N > 4s 1 < α,β < hold, there exists a γ 1 (0, 4Ns (N s) max{µ 1 α ( β) β µ, α β ( α) α ), β such that, for any γ (0,γ 1 ), there exists a solution (k(γ),l(γ)) of (1.), satisfying E(( k(γ)u ε,y, l(γ)u ε,y ) > A, ( k(γ)u ε,y, l(γ)u ε,y ) is a positive solution of (1.1). Remark 1.1. Before presenting our next result, we would like to mention that the following fractional Brezis-Nirenberg problem ( ) s u+λu = u u, x Ω, u = 0, x Ω, has a positive least energy solution U λ for λ s (Ω) < λ < 0 (see[]). Consider the following system: ( ) s u+λ 1 u = µ 1 u u+ αγ u α u v β in Ω, (1.5) ( ) s v+λ v = µ v v + βγ u α v β v in Ω, u = 0,v = 0 on Ω,

4 4 M. ZHEN, J. HE, AND H. XU where Ω is an open bounded domain in, λ 1,λ ( λ s (Ω),0), here λ s (Ω) is the first eigenvalue of ( ) s with Dirichlet boundary condition. By the idea of s-harmonic extension in [10] the arguements in [1], we can show the following result: If λ s (Ω) < λ 1 = λ = λ < 0, s N 4s, α,β > 0 < γ 4Ns (N s) min{µ 1 α (α )β µ, β β (α )α }, β or λ s (Ω) < λ 1 = λ = λ < 0, N > 4s, 1 < α,β < γ 4Ns (N s) max{µ 1 α ( β) β µ, α β ( α) α }, β then ( k 0 U λ, l 0 U λ ) is a positive least energy of (1.5), where (k 0,l 0 ) satisfies (1.) k 0 = min{k,(k,l) satisfies (1.)}. The paper is organized as follows. In section we introduce some preliminaries some Lemmas that will be used to prove Theorems 1.1, 1., 1.3. In section 3 we give two Propositions. In section 4 we prove Theorem 1.1. In section 5 we prove Theorem 1. in section 6 we prove Theorem Some preliminaries Let D s ( ) be Hilbert space as the completion of Cc ( ) equipped with the norm u D s( ) = C(N,s) RN u(x) u(y) dxdy, y x N+s u D (.1) S s = inf s( ) u D s( )\0 (, u dx) be the sharp imbedding constant of D s ( ) L ( ) S s is attained only on. By [3], S s is attained by ũ = κ(ε + x x 0 ) N s, that is Let S s = ũ D s( ) ( ũ dx). u = ũ, ũ

5 CRITICAL SYSTEM INVOLVING FRACTIONAL LAPLACIAN 5 by Lemma.1 in [], u µ = ( S S µ ) 1 u is a positive ground state solution of (.) ( ) s u = µ u u, in U ε,y = (S s ) 1 u is a positive ground state solution of (.3) ( ) s u = u u, in, with (.4) U ε,y D s( ) = U ε,y dx = (S s ) N s. The energy function associated with (1.1) is given by E(u,v) = 1 ( u D s( ) + v D s( ) ) 1 µ 1 u +µ v +γ u α v β dx. Define the Nehari manifold N = {(u,v) D s ( ) D s ( ) : u 0,v 0, u D s( ) = µ 1 u + αγ R N u α v β dx, v D s( ) = µ v + βγ R N u α v β dx}. A = inf E(u,v) = inf (u,v) N (u,v) N = inf (u,v) N s N ( u D s( ) + v D s( ) ) s µ 1 u +µ v +γ u α v β dx. N If (u,v) (0,0) u 0,v 0, then we claim (u,v) is a nontrivial solution of (1.1). Obviously, N, any nontrivial solution of (1.1) is in N. Lemma.1. If (u,v) N is any nontrivial solution of (1.1), then Proof. If u D s( ) > 0, v D s( ) > 0. u D s( ) = 0, v D s( ) = 0, then by the definition of S s, we have u dx S s ( u D s( ) ) = 0.

6 6 M. ZHEN, J. HE, AND H. XU Thus u 0,v 0, a.e. x, which contradicts to (u,v) (0,0). The following Lemma. show if the energy functional attained its minimum at some points (u,v) N, then (u,v) is a solution of (1.1). The main idea is use lagrange multiplier method to prove Lemma.. Lemma.. Assume γ < 0, if A is attained by a couple (u,v) N, then (u,v) is a solution of (1.1). Proof. Define N 1 = {(u,v) D s ( ) D s ( ) : u 0,v 0 G 1 (u,v) = u D s( ) µ 1 u + αγ R N u α v β dx = 0}, N = {(u,v) D s ( ) D s ( ) : u 0,v 0 G (u,v) = u D s( ) µ 1 u + αγ R N u α v β dx = 0}. Then N = N 1 N. By the definition of Fréchet derivative, we have E (u,v)(ϕ 1,ϕ ) = (u,ϕ 1 )+(v,ϕ ) µ 1 u uϕ 1 +µ v vϕ αγ α u α uϕ 1 v β +β u α v β vϕ dx, G 1 (u,v)(ϕ 1,ϕ ) = (u,ϕ 1 ) R µ 1 u uϕ 1 dx N αγ α u α uϕ 1 v β +β u α v β vϕ dx, G (u,v)(ϕ 1,ϕ ) = (v,ϕ ) R µ v vϕ dx N βγ α u α uϕ 1 v β +β u α v β vϕ dx, where (u,ϕ 1 ) = C(N,α) (u(x) u(y))(ϕ 1 (x) ϕ 1 (y)) y x N+s dxdy.

7 Then CRITICAL SYSTEM INVOLVING FRACTIONAL LAPLACIAN 7 E (u,v)(u,0) = G 1 (u,v) = 0, E (u,v)(0,v) = G (u,v) = 0, G 1(u,v)(u,0) = u D s( ) µ 1 u dx αγ α u α v β dx R N R N = ( αγ ) µ 1 u dx+( α) R N u α v β dx, G 1(u,v)(0,v) = v D s( ) µ v dx βγ β u α v β dx R N R N = ( βγ ) µ v dx+( β) R N u α v β dx, G 1 (u,v)(0,v) = αγ β u α v β dx > 0, R N G (u,v)(u,0) = βγ α u α v β dx > 0. Suppose that (u,v) N is a minimizer for E restricted to N, by the stard minimization theory, there exist two lagrange multipliers L 1,L R such that, E (u,v)+l 1 G 1 (u,v)+l G (u,v) = 0. Thus we have L 1 G 1 (u,0)+l G (u,0) = 0, L 1 G 1 (0,v)+L G (0,v) = 0, G 1 (u,v)(u,0)+g 1 (u,v)(0,v) = ( ) u D s( ) 0, G (u,v)(u,0)+g (u,v)(0,v) = ( ) v D s( ) 0. By Lemma.1, we have hence Define the matrix u D s( ) > 0, v D s( ) > 0, G 1 (u,v)(u,0) = G 1 (u,v)(u,0) > G 1 (u,v)(0,v), G (u,v)(0,v) = G (u,v)(0,v) > G (u,v)(u,0). M = ( G 1 (u,v)(u,0), G (u,v)(u,0) G 1 (u,v)(0,v), G (u,v)(0,v) ),

8 8 M. ZHEN, J. HE, AND H. XU then det(m) = G 1 (u,v)(u,0) G (u,v)(0,v) G 1 (u,v)(0,v)g (u,v)(u,0) > 0, which means L 1 = L = 0, that is E (u,v) = 0. Define functions F 1 (k,l) = µ 1 k + αγ k α l β 1 k > 0,l 0, F (k,l) = µ l + βγ k α l β (.5) 1 k 0,l > 0, l(k) = ( ) α β αγ k β (1 µ1 k ) β 0 < k µ k(l) = ( ) αl β α (1 µ βγ l ) α 0 < l µ, then F 1 (k,l(k)) 0, F (k(l),l) 0. 1, Remark.1. If (k,l) satisfies (1.), then ( ku ε,y, lu ε,y ) is a solution of (1.1), where U ε,y satisfy (.3) (.4). Hence the main work is to establish the conditions for the existence of solutions of (1.). In order to prove the existence of solution of (1.), we present the following Lemma.3. Lemma.3. Assume that N > 4s,1 < α,β <,γ > 0, then (.6) F 1 (k,l) = 0, F (k,l) = 0 k,l > 0, has a solution (k 0,l 0 ) such that (.7) F (k,l(k)) < 0 k (0,k 0 ), where (k 0,l 0 ) satisfies k 0 = min{k : (k,l) satisfies (1.)}. Similarly, (.6) has a solution (k 1,l 1 ), such that (.8) F 1 (k(l),l) < 0 l (0,l 1 ). Proof. By F 1 (k,l) = 0 k,l > 0, then l = l(k) k (0,µ 1 ). Substituting this into F (k,l) = 0, we have (.9) ( αγ ) α µ ( αγ )α β (1 µ1 k ) α βγ ( )α β + k β β k ( )( β) β (1 µ 1 k ) β β = 0.

9 Let CRITICAL SYSTEM INVOLVING FRACTIONAL LAPLACIAN 9 f(k) =µ ( αγ )α β (1 µ1 k ) α βγ ( )α β + k β ( αγ ) α β k ( )( β) β (1 µ 1 k ) β β, then (.9) has a solution is equivalent to f(k) = 0 has a solution in (0,µ 1 ). Since 1 < α,β <, we get that lim k 0 +f(k) =, f(µ 1 ) = βγ µ α β 1 > 0, then using the intermediate value theorem, we know that, there exists k 0 (0,µ 1 ) s.t. f(k 0 ) = 0, f(k) < 0 k (0,k 0 ). Let l 0 = l(k 0 ), then (k 0,l 0 ) is the solution of (.6) satisfies (.7). Remark.. It is easy to see, if ( k 0 U ε,y, l 0 U ε,y ) N, then A E( k 0 U ε,y, l 0 U ε,y ) = s N (k 0 +l 0 )S N s s. In order to prove A s N (k 0 + l 0 )S N s s, we give the following Lemma.4 Lemma.5 Lemma.4. Assume N > 4s,1 < α,β < (1.3) holds, then l(k)+k is strictly increasing on [0,µ 1 ], Proof. Since k(l)+l is strictly increasing on [0,µ ]. l (k) = ( αγ ) α β (k µ 1 k β β ) β = ( µ 1 αγ ) α k k (l) = ( βγ ) α = ( µ βγ ) α l β (µ 1 α (l β µ l α ) α α (µ 1 k α µ 1 β k β ) α β ( 1 k ) β α β ( µ 1 β k ), β µ α ( l β α lα ) l ) α β α ( µ α l ),

10 10 M. ZHEN, J. HE, AND H. XU hence l (( α µ 1 β ) ) = l (µ 1 ) = 0, k (( β µ α ) ) = k (µ ) = 0 l (k) > 0 k (0,( α µ 1 β ) ), l (k) < 0 k (( α µ 1 β ),µ 1 ), k (l) > 0 l (0,( β µ α ) ), k (l) < 0 l (( β µ α ),µ ). Since l (k) = β β ( µ 1 αγ ) β α β k β (µ 1 1 k ) β β [( α µ 1 β k ) (µ 1 1 k )( α( α) µ 1 β( β) k )], k (l) = α α ( µ βγ ) α β α l α (µ 1 l ) α α [( β µ α l ) (µ 1 l )( β( β) µ α( α) l )], we have l ( α) (k) = 0 k = ( µ 1 β(4 ) ) k1 (( α µ 1 β ),µ 1 ), k ( β) (l) = 0 l = ( µ α(4 ) ) l1 (( β µ α ),µ ). Then by (1.4) we obtain Hence l (k) min = l (k 1 ) = ( ( )µ 1 ) β β ( ) β β 1, αγ α k (l) min = k (l 1 ) = ( ( )µ ) α α ( ) α α 1. βγ β l (k) > 1, k (0,µ 1 ),

11 CRITICAL SYSTEM INVOLVING FRACTIONAL LAPLACIAN 11 k (l) > 1, l (0,µ ), which means l(k)+k is strictly increasing on [0,µ 1 ], k(l)+l is strictly increasing on [0,µ 1 ]. Lemma.5. Assume N > 4s, 1 < α,β < (1.3) hold, let (k 0,l 0 ) be the same as in Lemma.3. Then (.10) (k 0 +l 0 ) max{µ 1,µ } < 1 (.11) F (k,l(k)) < 0, k (0,k 0 ); F 1 (k(l),l) < 0, l (0,l 0 ). Proof. Since by Lemma.4, we have, That is Similarly Hence µ k 0 < µ 1, 1 = l(µ 1 )+µ 1 l(k 0 )+k 0 = l 0 +k 0. µ 1 (k 0 +l 0 ) 1. µ (k 0 +l 0 ) 1. (k 0 +l 0 ) max{µ 1,µ } < 1. To prove (.11) by Lemma.3, we only needs to show that (k 0,l 0 ) = (k 1,l 1 ). By (.7),(.8), we have l 0 l 1, k 1 k 0. Suppose by contradiction that k 1 > k 0, then l(k 1 )+k 1 > l(k 0 )+k 0, hence l 1 +k(l 1 ) = l(k 1 )+k 1 > l(k 0 )+k 0 = l 0 +k(l 0 ). Since k(l)+l is strictly increasing for l [0,µ ], therefore l 1 > l 0, which contradicts to l 0 l, then we have Similarly k 1 = k 0. l 1 = l 0.

12 1 M. ZHEN, J. HE, AND H. XU Define A ǫ = inf (u,v) N ǫ E ǫ (u,v) E ǫ (u,v) = 1 (u,v) D s (µ 1 u ǫ +µ v ǫ +γ u α ǫ v β ǫ ) B 1 N ǫ = {(u,v) D s (B 1 )\{(0,0)} : G ǫ (u,v) = (u,v) D s (µ 1 u ǫ +µ v ǫ +γ u α ǫ v β ǫ ) = 0} B 1 The following Lemma.6 was given in []. Lemma.6. [] If u µ = ( S S µ ) 1 u is a positive ground state solution of (.), then A < min{ inf E(u,0), inf E(0,v)}, (u,v) N (u,v) N = min{e(u µ1,0),e(0,u µ )}, = min{ s N µ N s s 1 S N s s, s N µ N s s A ǫ = A = E(u,v), S N s s } 3. two propositions Proposition 3.1. Assume c,d R satisfy (3.1) µ 1 k µ l k,l > 0. + αγ k α + βγ l β 1, k α l β 1, If s < N < 4s, α,β > (1.3) hold, then c + d k + l, where k,l R satisfy (1.).

13 CRITICAL SYSTEM INVOLVING FRACTIONAL LAPLACIAN 13 Proof. Let y = c+d,x = c d,y 0 = k +l,x 0 = k l we have Then, Let f 1 f y (x+1) µ 1 x + αγ x α y (x+1) µ + βγ x α x α f 1 (x),y 0 = f 1 (x 0 ), f (x),y 0 = f (x 0 ). 4 αγ(x+1) x α 4 ( )µ 1 (x) = x β (µ 1 x + αγ +βx (α )], ) [ αγ 4 βγ(x+1) α (x) = (µ + βγ x α )xα αx + ( )µ ]. ) [(β βγ g 1 (x) = ( )µ 1 x β +βx (α ), αγ g (x) = (β )x α α αx + ( )µ, βγ then, g 1(x) = 0 ( )µ 1 β xβ +β = 0, αγ αγ x = ( ) ( β x1, )µ 1 g 1 (x) > 0 x < ( αγ ) ( β, )µ 1 g 1(x) αγ < 0 x > ( ) ( )µ 1 g (x) = 0 x = α β x, g (x) > 0 x > α β, g α (x) < 0 x < β. β,

14 14 M. ZHEN, J. HE, AND H. XU Therefore, g 1 (x) is increasing in (0,( αγ ( )µ 1 ) αγ ( )µ 1 ) β ) is decreasing in (( β,+ ), g (x) is decreasing in (0, α ) is increasing in β ( α,+ ). β Hence αγ g 1 (x) max = g 1 (x 1 ) = (β )( ) ( β (α ), )µ 1 g (x) min = g (x ) = ( α )α + ( )µ. β βγ Then by (1.3), we have g 1 (x) max 0, g (x) min 0. Therefore, f 1 (x) is strictly decreasing in (0,+ ), f (x) is strictly increasing in (0,+ ). Hence we have y max{f 1 (x),f 1 (x)}, min(max{f 1 (x),f 1 (x)}), = min (max{f 1(x),f 1 (x)}) = y f 1 (x)=f (x) 0, which means c+d k +l. Proposition 3.. Assume N > 4s,1 < α,β < (1.3) holds, then k +l k 0 +l 0, (3.) F 1 (k,l) 0, F (k,l) 0, k,l > 0, (k,l) (0,0), has an unique solution (k,l) = (k 0,l 0 ). Proof. Obviously (k 0,l 0 ) satisfies (3.). Suppose k, l is another solution of (3.). Without loss of generality, assume that k > 0, then l > 0. In fact, if l = 0, then k k0 +l 0, therefore F 1 ( k,0) = µ 1 k 1 0, 1 µ 1 k µ 1 (k 0 +l 0 ),

15 CRITICAL SYSTEM INVOLVING FRACTIONAL LAPLACIAN 15 which contradicts with Lemma.5. In the following, we prove that k = k 0. Suppose by contradiction that k < k 0, by the proof of Lemma.4, we have k(l) is strictly increasing on (0,( β ) µ α ), strictly decreasing on (( β ) µ α,µ ). On the one h, since therefore, there exist l 1,l satisfing such that k(0) = k(µ ) = 0, 0 < k < k 0, 0 < l 1 < l < µ, k(l 1 ) = k(l ) = k (3.3) F ( k,l) < 0 k < k(l) l 1 < l < l. Futhermore we have F 1 ( k, l) 0, F ( k, l) 0, l > l( k), l l1 or l l. By (.11), we see F ( k,l( k)) < 0, by (3.3) we get that, l 1 < l < l, therefore l l. On the other h, let l 3 = k 0 +l 0 k, then l 3 > l 0 k(l 3 )+k 0 +l 0 k = k(l 3 )+l 3 > k(l 0 )+l 0 = k 0 +l 0, that is k(l 3 ) > k. By (3.3) we have l 1 < l 3 < l. Since we obtained that k + l k 0 +l 0, l k0 +l 0 k = l 3 < l. This contradicts to l l, the proof completes.

16 16 M. ZHEN, J. HE, AND H. XU Proof. For any (u,v) N, u D s( ) = Therefore Similarly Hence 4. proof of theorem 1.1 µ 1 u R N µ 1 u u D s( ) µ N s s 1 S N s s. v D s( ) µ N s s S N s s. + αγ u α v β dx dx µ 1 S s ( u D s( ) ). A s N ( u D s( ) + v D s( ) (µ N s s 1 +µ N s s )S N s By lemma.1 in [], we have is the solution of the equation satisfies w µi = ( S S µ i ) 1 u, ( ) s u = µ i u u in w µi = (µ i ) s N 4s, ( ) s u = µ i u u in i = 1,. Let e 1 = (1,0, 0), (U R (X),V R (X)) = (w µ1 (x), w µ (x + Re 1 ), where R is a positive constant. Since V R (x) L ( ) is a solution of ( ) s u = µ i u u in, i = 1,, we have, V R (X) 0 in L ( ) as R +, hence lim URV α β Rdx = lim R + R + lim ( R + (β 1) 1 R URV α α 1 R V dx, V R dx) α 1 ( U 1 R Therefore, for R > 0 sufficiently large, the system s. VR dx) β

17 CRITICAL SYSTEM INVOLVING FRACTIONAL LAPLACIAN 17 (4.1) U R D s(r N ) = µ 1 U R dx = t R V R D s( ) = µ V R dx = s R has a solution (t R,s R ) with Hence µ 1 U R dx+t R α s R β µ V R α dx+t R s R β lim ( t R 1 +s R 1) = 0 R + ( t R U R, S R V R ) N. A = inf (u,v) N E(u,v) E( t R U R, S R V R ), = s N (t R U R D s( ) )+s R V R D s( ) ), αγ U α R V β R dx, αγ U α R V β R dx, ( ) s N (t Rµ N s s 1 U ε,y D s( ) +s Rµ N s s U ε,y D s( ) ). Let R +, we get Therefore A s N (µ N s s 1 +µ N s s )S N s (4.) A = s N (µ N s s 1 +µ N s s )S N s Suppose that A is attained by some (u,v) N, then E(u,v) = A. By Lemma., we know (u,v) is a nontrivial solution of (1.1). By Strong maximum principle for fractional laplacian( Proposition.17 in[9], Lemma 6 in [15]) comparison principle in [13], we may assume that u > 0,v > 0 u α v β dx > 0, then u D s( ) µ 1 u dx µ 1 S s ( u D s( ) ). Hence u D s( ) µ N s s 1 S N s s. Similarly v D s( ) µ N s s S N s s. s. s.

18 18 M. ZHEN, J. HE, AND H. XU Then A = E(u,v) = s N ( u D s( ) + v D s( ) ) s N (µ N s s 1 +µ N s s )S N s which contradicts to (4.). Finally we prove ( ) Proof. Let U α R V β R dx µ 1 U R dx = a, we have a 0 as R sufficiently large. Then (4.1) has a solution is equivalent to that the following system has a solution at the neighbourhood of point (1, 1). {k + αγ k α l β a = 1, l + βγ k α l β a = 1, then using Taylor expansion at (1,1), we have (1+ (k 1)+O((k 1) )) + αγ (1+ α (k 1)+O((k 1) ))(1+ β (l 1)+O(l 1) )a = 1, (1+ (l 1)+O((l 1) )) + βγ (1+ α(k 1)+O((k 1) ))(1+ β (l 1)+O((l 1) ))a = 1. Therefore Then let k 1 = αγ ( α +O((k 1) +(l 1) ), l 1 = βγ ( α +O((k 1) +(l 1) ). αγ )(k 1)a ( β αγ )(l 1)a ( )a )(k 1)a βγ ( β )(l 1)a βγ ( )a a 1 = αγ (α ),b 1 = αγ ( β ),c 1 = αγ ( )a, a = βγ ( α ),b = βγ (β ),c = βγ ( )a, we have that { k 1 = a 1 a(k 1)+b 1 a(l 1)+c 1 +O((k 1) +(l 1) ), (4.3) l 1 = a a(k 1)+b a(l 1)+c +O((k 1) +(l 1) ). Let x = ( k 1 l 1 ), s,

19 then CRITICAL SYSTEM INVOLVING FRACTIONAL LAPLACIAN 19 c = ( ) a1 a b B = 1 a, a a b a ( c1 +O((k 1) +(l 1) ) c +O((k 1) +(l 1) ) x = Bx+c T(x). We claim that, T is a contraction mapping. In fact ), T(x 1 ) T(x ) B x 1 x, where B = λ, λ is maximum eigenvalue of B T B, since a 0, there exists 0 < α < 1, s.t., B α. By mapping contraction theorem, we get that the systems (.1) has a solution (t R,s R ) with lim R + ( t R 1 +s R 1) = 0. Proof. By Remark., we have 5. proof of theorem 1. (5.1) A E( k 0 U ε,y, l 0 U ε,y ) = s N (k 0 +l 0 )S N s s. Let (u n,v n ) N be a minimizing sequence for A, that is E(u n,v n ) A as n. Define c n = ( u n dx), dn = ( v n dx), then S s c n u n D s( ) = µ 1 u n + αγ R u n α v N n β dx, µ 1 (c n ) + αγ ( µ 1 (c n ) S s d n v n D s( ) = µ (d n ) u n ) α ( v n ) β, αγ (c n) α (dn ) β, µ 1 v n + αγ R u n α v N n β dx, βγ (c n) α (dn ) β.

20 0 M. ZHEN, J. HE, AND H. XU Dividing both side of inequality by S s c n S s d n. Let we get that is c n = c n µ 1 c n µ c n S s + αγ + βγ, dn = d n S s β α c n dn 1, c n α β dn, 1, F 1 ( c n, d n ) 0, F ( c n, d n ) 0. Consquently, when s < N < 4s, α,β >, Proposition 4.1 ensures that Therefore c n + d n k +l = k 0 +k 0. c n +d n (k 0 +l 0 )S s = (k 0 +l 0 )S N s s s Since (u n,v n ) N is a minimizing sequence for A, we have (5.) E(u n,v n ) = s N ( u n D s( ) + v n D s( ) ). By the definition of S s (.1) (5.) we have, S s (c n +d n ) ( u n D s( ) + v n D s( ) ) = N s E(u n,v n ) = N s A+O(1). Since we have that is thus Hence S s (c n +d n ) (k 0 +l 0 )S N s s, A s N (k 0 +l 0 )S N s s, (k 0 +l 0 )S N s s S s (c n +(d n ) (k 0 +l 0 )S N s s, c n +d n (k 0 +l 0 )S s A = lim n + E(u n,v n ) lim n + as n +, s N S s(c n +d n ) s N (k 0 +l 0 )S N s A = s N (k 0 +l 0 )S N s s = E( k 0 U ε,y, l 0 U ε,y ). s.

21 CRITICAL SYSTEM INVOLVING FRACTIONAL LAPLACIAN 1 6. proof of theorem 1.3 Proof. To obtain the existence of (k(γ), l(γ) for γ > 0, we define functions F 1 (k,l,γ) = µ 1 k + αγ k α β l 1 k,l > 0, let then Denote F (k,l,γ) = µ l + βγ k α l β 1 k,l > 0, D = k(0) = µ 1, l(0) = µ, F 1 (k(0),l(0),0) = F (k(0),l(0),0) = 0, k F 1 (k(0),l(0),0) = µ 1 k 4 > 0, l F 1 (k(0),l(0),0) = k F (k(0),l(0),0) = 0, l F (k(0),l(0),0) = µ l 4 > 0. ( k F 1 (k(0),l(0),0), l F 1 (k(0),l(0),0) k F (k(0),l(0),0), l F (k(0),l(0),0) since det(d) > 0, by implicit function theorem, we see that k(γ), l(γ) are well defined of class in C 1 in ( γ,γ ), for some γ > 0 F 1 (k(γ), l(γ), γ) = F (k(γ), l(γ), γ) = 0. Thus ( k γ U ε,y, l γ U ε,y ), is a solution of (1.1). Since lim γ 0 N s s (k(γ)+l(γ)) = k(0)+l(0) = µ there exists γ 1 (0,γ ) such that By Lemma 6, we get ), 1 +µ N s s, k(γ)+l(γ) > min{µ N s s 1,µ N s s } γ (0,γ 1 ). E(( k γ U ε,y, l γ U ε,y )) = s N (k(γ)+l(γ))s N s s, > min{ s N µ N s s 1 S N s s, s N µ N s s > A ǫ = A = E(u,v). S N s s } that is ( k γ U ε,y, l γ U ε,y ) is another positive solution of (1.1).

22 M. ZHEN, J. HE, AND H. XU References [1] A.Capella, J.Pavila, L.Dupaigne Y.Sire, Regularity of radial extremal solutions for some non-local semilinear equations, Comm.Part Diff.Equal., (011) [] B.Barrious, E.Colorado, A.de Pablo U.Sánchez, On some critical problems for the fractional Laplacian operator, J.Differential Equations, (01) [3] E.Colorado, A.de Pablo U.Sánchez, On some critical problems for the fractional Laplacian operator, J.Differential Equations, (01) [4] E.Colorado, A.de Pablo U.Sánchez, Perturbation of a critical fractional equations, Pacific J. Math., (014) [5] F.Tao, X.Wu, Existence multiplicity of positive solutions for fractional schodinger equations with critical growth, Nonliner. Analysis, (017) [6] G.A,cotsiolis, N.K.Tavoularis, Best constant for sobolev inequality for higher order fractional derivatives, J.Math.Anal.Appl., (004) 5-36 [7] J.Tan, The Brézis-Nirenberg type problem involving the square root of the Laplacian, Calc.Var. Partical Differential Equations, (011) [8] J.Marcos, D.Ferraz, Concentration-compactness principle for nonlocal scalar field equations with critical growth, J.Math.Anal.Appl., (017) [9] L.Silvestre, Regularity of the obstacle problem for a fractional power of the Laplacian operator, Comm.Pure Appl.Math., (007)67-11 [10] L.Caffarelli, L. Silvestre, An extension problem related to the fractional Laplacian, Comm.Partial Differential Equations, (007) [11] L.Caffarelli, J.Roquejoffre Y.Sire, Variational problems for free boundaries for the fractional laplacian, J.Eur.Math.Soc., (010) [1] Q.Wang, Positive least energy solutions of fractional laplacian systems with critical exponent, Electronic Journal of Differential Equations, (016)1-16. [13] Ros-Oton, Xavier, Nonlocal elliptic equations in bounded domains:a survey.publ.mat., (016)3-6. [14] R.Servadei, E.Valdinoci, The Brezis-Nirenberg result for the fractional laplacian, Trans.Amer.Math.soc., (015) [15] R.Servadei, E.Valdinoci, Weak viscosity solution of the fractional Laplace equation, Publ.Mat., (014) [16] X.Cabre, Y.Sure, Nonlinear equations for fractional laplacian,regularity Maximum principle Hamiltonian, Ann.Inst.Henri poincare.nonlincaire, (014)3-53. [17] X.He, M.Squassina W.Zou, The Nehari manifold for fractional laplacian systems involving critical nonlinearities, mathematical, 015 [18] X.Zheng, J.Wang, Symmetry results for systems involving fractional Laplacian,India.Journal of Pure Applied Mathematics, (014)39-5 [19] X.Shang, J.Zhang Y.Yang, Positive solutions of nonhomogeneous fractional Laplacian problem with critical exponent,comm. Pure Appl.Anal., (014) [0] Z.Chen, W.Zou, An optimal constant for the existence of least energy solution of a coupled Schrodinger system, Cale.var Partial Differential Equation, (013) [1] Z.Chen, W.Zou, Positive least energy solutions phase seperation for coupled Schrodinger equations with critical exponent, Arch.Ration.Mech.Anal., (01)

23 CRITICAL SYSTEM INVOLVING FRACTIONAL LAPLACIAN 3 [] Z.Guo, W.Zou, On critical systems involving fractional laplacian, J.Math.Anal.Appl., (017) M. Zhen School of Mathematics statistics, Huazhong University of Sciences Technology, Wuhan , China. address: d @hust.edu.cn. J. He School of Mathematics statistics, Huazhong University of Sciences Technology, Wuhan , China. address: taoismnature@hust.edu.cn. H. Xu School of Mathematics statistics, Huazhong University of Sciences Technology, Wuhan , China. address: hyxu@hust.edu.cn.