There are three possible reasons:
  1. Attitude of the person giving the offering
  2. Type of Offering – one was a blood sacrifice and the other was an offering of grain/fruit
  3. Quality of Offering

Let’s see how the Scriptures recount this incident:
Genesis 4:2-5
And she again bare his brother Abel. And Abel was a keeper of sheep, but Cain was a tiller of the ground.
And in process of time it came to pass, that Cain brought of the fruit of the ground an offering unto the LORD.
And Abel, he also brought of the firstlings of his flock and of the fat thereof. And the LORD had respect unto Abel and to his offering:
But unto Cain and to his offering he had not respect. And Cain was very wroth, and his countenance fell

It is interesting to note that, when God initially intended all humans to eat only a vegetarian diet (Gen 1:29). At this time, meat was not yet permitted to be eaten. It was only after the flood that God allowed people to eat meat (Gen 9:3).
Therefore Abel was rearing sheep not for meat, but most probably for clothing, and for sacrificial offerings.

We know this because after Adam and Eve sinned, they immediately became aware of their nakedness, and attempted to cover themselves with fig leaves:

Genesis 3:7
And the eyes of them both were opened, and they knew that they were naked; and they sewed fig leaves together, and made themselves aprons.
Garments in the Bible refer to righteousness. The act of covering themselves with fig leaves depict their own self-righteousness which was not acceptable to the Lord. To atone for their sins, a blood sacrifice was required. In fact, the Hebrew word for atonement (kaphar) means a covering.

In the Bible, self righteousness is compared to filthy rags (Isaiah 64:6)

The only acceptable atonement is a blood sacrifice:
Hebrews 9:22
And almost all things are by the law purged with blood; and without shedding of blood is no remission

Therefore, God’s plan of redemption is that of a substitutionary blood sacrifice. This was indicated when God gave Adam and Eve garments of skin:
Genesis 3:21
Unto Adam also and to his wife did the LORD God make coats of skins, and clothed them.

Now remember, the Old Testament is but a picture, a shadow of things to come. Therefore the sacrifice of animals was but a picture of the sacrifice of the Son of God, Jesus, on the cross of Calvary.

Even though the Levitical law of sacrifices had not yet been written during Abel’s day, it is most probable that he was aware of the instituted method of atonement and reconciliation. This must have been communicated either through his parents or by God Himself.

This instruction was the Word on how to give an acceptable sacrifice to the Lord. And it is the Word that brings faith.
We can connect the following Scriptures together:

Hebrews 11:4
By faith Abel offered unto God a more excellent sacrifice than Cain, by which he obtained witness that he was righteous, God testifying of his gifts: and by it he being dead yet speaketh.

Notice that Abel brought a more excellent sacrifice and not that he had a better attitude. It was by faith, that Abel brought a more excellent sacrifice.

And, we know that:
Romans 10:17
So then faith cometh by hearing, and hearing by the word of God.

So, Abel ‘heard’ the word of God, and had faith that it is only a substitutionary blood sacrifice that can redeem mankind. In other words, he was looking forward to the cross.

Cain, on the other hand did not believe the Word of God. He had his own way – presumptuous thinking

Jude 1:11
Woe unto them! for they have gone in the way of Cain, and ran greedily after the error of Balaam for reward, and perished in the gainsaying of Core.

Also, God is not a respecter of persons. There is an ordained way – through the Cross. Jesus is the way.
It is not man’s way. It is God’s way. A person may be sincere, but can be sincerely mistaken.

God is the same yesterday, today and forever. His plan of redemption through the blood of Christ was conceived even before He created mankind. That’s the beauty about the Bible – it is consistent, and the answers usually are found from Scripture itself.