The ministry of Jesus recorded in the Gospels corrects many false notions, both of His disciples and current generation. The correction for the false myths of the 2nd and 3rd century B.C.E. regarding these Nephilim were addressed by Paul in 1Tim. 1.4: nor to devote themselves to myths and endless genealogies, which promote speculations rather than the stewardship from God that is by faith (ESV), (see also 1Tim. 4.7). To another of his proteges, Paul specifies exactly what he is referencing: not devoting themselves to Jewish myths and the commands of people who turn away from the truth (Tit. 1.14 ESV).

The Nephilim seem to be the offspring of the sons of God and daughters of men, but it is not entirely clear. One thing is clear, the Nephilim were humans and not some kind of angel/human composite: The Nephilim were on the earth in those days, and also afterward, when the sons of God came in to the daughters of man and they bore children to them. These were the mighty men who were of old, the men of renown (Gen. 6.4 ESV).

I view this text in Genesis 6.1-4 as an extension of Gen. 3.15: And I will put enmity between you and the woman, And between your seed and her Seed;
He shall bruise your head, And you shall bruise His heel (NKJV). The world was divided initially along lines of these two peoples: the sons and daughters of God and the sons and daughters of (fallen) man. The populations were separated initially along redemptive lines similar to how Israel was separate from the nations. It seems that the males of the sons of God initiated a break in faith and joined themselves to some daughters of men and the resulting offspring exhibited a natural phenomenon known as hybrid vigor.

This effect manifests itself when two separate (inbred) genetic populations are combined to produce superior size or growth. Since the whole of the human gene pool was contained in two separate peoples during this time, the effect (hybrid vigor) should have been stronger but also some variation occurring in the union. One thing is almost certain: not all of the offspring were giants. Those normal-sized offspring would still carry genes for gigantism which is later expressed in subsequent generations. This is why there were Nephilim both before and after the flood, probably by the time of Noah many in the general population carried the genetics. 

The two populations may have become morphologically distinct

They were human giants according to Num. 13.33: And there we saw the Nephilim (the sons of Anak, who come from the Nephilim), and we seemed to ourselves like grasshoppers, and so we seemed to them (ESV).

Whatever they were, it seems they didn’t have special spiritual ability since other humans conquered them: In the fourteenth year Chedorlaomer and the kings who were with him came and defeated the Rephaim in Ashteroth-karnaim, the Zuzim in Ham, the Emim in Shaveh-kiriathaim, and the Horites in their hill country of Seir as far as El-paran on the border of the wilderness (Gen. 14.5 ESV). In an parenthetical remark concerning the peoples of the east during Israel’s Wilderness Wanderings, Moses notes the human history: (The Emim formerly lived there, a people great and many, and tall as the Anakim. Like the Anakim they are also counted as Rephaim, but the Moabites call them Emim. The Horites also lived in Seir formerly, but the people of Esau dispossessed them and destroyed them from before them and settled in their place, as Israel did to the land of their possession, which the LORD gave to them.) (Dt.2.10-12 ESV).