Vjylku

I suggest: "Vanity" describes either Adam's sin [ the first recorded sin] or God's reaction to that sin, the curse of Gen 3.

The A.V. always translates "mataiotei" as "vanity". It comes in three places:

1) In Eph 4:17 vanity, the hardness of human hearts not the ground, sounds like a human attitude, sin, rather than God's response to sin.

2) In 2 Peter 2:18 again sounds like sin rather than God's response.

3) In Romans 8:20 it is more ambiguous. Hence the reason for this question.

In 1 Peter 1:18 we are not redeemed from the curse but from the our vain [mataios] conduct which brought the curse.

In Romans 1:24 God did give them over to uncleanness, but, before that happened they were already vain. Rom 1:21. Mataios again seen as a starting point.

God's cursing is not vain. Nothing God does is vain.

The curse did not render Adam and Eve's efforts futile, but difficult.

What is vain/futile is that man should not obey God's spoken commands as in Gen 2:16-17 and other commands e.g. the Ten Commandments.

If I am right that "vanity" is sin and not God's response to sin, does, "for the creature was made subject to vanity" challenge this conclusion?

The Westminster Confession of Faith says that God is not the author of sin thus making something without Him a first cause (see John 1:3). God is holy and uncreated, sin is created. The uncreated does not become created by creating that which is other than Himself, namely, sin.

Although, as you note, ματαιότης (mataiotēs) appears only 3 times in the New Testament, it appears over 50 times in the Septuagint. The meaning of the word generally means something like futility, pointless existence, or, as one Eastern Orthodox commentator describes it, "a departure from the norm or from God's will for man and his life."¹. I think this meaning is captured our expression "in vain" (e.g. "He did it in vain"). Examples from the Septuagint include:

Ῥήματα Ἐκκλησιαστοῦ υἱοῦ Δαυιδ βασιλέως Ισραηλ ἐν Ιερουσαλημ. Ματαιότης ματαιοτήτων, εἶπεν ὁ Ἐκκλησιαστής, ματαιότης ματαιοτήτων, τὰ πάντα ματαιότης.

The words of the Preacher, the son of David, king of Israel in Jerusalem. Vanity of vanities, said the Preacher, vanity of vanities; all is vanity (Ecclesiastes 1:1-2, Brenton translation)

υἱοὶ ἀνθρώπων, ἕως πότε βαρυκάρδιοι; ἵνα τί ἀγαπᾶτε ματαιότητα καὶ ζητεῖτε ψεῦδος;

O ye sons of men, how long will ye be slow of heart? wherefore do ye love vanity, and seek falsehood? (Psalm 4:2 LXX)

οὐκ ἐκάθισα μετὰ συνεδρίου ματαιότητος καὶ μετὰ παρανομούντων οὐ μὴ εἰσέλθω,

I have not sat with the council of vanity, and will in nowise enter in with transgressors (Psalm 25:4 LXX)

The other key word here is κτίσις (ktisis), which your title translates as creature, but was more widely understood (at least in antiquity) to mean creation. This would be consistent with how modern versions translate the word and is probably the meaning intended by the KJV (AV) translators as well. According to the Oxford English Dictionary, the word "creature" in Elizabethan times meant anything created, not necessarily something animate. Take, for example, the Coverdale version of 2 Peter 3:4 (c. 1535):

For sence the fathers fell on slepe, euery thinge contynueth as it was from the begynnynge of ye creature

The understanding of Romans 8:20 in antiquity was that man's transgression corrupted (made vain) not only himself but all creation. The explanation offered by John Chrysostom in the 4th century was:

What is the meaning of, “the creation was made subject to vanity?” Why that it became corruptible. For what cause, and on what account? On account of thee, O man. For since thou hast taken a body mortal and liable to suffering, the earth too hath received a curse, and brought forth thorns and thistles.²

^{1.Archbishop Dmitry Royster, St. Paul's Epistle to the Romans: A Pastoral Commentary (St. Vladimir's Seminary Press, 2008), p.210.
2.Homily XIV on Romans (tr. from Greek; Nicene and Post-Nicene Fathers series, 1.11}