The Bible itself sets a precedent for rejoicing in Scripture with singing. The longest psalm, Psalm 119, is about delighting in the beauty and perfection of God’s Word.
Your testimonies are wonderful;
Therefore my soul keeps them.
The entrance of Your words gives light;
It gives understanding to the simple.
In this psalm the psalmist continues for 176 verses; he cannot praise God enough for His Word.
It is right and good to rejoice in God’s Word. “All Scripture is breathed out by God and profitable for teaching, for reproof, for correction, and for training in righteousness, that the man of God may be complete, equipped for every good work” (2 Timothy 3:16–17). It alone is sufficient to guide us in life. It alone certainly and infallibly points us to Christ. The 1689 London Baptist Confession begins with the declaration: “The Holy Scripture is the only sufficient, certain, and infallible rule of all saving knowledge, faith, and obedience.”
Here are some hymns that I would recommend that celebrate the beauty and sufficiency of Scripture:
How Firm a Foundation (from John Rippon’s Selection of Hymns, 1787)
1. How firm a foundation, ye saints of the Lord,
Is laid for your faith in His excellent Word!
What more can He say than to you He hath said,
To you who for refuge to Jesus have fled?
[Baptist Hymnal, 2008 #456]
Holy Bible, Book Divine (John Burton, Sr., 1803)
1. Holy Bible, Book divine,
Precious Treasure, thou art mine:
Mine to tell me whence I came;
Mine to teach me what I am.
[Baptist Hymnal, 2008 #345]
Thy Word Is Like a Garden, Lord (Edwin Hodder, 1863)
1. Thy Word is like a garden, Lord,
With flowers bright and fair;
And every one who seeks may pluck
A lovely cluster there.
Thy Word is like a deep, deep mine;
And jewels rich and rare
Are hidden in its mighty depths
For every searcher there.
[Trinity Hymnal—Baptist edition, 1995 #450]
How Shall the Young Direct Their Way? (from Psalm 119:9–16; The Psalter, 1912)
1. How shall the young direct their way?
What light shall be their perfect guide?
Thy Word, O Lord, will safely lead,
If in its wisdom they confide.
Sincerely I have sought Thee, Lord,
O let me not from Thee depart;
To know Thy will and keep from sin
Thy Word I cherish in my heart.
2. O blessed Lord, teach me Thy Law,
Thy righteous judgments I declare;
Thy testimonies make me glad,
For they are wealth beyond compare.
Upon Thy precepts and Thy ways
My heart will meditate with awe;
Thy Word shall be my chief delight,
And will not forget Thy Law.
[Trinity Hymnal—Baptist edition, 1995 #264]
Sufficient Is the Word of God (Ken Puls, 2000)
(words ©2000 Kenneth Puls)
1. Sufficient is the Word of God
That we may know His will,
And trust each day His providence
By faith content and still.
The secret things belong to God;
Our lives are His to lead.
But He has given us His Word
And that is all we need.
2. How vain the questions in my heart
To know what lies ahead,
When I neglect to read His Word
And know not what He said.
I need not look beyond His Word
To know His will for me;
If I but walk within its light
My path I’ll plainly see.
We Have an All-Sufficient Word (Ken Puls, 1994)
(words ©1998 Kenneth Puls)
1. We have an all-sufficient Word
To make the simple wise;
Upon the heart the Spirit writes
And souls from death arise.
2. With such a mighty, wielded Sword,
What more could saints require
To fight the darkness of their sin,
And warn men of hell fire?
The Spirit Breathes Upon the Word (William Cooper, 1779)
1. The Spirit breathes upon the Word,
And brings the truth to sight;
Precepts and promises afford
A sanctifying light.
2. A glory gilds the sacred page,
Majestic like the sun:
It gives a light to every age;
It gives but borrows none.
[Trinity Hymnal—Baptist edition, 1995 #258]
Father of Mercies in Thy Word (Anne Steele, 1760)
1. Father of mercies, in Thy Word
What endless glory shines;
Forever be Thy name adored
For these celestial lines.
2. Here may the wretched sons of want
Exhaustless riches find;
Riches above what earth can grant
And lasting as the mind.
[Trinity Hymnal—Baptist edition, 1995 #259]
God in the Gospel of His Son (Benjamin Beddome, 1787)
1. God in the gospel of His Son,
Makes His eternal counsels known;
Where love in all its glory shines,
And truth is drawn in fairest lines.
[Trinity Hymnal—Baptist edition, 1995 #262]
The Heavens Declare Thy Glory, Lord (Isaac Watts, 1719, from Psalm 19)
1. The heavens declare Thy glory, Lord;
In every star Thy wisdom shines;
But when our eyes behold Thy Word,
We read Thy name in fairer lines.
6. Thy noblest wonders here we view
In souls renewed, and sins forgiven:
Lord, cleanse my sins, my soul renew,
And make Thy Word my guide to heaven.
[Trinity Hymnal—Baptist edition, 1995 #263]
O Spirit Now We Thank You (Ken Puls,1996)
(words ©1998 Kenneth Puls)
1. O Spirit, now we thank You
For giving us Your Word.
Please bless its proclamation,
The truths that we have heard.
Indwell us and empow’r us,
And cause us to obey;
Shine now the light of Scripture
On all we do and say.
2. Great Artist of the Scriptures,
In beauty You have made
God’s Word to shine in glory
That cannot fail or fade.
In poetry and proverbs,
Through narrative and line;
In prophecy and hist’ry,
God’s truth in splendor shines.
[Psalms, Hymns and Spiritual Songs, 2004 (Founders Press) #63]
Many more could be added to this list. What would you include as fitting songs to thank God for giving us His unfailing and unfading Word?