You really are making your life complex.
Note your question
Will SME11 support DNSSEC?
was already answered....
I did a bit of research
If you mean used AI to fill in the gaps in your knowledge because you don't like the answers you were given then see previous replies about using AI to find answers on SME.
I'll précis it.
Don't.
It will hallucinate and use generic, incorrect, answers. YMMV.
and found that Rocky Linux 8 can support DNS over HTTPS and DNS over TLS which is probably all that is really needed to overcome the problem I was having.
So what DNS system does SME use? (Clue - not BIND which is slated for 12)
So as previously mentioned likely it won't.... You'll just break your network.
Of the two it looks like DNS over TLS will be the easiest to implement since it looks like a matter of creating one setting.
If you think it is that simple then open a NFR bug, after searching for dns, bind etc, and then add your simple patch.
Bear in mind if it
was that simple we would have already done it........
Do you know what other services depend on DNS?
From the internet result I found:I'm guessing
Stop guessing. Do proper research and don't use AI.
Otherwise you will likely break your server, and lookups.
In the absence of SME doing DNSSEC we have told you your options.
(Not sure on the sudden hurry right now as DNSSEC & DoH has been around for years but we are few, get almost no help (but users expect us to help them), and have to make priorities and choices about what we build)
Your time would be better spent with a Rocket account where you can learn, help, and understand.
Then you won't become and XY problem.