I would like to set up a custom domain for my Litmos account but the domain I would like to use is root URL (main) domain. (example.com)

Custom domain set-up fails for a root domain (example.com) because a CNAME cannot be created on the main domain. Use a subdomain (learning.example.com, university.example.com, academy.example.com) or www.example.com and point its CNAME to Litmos.