# --- Previous configuration (commented out) --- # Allow Semrush Site Audit # User-agent: SemrushBot # User-agent: SiteAuditBot # Allow: / # # Block other bots # User-agent: * # Disallow: / # # Sitemap # Sitemap: https://thesector.mw-aws.com/sitemap.xml # --- Current configuration (for SEO audit) --- # Allow all crawlers for SEO audit User-agent: * Allow: / # Sitemap Sitemap: https://thesector.com.au/sitemap.xml