User-agent: Screaming Frog SEO Spider Allow: / User-agent: * Disallow: /*? Disallow: /profile Disallow: /au/profile Disallow: /users/* Disallow: /au/users/* Disallow: /sports/ Host: https://justcasinoio.com