User-agent: * Disallow: /wp-content/upgrade/ Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://www.heygirls.com/sitemap_index.xml User-agent: Screaming Frog SEO Spider Allow: / Disallow: /*.gif$ Disallow: /*.jpg$ Disallow: /*.png$ Disallow: /*.bmp$ Disallow: /*.xml$ Disallow: /*.css$ Disallow: /*.js$