HTTPS surface reachable (robots ✓, sitemap ✓, title ✓)
Why it matters: Public files — robots.txt, sitemap.xml, head meta — are what attackers see first during reconnaissance. Misadvertised paths, stale sitemaps, and verbose generators leak more than intended (ISO 27001 A.8.9).
robots.txt
present
User-agent: *
# Files
Disallow: /CHANGELOG.txt
Disallow: /cron.php
Disallow: /INSTALL.mysql.txt
Disallow: /INSTALL.pgsql.txt
Disallow: /INSTALL.sqlite.txt
Disallow: /install.php
Disallow: /INSTALL.txt
Disallow: /LICENSE.txt
Disallow: /MAINTAINERS.txt
Disallow: /update.php
Disallow: /UPGRADE.txt
Disallow: /xmlrpc.php
Disallow: /img/placeholder.gif
# Paths (clean URLs)
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /filter/tips/
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register/
Disallow: /user/password/
Disallow: /user/login/
Disallow: /user/logout/
Disallow: /internal-api/
# Paths (no clean URLs)
Disallow: /?q=admin/
Disallow: /?q=comment/reply/
Disallow: /?q=filter/tips/
Disallow: /?q=node/add/
Disallow: /?q=search/
Disallow: /?q=user/password/
Disallow: /?q=user/register/
Disallow: /?q=user/login/
Disallow: /?q=user/logout/
Disallow: /pugpig/
Disallow: /videorequest
Disallow: /api/
Disallow: /api/html/
Disallow: /api/user/
Disallow: /71347885/
Disallow: /cb
# Ignore liveblog pagination and swipe tracking
Disallow: *itm_channel=native
Disallow: *?page=
# Ignore refresh URLs
Disallow: /*ILC-refresh
User-agent: Nutch
Disallow: /
Sitemap: https://www.independent.co.uk/sitemaps/googlenews
Sitemap: https://www.independent.co.uk/sitemap.xml
sitemap.xml
present — 385 url(s)
head
- title
- The Independent | Latest news and features from US, UK and worldwide
- description
- Latest news, comment and features from The Independent US
social
- og:locale
- en_GB
- og:site_name
- The Independent
- og:type
- website
- og:image
- https://www.independent.co.uk/img/shortcut-icons/icon-96x96.png
- og:url
- https://www.independent.co.uk/us
- og:title
- The Independent | Latest news and features from US, UK and worldwide | The Independent
- og:description
- Latest news, comment and features from The Independent US
- twitter:card
- summary
- twitter:site
- @Independent
- twitter:image
- https://www.independent.co.uk/img/shortcut-icons/icon-96x96.png
- twitter:title
- The Independent | Latest news and features from US, UK and worldwide | The Independent
- twitter:description
- Latest news, comment and features from The Independent US