HTTPS surface reachable (robots ✓, sitemap ✗, title ✓)
Why it matters: Public files — robots.txt, sitemap.xml, head meta — are what attackers see first during reconnaissance. Misadvertised paths, stale sitemaps, and verbose generators leak more than intended (ISO 27001 A.8.9).
robots.txt
present
Sitemap: https://gizmodo.com/sitemap_index.xml
Sitemap: https://gizmodo.com/sitemap-news.xml
Sitemap: https://gizmodo.com/download/downloads.xml.gz
User-agent: *
Disallow: /download/*-thankyou$
Disallow: /download/ajax/
Disallow: /stats/
Disallow: /api/
Disallow: /ajax/
Disallow: /embed/
Disallow: /setbucket*
Disallow: /game/score/*
Disallow: /game/summary/*
Disallow: /gateway/*
Disallow: /showcase/*
Disallow: /search$
Disallow: /search?
Disallow: /?s=
Disallow: /search/
Disallow: /wp/wp-admin/
Allow: /wp/wp-admin/admin-ajax.php
# Allow Amazon Ad Bot
User-agent: AmazonAdBot
Allow: /
# Disallow AI training data collection
User-agent: AI2Bot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Cohere-ai
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: FirecrawlAgent
Disallow: /
User-agent: ImagesiftBot
Disallow: /
User-agent: Kangaroo Bot
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Omgilibot
Disallow: /
User-agent: PanguBot
Disallow: /
User-agent: Timpibot
Disallow: /
User-agent: Webzio-Extended
Disallow: /
# Disallow site scrapers
User-agent: HTTrack
Disallow: /
User-agent: Nutch
Disallow: /
User-agent: Scrapy
Disallow: /
head
- title
- Gizmodo | The Future Is Here
- description
- Dive into cutting-edge tech, reviews and the latest trends with the expert team at Gizmodo. Your ultimate source for all things tech.
social
- og:locale
- en_US
- og:type
- website
- og:title
- Gizmodo | The Future Is Here
- og:description
- Dive into cutting-edge tech, reviews and the latest trends with the expert team at Gizmodo. Your ultimate source for all things tech.
- og:url
- https://gizmodo.com/
- og:site_name
- Gizmodo
- og:image
- https://gizmodo.com/app/uploads/2025/10/gizmodo-social-1200x675-1.jpg
- og:image:width
- 1200
- og:image:height
- 675
- og:image:type
- image/jpeg
- twitter:card
- summary_large_image
- twitter:site
- @Gizmodo