HTTPS surface reachable (robots ✓, sitemap ✗, title ✓)
Why it matters: Public files — robots.txt, sitemap.xml, head meta — are what attackers see first during reconnaissance. Misadvertised paths, stale sitemaps, and verbose generators leak more than intended (ISO 27001 A.8.9).
robots.txt
present
User-agent: Adsbot-Google
User-agent: Alexabot
User-agent: AppleBot
User-agent: archive.org_bot
User-agent: Assaybot
User-agent: baiduspider
User-agent: bitlybot
User-agent: bingbot
User-agent: BingPreview
User-agent: coccocbot
User-agent: Discordbot
User-agent: DuckDuckBot
User-agent: facebookexternalhit
User-agent: Google-Site-Verification
User-agent: Google-Sitemaps
User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-Mobile
User-agent: Googlebot-News
User-agent: Googlebot-Video
User-agent: gsa-crawler
User-Agent: HatenaBlog-bot
User-agent: ia_archiver
User-agent: Mediapartners-Google
User-agent: msnbot
User-agent: NAVER
User-agent: PetalBot
User-agent: Pingdom
User-agent: Pinterest
User-agent: redditbot
User-agent: Slackbot
User-agent: Slurp
User-agent: snapchat
User-agent: Twitterbot
User-agent: wp.com
User-agent: Yandex
User-agent: YandexImages
User-agent: YandexVideoParser
User-agent: Yeti
User-agent: ZoomBot
Disallow: /gp/
Disallow: /report_abuse.gne
Disallow: /abuse
Disallow: /images/*
Disallow: /apps/*
Disallow: /tools/demos/*
Disallow: /search
Disallow: /services/oauth
Disallow: /groups/10millionphotos/
Disallow: /photos/youpy/
Disallow: /photos/i_love_u_get_away_from_me/
Disallow: /faves-i_love_u_get_away_from_me/
Disallow: /photos/gbachelie/
Disallow: /photos/archivesact/6011019532/nearby/
Disallow: /yss_fragment.gne
Disallow: /photos/tags/*/page*
Disallow: /photos/*/lightbox/
Disallow: /groups/*/lightbox/
Disallow: /explore/*/lightbox/
User-agent: magpie-crawler
Disallow: /
User-agent: *
Disallow: /
Sitemap: https://www.flickr.com/sitemap/index/users/sitemap-index-users-00000000.xml.gz
Sitemap: https://www.flickr.com/sitemap/index/tags/sitemap-index-tags-00000000.xml.gz
Sitemap: https://www.flickr.com/sitemap/index/sets/sitemap-index-sets-00000000.xml.gz
Sitemap: https://www.flickr.com/sitemap/index/photos/sitemap-index-photos-00000000.xml.gz
Sitemap: https://www.flickr.com/sitemap/index/groups/sitemap-index-groups-00000000.xml.gz
head
- title
- Flickr | The best place to be a photographer online.
- description
- The nicest place on the internet is here for you: inspiration, community, creativity, art, passion, and a heaping scoop of weirdness await you. Join for free.