scrapeboard/robots.txt
Cocktail Frank 4804ab487d Increase join timeout, add new documentation and web page images
Increase join timeout to 10 seconds to give players more time to join a
networked match.

Add a robots.txt file that prevents crawling irrelevant files.

Add news and calendar dates to web page. Perform line length linting
on the HTML.

Add documentation screenshot, screen capture of VSW documentation, and
photos from alt.ctrl.GDC to doc folder.

Add low-res photos from TooManyGames, photos from DerpyCon, an
installation photo from MAGFest 2024, a photo from Chinatown Fair, and
a 2025 MAGFest header image to the web page resources.

Start a Press Kit file with basic information chart filled out.
2024-12-14 17:54:26 -05:00

14 lines
284 B
Plaintext

# Rules for Web crawlers to follow while crawling https://scrape.nugget.fun
#
# Allow main page access, www/, and library for fonts and JS
User-agent: *
Allow: /$
Allow: /favicon.ico
Allow: /*.php
Allow: /www/
Allow: /../../lib/
Allow: /feed
Disallow: /www/subscribe.php
Disallow: /