As a significant number of businesses both big and small are running WordPress installations, below is a typical example of a robots.txt file for WordPress to help illustrate how robots.txt functions:
User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Sitemap: https://example.com/sitemap_index.xml
This robots file allows all crawlers to find the /wp-content/uploads/ file which is great for indexing pages content and media. At the same time this has disallowed /wp-content/plugin/ and /wp-admin/ folders. Essentially, these pages are the administrative (either plugin components or user administration) sections and there is no need for the search engine to waste time exploring these folders or presenting these within the search engine index. Of note, we also see the inclusion of the address of the sitemap which surprisingly is a simple but continually overlooked operation.