Plugin Manager Plugin which generates a robots.txt file, based on DenySpider attributes on classes or methods within controllers. If UserSessionMiddleware.Plugin is also installed, will check to see if a bot is trying to access a page it has been denied, and return a 403 forbidden result.
$ dotnet add package Spider.PluginThis package is one of many packages that can be used with Plugin Manager which can be used to extend any c#/.net based application (MVC, Winform, WPF, MAUI etc) by using a Modular Approach.
Spider plugin has been designed to manage robots that visit a site, it completes this by performing 3 tasks
This is achieved by adding DenySpiderAttribute to a controller class or action method, the following code sample demostrates 2 action methods that are denied to bots:
[DenySpider]
[Breadcrumb(nameof(Languages.LanguageStrings.Privacy))]
public IActionResult Privacy()
{
return View(new BaseModel(GetBreadcrumbs(), GetCartSummary()));
}
[DenySpider("*")]
[ResponseCache(Duration = 0, Location = ResponseCacheLocation.None, NoStore = true)]
public IActionResult Error()
{
return View(new ErrorViewModel { RequestId = Activity.Current?.Id ?? HttpContext.TraceIdentifier });
}