Methanol Web Crawling System 1.7.0

Methanol 1.7.0

This manual is for the Methanol Web Crawling System, version 1.7.0.

Understanding and running Methanol

• Introduction: Introduction to Methanol.
• Methanol Programs: Get introduced to the programs included in the Methanol suite.
• Installation: How to install Methanol and its programs.
• Post-Installation Setup and Testing: Set Up and Test the Server Environment.

Customization

• Configuration Files: Basic to advanced configuration file topics.
• Parsers: Extract and parse data.
• Scripting the Client: Write your own logic.
• System Hook Scripts: Create hooks for various system events
• Modules: Module overview and C programming API.
• Methanol Protocol: For those who would like to understand the language of the daemons.

Copying and license

• Copying

Appendices

• Robots Exclusion Standard: Notes about robots.txt

Indices

• Index