How to use the command 'xidel' (with examples)

Xidel is a powerful command-line tool that allows users to download and extract data from HTML, XML, and JSON. Whether you’re working with web pages, APIs, or RSS feeds, Xidel simplifies the extraction of essential information using various querying methods like XPath, CSS selectors, and pattern matching. The tool is particularly useful for web scraping and data mining applications, enabling users to automate the retrieval of data for analysis or integration into other systems.

Use case 1: Print all URLs found by a Google search

Code:

xidel https://www.google.com/search?q=test --extract "//a/extract(@href, 'url[?]q=([^&]+)&', 1)[. != '']"

Motivation:

This example is a perfect demonstration of scraping search results from Google. Since URLs form the building blocks of the web and often lead to invaluable content and additional resources, capturing these links is crucial for numerous applications like data collection, web crawling, and competitive analysis.

Explanation:

• xidel: The command to invoke the Xidel tool.
• https://www.google.com/search?q=test: The URL to Google search results for the query “test”.
• --extract "//a/extract(@href, 'url[?]q=([^&]+)&', 1)[. != '']": This extraction argument uses XPath to locate all <a> tags and apply pattern matching to extract the href attribute containing URLs. The regular expression url[?]q=([^&]+)& captures the actual URL by matching content between the q= and & delimiters.

Example Output:

https://www.example.com
https://www.testsite.org
...

Use case 2: Print the title of all pages found by a Google search and download them

Code:

xidel https://www.google.com/search?q=test --follow "//a/extract(@href, 'url[?]q=([^&]+)&', 1)[. != '']" --extract //title --download '{$host}/'

Motivation:

This is an advanced scraping example where not only the links are extracted, but the content of each link (titles) is also deduced. Additionally, it downloads the pages for offline analysis. Such an operation is typical when assessing the relevance of search results or when archiving data.

Explanation:

• xidel: Invokes the Xidel command-line tool.
• https://www.google.com/search?q=test: The Google search URL for the query “test”.
• --follow "//a/extract(@href, 'url[?]q=([^&]+)&', 1)[. != '']": This instructs Xidel to automatically follow and extract URLs, employing the same XPath pattern recognition and link extraction as in the previous use case.
• --extract //title: This command extracts the <title> elements from the followed pages.
• --download '{$host}/': Downloads the content of each followed link into the local directory using the host name to create folder structures.

Example Output:

Example Site Title
Another Page Title
...

Use case 3: Follow all links on a page and print the titles, with XPath

Code:

xidel https://example.org --follow //a --extract //title

Motivation:

Understanding the titles of pages linked from a single web page helps grasp what content is being linked. This use case highlights how to follow links automatically and scrape their titles, a vital action in site structure analysis or content discovery.

Explanation:

• xidel: Runs the Xidel tool from the command line.
• https://example.org: URL of the webpage to scrape.
• --follow //a: Follows all links (<a> tags) from the base page to their target pages.
• --extract //title: Extracts the title from each visited page using XPath.

Example Output:

First Linked Page Title
Second Linked Page Title
...

Use case 4: Follow all links on a page and print the titles, with CSS selectors

Code:

xidel https://example.org --follow "css('a')" --css title

Motivation:

This example showcases the use of CSS selectors instead of XPath for extracting links and titles. CSS selectors may be more intuitive for those familiar with web development, providing a different method to reach the same goal of link-following and title extraction.

Explanation:

• xidel: Executes the command tool.
• https://example.org: The URL of the starting page.
• --follow "css('a')": Utilizes CSS syntax to follow links. The 'a' selector targets all anchor tags, which traditionally contain hyperlinks.
• --css title: Uses CSS selectors to extract the title element of each followed page.

Example Output:

Linked Page Title One
Linked Page Title Two
...

Use case 5: Follow all links on a page and print the titles, with pattern matching

Code:

xidel https://example.org --follow "<a>{.}</a>*" --extract "<title>{.}</title>"

Motivation:

Pattern matching uses straightforward literal structures which may be helpful when the extraction schema is complex, or data is non-standard. Here, we illustrate the flexibility of the tool in using pattern constructs.

Explanation:

• xidel: Initiates the Xidel tool.
• https://example.org: The target website from where links will be followed.
• --follow "<a>{.}</a>*": Pattern format to target anchor tags.
• --extract "<title>{.}</title>": Extracts the title tag’s content as defined by this simple pattern template.

Example Output:

Page Title Example
Example Title for Page
...

Use case 6: Read the pattern from example.xml

Code:

xidel path/to/example.xml --extract "<x><foo>ood</foo><bar>{.}</bar></x>"

Motivation:

XML files often form the backbone of data interchange between systems. This example utilizes pattern matching within XML, making it relevant for tasks like data validation or integration testing when looking for specific content.

Explanation:

• xidel: Runs the tool.
• path/to/example.xml: Specifies the local path to an XML file.
• --extract "<x><foo>ood</foo><bar>{.}</bar></x>": Extracts data based on a template structure, requiring an exact pattern match to succeed ("ood" must be present for the extraction to take place).

Example Output:

Bar Element Content
Additional Content
...

Use case 7: Print all newest Stack Overflow questions with title and URL

Code:

xidel http://stackoverflow.com/feeds --extract "<entry><title>{title:=.}</title><link>{uri:=@href}</link></entry>+"

Motivation:

RSS feeds publish updates and news worldwide, serving as a data source for web-based content. Extracting titles and links from feeds supports information aggregation, allowing one to stay updated on topics or participate in discussions—vital for timely reactions in fast-paced environments like technical discussions on Stack Overflow.

Explanation:

• xidel: Begins using the Xidel tool.
• http://stackoverflow.com/feeds: Request to access Stack Overflow’s RSS feed.
• --extract "<entry><title>{title:=.}</title><link>{uri:=@href}</link></entry>+": Pattern-based extraction aligns with the structure of RSS entries, capturing titles and link URIs directly attributed to each post.

Example Output:

Question Title One
https://stackoverflow.com/questions/question-1
Question Title Two
https://stackoverflow.com/questions/question-2
...

Use case 8: Check for unread Reddit mail

Code:

xidel https://reddit.com --follow "form(css('form.login-form')[1], {'user': '$your_username', 'passwd': '$your_password'})" --extract "css('#mail')/@title"

Motivation:

Automating the check for unread mail or notifications streamlines staying informed without manual interaction, making it vital for keeping up with community engagements or moderating activities. This illustrates automatic form submission and extraction of specific data points.

Explanation:

• xidel: Activates the command tool.
• https://reddit.com: Accesses the Reddit website.
• --follow "form(css('form.login-form')[1], {'user': '$your_username', 'passwd': '$your_password'})": Demonstrates automatic form submission using CSS selection to choose the login form, with placeholders for user credentials.
• --extract "css('#mail')/@title": Retrieves the title attribute of the mail icon/layout area, indicating unread messages or notifications.

Example Output:

Secure a proper login first approach, an output typically would show 
something like:
"1 Unread message" 
or "No new messages" 

Conclusion:

Whether for extracting complex web data or automating repetitive web-based tasks, Xidel stands out as a versatile, command-line tool bridging the gap between textual data sources and their practical applications. By leveraging XPath, CSS, and pattern matching, users can tailor its capabilities to varied needs, rendering Xidel a valuable resource in web scraping, data integration, and automated analysis.