Understanding the 'curl' Command (with Examples)
The curl command-line tool is an essential utility for network interaction and data transfer, catering to a variety of communication protocols including HTTP, HTTPS, FTP, and more. It helps users to transfer data across different network types, interacting directly with servers to download files, submit requests, and inspect web resources. With its rich feature set and versatility, curl is indispensable for developers, system administrators, and anyone involved in network operations.
Use case 1: Make an HTTP GET Request and Dump the Contents in stdout
Code:
curl https://example.com
Motivation:
Using curl in its simplest form allows users to fetch the contents of a URL through an HTTP GET request. This feature is invaluable for quickly testing server endpoints or inspecting web pages and their contents straight from the command line, without the need for a web browser.
Explanation:
curl: Invokes thecurlcommand.https://example.com: Specifies the URL of the server from which data will be fetched.
Example Output:
<!doctype html>
<html>
<head>
<title>Example Domain</title>
...
</body>
</html>
This output demonstrates the HTML content of the fetched webpage.
Use case 2: Make an HTTP GET Request, Follow Redirects, and Dump Headers and Contents
Code:
curl --location --dump-header - https://example.com
Motivation:
This command is crucial when dealing with resources that often redirect (use HTTP status codes 3xx) to another location. Following redirects programmatically ensures that users or scripts retrieve the final data. Dumping headers provides insight into communication between client and server, essential for debugging and understanding server behavior.
Explanation:
--location: Tellscurlto follow any 3xx redirects until reaching the final destination.--dump-header -: Directscurlto print the HTTP headers received from the server in the standard output (stdout).https://example.com: The URL whose contents and headers are to be fetched.
Example Output:
HTTP/1.1 200 OK
Content-Type: text/html
...
<html>
<head>
<title>Example Domain</title>
...
</body>
</html>
This shows both the HTTP response headers and the HTML content of the webpage.
Use case 3: Download a File, Saving the Output Under the Filename Indicated by the URL
Code:
curl --remote-name https://example.com/filename.zip
Motivation:
Automating the download of files directly to disk without manually specifying the filename can simplify batch operations and scripts. This is particularly helpful in environments where downloading files is frequent, and filenames are unique.
Explanation:
--remote-name: Instructscurlto save the downloaded file using the basename extracted from the URL (filename.zipin this case).https://example.com/filename.zip: The URL from which the file will be downloaded.
Example Output:
The file filename.zip will be saved to the current directory.
Use case 4: Send Form-Encoded Data (POST Request)
Code:
curl -X POST --data 'name=bob' http://example.com/form
Motivation:
Sending data to web servers in forms is a common requirement, especially for submitting forms programmatically. curl simplifies this task by allowing a user to specify POST data directly on the command line, making it ideal for testing form submissions and API endpoints easily.
Explanation:
-X POST: Specifies the HTTP method to be used, which in this case, is POST.--data 'name=bob': Sends the data in a form-encoded string as part of the POST request body.http://example.com/form: The endpoint to which the form data is being sent.
Example Output:
The exact output will vary based on the server’s configuration and response. If successful, it might confirm the form submission.
Use case 5: Send a Request with an Extra Header, Custom Method, and over a Proxy
Code:
curl -k --proxy http://127.0.0.1:8080 --header 'Authorization: Bearer token' --request GET https://example.com
Motivation:
This use case is tailored for more advanced HTTP/HTTPS interactions, often needed during development and testing phases. The ability to add custom headers and interact through a proxy, such as for intercepting requests via Burp Suite, is crucial when security testing or when operating in environments with strict network policies.
Explanation:
-k: Ignores certificate validation errors, useful for self-signed certificates.--proxy http://127.0.0.1:8080: Directs traffic through a specified proxy server.--header 'Authorization: Bearer token': Adds a custom Authorization header for authenticated requests.--request GET: Specifies the HTTP method, “GET” is used here but could be replaced with others.https://example.com: The designated target URL for the request.
Example Output:
The response will depend on the endpoint, often confirming that a proxy was used and any headers added were respected.
Use case 6: Send Data in JSON Format with Correct Content-Type Header
Code:
curl --data '{"name":"bob"}' --header 'Content-Type: application/json' http://example.com/users/1234
Motivation:
Working with APIs frequently requires sending JSON data. Properly setting the Content-Type as application/json ensures that the server processes the data accurately. This ability makes testing JSON-based APIs straightforward, without the use of additional tools.
Explanation:
--data '{"name":"bob"}': Sends JSON data as the request payload.--header 'Content-Type: application/json': Sets theContent-Typeheader to indicate JSON content.http://example.com/users/1234: The API endpoint being targeted.
Example Output:
An expected successful output might involve a confirmation message or the updated JSON resource as returned by the server.
Use case 7: Pass Client Certificate and Key for a Resource, Skipping Certificate Validation
Code:
curl --cert client.pem --key key.pem --insecure https://example.com
Motivation:
In some secure network environments, accessing certain resources requires presenting a client certificate and key. Ignoring certificate validation (--insecure) is mostly used for development purposes or when accessing servers with self-signed certificates.
Explanation:
--cert client.pem: The client certificate file used for authentication.--key key.pem: The private key associated with the client certificate.--insecure: Skips verification of the server’s certificate.https://example.com: The URL being accessed requiring client certificate authentication.
Example Output:
Responses will vary, typically involving a successful or error message from the server indicating the secure interaction.
Use Case 8: Resolve a Hostname to a Custom IP Address, with Verbose Output
Code:
curl --verbose --resolve example.com:80:127.0.0.1 http://example.com
Motivation:
Override DNS resolution temporarily to test web services locally or redirect traffic to a different server without altering system-wide DNS settings. The verbose output provides detailed request-response interactions, crucial for debugging.
Explanation:
--verbose: Enablescurlto output more detailed progress and debugging information.--resolve example.com:80:127.0.0.1: Points the domainexample.comto IP127.0.0.1on port80for this request.http://example.com: The intended URL to be resolved using the custom IP.
Example Output:
The verbose output would include connection details, used IP, request-response headers, and HTML content of the page.
Conclusion
The curl command is a multi-faceted tool, allowing users to interact with the web in versatile ways, ranging from simple GET requests to complex data transfers involving certificates, proxies, and custom headers. Each example above demonstrates how curl can be harnessed to perform specific tasks efficiently, making it a must-have for professionals dealing with web technologies.
