While I try hard to write things described as "* of the day" regularly there are times when life throws you a curve ball. Yesterday's curve ball was the intersection of sick child, snow and a failed sewage lift pump led to, well, personal hell at home. After a very busy work day I spent the evening that I had allocated to writing, well, playing clean up man before the real plumber showed up. Anyway … on to curl!

The standard Linux tool curl seems to always be present on Linux / OSX systems so I can skip over the installation issues and I suspect most of us know the basics i.e.:

curl http://foo.com/bar.html

will fetch the file bar.html from the foo.com website and place it in the current directory. Given that our goal here was to discuss how curl was used in John Graham Cummings hncomments bash script and that's all we need, I could stop here but let's go for the gusto and give some examples of advanced usage of curl.

Output Redirection with curl

Normally the output of curl is written to the same filename that was used on the server. You can use the > character to redirect output to a new file as showing here:

curl yoururl.com > yoururl.html

More

Send a Custom Header with curl

If you want to send a custom header with curl then you can inject it with –header:

curl –header 'Content-Type: application/json' http://yoururl.com

More

Get Headers with curl

You can get a feel for the Internet http headers with –head:

curl –head http://fuzzyblog.io

curl --head http://fuzzyblog.io
HTTP/1.1 200 OK
Server: GitHub.com
Date: Fri, 10 Feb 2017 02:00:34 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 1125
Last-Modified: Mon, 16 Jan 2017 18:12:56 GMT
Access-Control-Allow-Origin: *
Expires: Fri, 10 Feb 2017 02:10:34 GMT
Cache-Control: max-age=600
Accept-Ranges: bytes
X-GitHub-Request-Id: EA7C:71BB:11DF3A8:1883A50:589D1EC2

This is particularly useful if you build tools like crawlers / indexers because you can use the intersection of Last-Modified and Control-Length to make a hash key that you use for determining if you should index something. HTTP head operations are very cheap because only the header is transferred. More on Head

Testing Advanced Options

If you're looking for help with curl then try the Online Curl tool which gives you a user interface that has generates a curl command as its output.

References

Here are some more references: