cURL is a handy tool which allows you to transfer or download files to and from a server. The cURL utility supports a whole host of protocols but today we will only be covering HTTP.
When you invoke ‘cURL’ from the command line with no options and just a URL it will retrieve the web page and return its contents on screen. ‘cURL’ uses the HTTP GET method by default.
sam@asus:~/unix% curl http://127.0.0.1/index.html <html> <head><title></title></head> <body> <h1>Hello, World!</h1> <p>This is a test page<p> </body> </html> sam@asus:~/unix%
Retrieving a file with cURL is really easy. The ‘-o’ or ‘-O’ options allow you to transfer a file from a remote server to your local computer.
sam@asus:~/unix% curl -O http://ipv4.download.thinkbroadband.com/5MB.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5120k 100 5120k 0 0 1217k 0 0:00:04 0:00:04 --:--:-- 1217k
sam@asus:~/unix%
or
sam@asus:~/unix% curl -o 5-MB-FILE.zip http://ipv4.download.thinkbroadband.com/5MB.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5120k 100 5120k 0 0 1085k 0 0:00:04 0:00:04 --:--:-- 1085k
sam@asus:~/unix%
cURL gives you to open to specify a particular HTTP method such as GET or POST with the ‘--request’ option.
sam@asus:~/unix% curl --request GET http://127.0.0.1/index.html <html> <head><title></title></head> <body> <h1>Hello, World!</h1> <p>This is a test page<p> </body> </html> sam@asus:~/unix%
To get HTTP Header information using a ‘HEAD’ from the URL requested the ‘-I’ option is used.
sam@asus:~/unix% curl -I http://127.0.0.1/index.html HTTP/1.1 200 OK Date: Wed, 03 Apr 2019 00:58:22 GMT Server: Apache/2.4.18 (Ubuntu) Last-Modified: Wed, 03 Apr 2019 00:33:23 GMT ETag: "6c-5859566b61b0c" Accept-Ranges: bytes Content-Length: 108 Vary: Accept-Encoding Content-Type: text/html sam@asus:~/unix%
If you wanted to construct your own HTTP Header, cURL gives you the ability to do so with the ‘-H’ option. The -H option requires a string with the format of “header-name: value”. For instance if you would like to add an additional HTTP header of “Test: Value” you would request the URL as so:
sam@asus:~/unix% curl -H "Test: Value" http://127.0.0.1/index.html <html> <head><title></title></head> <body> <h1>Hello, World!</h1> <p>This is a test page<p> </body></html> sam@asus:~/unix%
To send a HTTP POST request with cURL a few additional arguments are needed. The ‘-H’ or header argument and the data to be sent with the ‘-d’ argument. Note, we also included the ‘-v’ or verbose argument to get the request and response headers to check for errors.
sam@asus:/var/www/html% curl http://127.0.0.1/post.php -v -H "Content-Type: application/x-www-form-urlencoded" -d 'fname=john&lname=doe&mesg=unixisfun' * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 80 (#0) > POST /post.php HTTP/1.1 > Host: 127.0.0.1 > User-Agent: curl/7.47.0 > Accept: */* > Content-Type: application/x-www-form-urlencoded > Content-Length: 35 > * upload completely sent off: 35 out of 35 bytes < HTTP/1.1 200 OK < Content-Type: application/x-www-form-urlencoded < Date: Wed, 03 Apr 2019 01:53:53 GMT < Server: Apache/2.4.18 (Ubuntu) < Content-Length: 35 < Connection: close < Content-Type: text/html; charset=UTF-8 < Hello jon doe!, unixisfun * Closing connection 0 sam@asus:/var/www/html%
in the ‘-H’ option we specified the ‘Content-type’ header as ‘application/x-www-form-urlencoded’. cURL also supports multipart/form-data requests. The ‘-d’ option specifies the POST body content which is ‘fname=john&lname=doe&mesg=unixisfun’.
cURL also comes with the ability to use a proxy when retrieving data from a server.
sam@asus:~/unix% curl -v --proxy 94.232.126.225:35445 http://www.acme.com/ * Trying 94.232.126.225... * Connected to 94.232.126.225 (94.232.126.225) port 35445 (#0) > GET http://www.acme.com/ HTTP/1.1 > Host: www.acme.com > User-Agent: curl/7.47.0 > Accept: */* > Proxy-Connection: Keep-Alive > < HTTP/1.1 200 OK < Date: Wed, 03 Apr 2019 21:21:19 GMT < Expires: -1 < Cache-Control: private, max-age=0 < Content-Type: text/html; charset=ISO-8859-1 < P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info." < Server: gws < X-XSS-Protection: 0 < X-Frame-Options: SAMEORIGIN < Set-Cookie: 1P_JAR=2019-04-03-21; expires=Fri, 03-May-2019 21:21:19 GMT; path=/; domain=.acme.com < Set-Cookie: NID=180=IPr5TDaCP3AF5ZEoBOKm01A1unSo; expires=Thu, 03-Oct-2019 21:21:19 GMT; path=/; domain=.acme; HttpOnly < Accept-Ranges: none < Vary: Accept-Encoding < Transfer-Encoding: chunked < < <html> <head><title></title></head> <body> <h1>Hello, World!</h1> <p>This is a test page<p> </body></html> sam@asus:~/unix%
No comments:
Post a Comment