Curl

Basic Curl Commands

curl is used to make requests from the command line. At it's most simple it sends a GET request and outputs the response to stdOut i.e.

$ curl https://ifconfig.co/
1.2.3.4

Header Information

curl can be used to get to display the headers for a response as well. The quickest way of doing this is to send a HEAD request like so

$ curl -I http://www.example.com
HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: max-age=604800
Content-Type: text/html
Date: Wed, 11 Oct 2017 18:08:42 GMT
Etag: "359670651"
Expires: Wed, 18 Oct 2017 18:08:42 GMT
Last-Modified: Fri, 09 Aug 2013 23:54:35 GMT
Server: ECS (lga/1391)
X-Cache: HIT
Content-Length: 1270

Making POST requests

To make a POST request, you need to do the following

• Tell curl to use POST
• Give it the data to send
• Indicate the type of data that is being sent through

This can be done like so

$ curl -X POST -d '{"test":"test"}' --header "Content-Type: application/json" http://www.example.com 

Using Curl as a Debugging Tool

The main reason that I use curl is to debug API endpoints. To help with this I have a couple of simple scripts that make this easier to work with

#!/usr/bin/env bash
readonly DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )";
cd $DIR;
set -e
set -u
set -o pipefail
standardIFS="$IFS"
IFS=$'\n\t'

if (( $# < 1 ))
then
    echo "
This is used to POST JSON to a URL.

Run it like this:

${DIR}/$0 [url] [file]

where [url] is the URL that the request should be made to and
[file] is the location of a file containing the JSON to POST - defaults to post.json
"
    exit 1;
fi

postData=${2:-"post.json"}

# This will give you a copy and pastable version of the command that is about to be run
echo "curl -k -X POST -d '"`cat ${postData} | json_reformat -m`"' -s --header \"Content-Type: application/json\" "$1" | json_reformat"

curl \
    -k                                          `# Ignore self signed certificates` \
    -s                                          `# Do not output progess - usefull as we are piping the result` \
    -X POST                                     `# This is a post request` \
    -d @${postData}                             `# Using this data` \
    --header "Content-Type: application/json"   `# We are sending through JSON` \
    --cookie "XDEBUG_SESSION=PHPSTORM"          `# Send a PhpStorm Xdeug cookie` \
    $1 |                                        `# Where to make the request` \
    json_reformat |                             `# Make the returned JSON easier to read` \
    highlight --syntax=json -O xterm256         `# Add some colours to the response` \

Note

If you don't have json_reformat or highlight installed run the following

sudo dnf install -y yajl       # Yet Another JSON Library - includes json_reformat 
    sudo dnf install -y highlight  # Highlight - A utility that converts sourcecode to ...  ANSI escape sequences with syntax highlighting

Performance Testing

#!/usr/bin/env bash
readonly DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )";
cd $DIR;
set -e
set -u
set -o pipefail
standardIFS="$IFS"
IFS=$'\n\t'
echo "
===========================================
$(hostname) $0 $@
===========================================
"

if (( $# < 1 ))
then
    echo "
This is used to get details about how long a request takes.

Run it like so

${DIR}/$0 [url]

Where [url] is the URL the request should be made to
"
    exit 1
fi


echo "
    time_namelookup:  %{time_namelookup}\n
       time_connect:  %{time_connect}\n
    time_appconnect:  %{time_appconnect}\n
   time_pretransfer:  %{time_pretransfer}\n
      time_redirect:  %{time_redirect}\n
 time_starttransfer:  %{time_starttransfer}\n
                    ----------\n
         time_total:  %{time_total}\n
" > curlFormat.txt

curl -w "@curlFormat.txt" -s -o /dev/null $@

rm curlFormat.txt

Displays some nice performance info like so

ross@ross:~$ ./curlTimer/curlPerformance.bash https://www.google.com

===========================================
ross ./curlTimer/curlPerformance.bash https://www.google.com
===========================================

    time_namelookup:  0.028420
       time_connect:  0.037841
    time_appconnect:  0.318792
   time_pretransfer:  0.318924
      time_redirect:  0.000000
 time_starttransfer:  0.329161
                    ----------
         time_total:  0.329243