A Complete Guide to Sending curl Headers & Debugging
Whether you’re automating API calls, scraping data, or troubleshooting network issues, understanding how to work with HTTP headers in curl is essential. This guide covers everything from sending basic headers to advanced debugging techniques, with practical examples—including using residential proxies—to ensure you can implement each step confidently.
Why HTTP Headers Matter
HTTP headers are key-value pairs sent in requests and responses to convey metadata such as content type, authentication tokens, and caching directives. When you issue a curl command:
Request headers tell the server what you expect or who you are (e.g., Accept: application/json, Authorization: Bearer <token>).
Response headers inform your client about the payload (e.g., Content-Type: text/html; charset=UTF-8) and directives (e.g., Cache-Control).
Proper header management lets you:
- Authenticate securely with APIs.
- Specify payload formats.
- Control caching, cookies, redirects, etc.
Sending Single and Multiple Headers
1. Adding a Single Header
Use the -H (or --header) flag:
bash
curl -H "Content-Type: application/json" https://api.example.com/data
2. Adding Multiple Headers
Simply repeat -H for each header:
bash
curl \
-H "Authorization: Bearer YOUR_TOKEN" \
-H "Accept: application/json" \
https://api.example.com/data
This method works for any number of headers—just ensure each has the correct Name: value format.
Overriding and Removing Default Headers
By default, curl sends headers like Host, User-Agent, and Accept. To override one, provide a header of the same name:
bash
curl -H "User-Agent: MyCustomAgent/1.0" https://example.com
To remove a default header entirely, provide it with an empty value:
bash
curl -H "User-Agent:" https://example.com
This can be vital when testing servers that reject certain agents or when simulating minimal clients.
Viewing Response Headers
Inspecting the response headers is key to troubleshooting and understanding server behavior.
HEAD request only
bash
curl -I https://example.com
Include headers with the body
bash
curl -i https://example.com
Verbose debug output
bash
curl -v https://example.com
Here you will see the request and response headers, TLS handshake details, and connection information.
Advanced Header Techniques
1. Saving Headers to a File
Dump response headers into a file for further analysis:
bash
curl -D response-headers.txt -o response-body.txt https://example.com
- -D <file> saves headers.
- -o <file> saves the body separately.
2. Empty and Conditional Headers
Empty header (some servers interpret differently from removal):
bash
curl -H "X-Custom-Header;" https://example.com
Conditional requests with If-Modified-Since:
bash
curl -H "If-Modified-Since: Wed, 21 Oct 2020 07:28:00 GMT" https://example.com
API Authentication Example
1. Obtain a token (OAuth, JWT, or API key).
2. Include it in every request:
bash
curl \
-H "Authorization: Bearer ABC123XYZ" \
-H "Accept: application/json" \
https://api.yourservice.com/userinfo
3. Automate by exporting to an environment variable:
bash
export TOKEN="ABC123XYZ"
curl -H "Authorization: Bearer $TOKEN" https://api.yourservice.com/data
Using OkeyProxy for Geo-Blocked or Rate-Limited APIs
When to use a residential proxy:
- Bypass IP-based geo-restrictions.
- Spread requests across many IPs to avoid rate limits.
1. Basic OkeyProxy Integration
Sign up, choose a residential proxy, and get your OkeyProxy endpoint. Assume it is http://user:[email protected]:8000. Combine proxy and headers:
bash
curl \
-x http://user:[email protected]:8000 \
-H "Accept: application/json" \
https://api.geoservice.com/content
2. Rotating Residential Proxy Setup
If you have a pool of proxies:
bash
curl \
-x http://user:[email protected]:8000 \
-H "Accept: application/json" \
https://api.highvolume.com/data
Automate rotation in a shell loop:
bash
for proxy in proxy1 proxy2 proxy3; do
curl -x http://user:pass@$proxy:8000 \
-H "Accept: application/json" \
https://api.highvolume.com/data
done
This approach distributes requests and reduces the chance of blocks. Click here to check our high quality and cost-effective rotating residential proxy service.
Cookie and Session Management
Some workflows require cookies:
Send cookies:
bash
curl --cookie "sessionid=abc123" https://example.com/dashboard
Save cookies for reuse:
bash
curl --cookie-jar cookies.txt -c cookies.txt -L https://example.com/login
Load saved cookies:
bash
curl --cookie cookies.txt https://example.com/dashboard
Troubleshooting Tips
1. Syntax errors: Ensure no extra spaces around the colon in headers.
2. Case sensitivity: Header names are case-insensitive, but values sometimes are not.
3. Verbose mode: Use -v to see where things break.
4. Test endpoint: Send headers to httpbin.org/headers to confirm what the server receives:
bash
curl -H "X-Test: 123" https://httpbin.org/headers
Best Practices and Security
Never expose sensitive tokens in shared scripts. Use environment variables.
Use HTTPS (https://) to encrypt headers (especially Authorization).
Validate responses: Check status codes (-w '%{http_code}') to ensure authentication success.
Conclusion
From setting simple headers to orchestrating rotating residential proxies with OkeyProxy, mastering curl headers unlocks powerful capabilities for developers and sysadmins. Whether you’re a beginner looking to understand the basics or a seasoned professional optimizing large-scale API calls, these steps and examples give you a solid, reproducible workflow.
