Scraping URLs containing query parameters

If the URL you want to scrape contains query parameters, it is required to first URL-encode it. This is important because otherwise our API wouldn't be able to differentiate between query params used in Scraping Fish API call vs. query params you want to pass to the desired web page.

Python
NodeJS
cURL

import requests
from urllib.parse import quote_plus

api_key = "[your API key]"
url = quote_plus("https://example.com?example=param&second=parameter")

response = requests.get(f"https://scraping.narf.ai/api/v1/?api_key={api_key}&url={url}")
print(response.content)

const axios = require("axios");

const api_key = "[your API key]";
const url = encodeURIComponent("https://example.com?example=param&second=parameter");
    
const response = await axios.get(`https://scraping.narf.ai/api/v1/?api_key=${api_key}&url=${url}`);
console.log(response.data);

curl -G --data-urlencode 'url=https://example.com?example=param&second=parameter' \
'https://scraping.narf.ai/api/v1/?api_key=[your API key]'

info

It is a good practice to always encode the URL, regardless of using query params or not. Using requests Python package or axios in NodeJS, parameters are automatically URL-encoded, unless you construct the URL manually with template strings as in the examples above.

Below is a list of links to documentation for URL encoding methods in selected popular programming languages:

You can also use the form below for URL encoding: