Web Scraping in R: A Comprehensive Guide with OkeyProxy

Web scraping transforms unstructured web pages into clean datasets for analysis, reporting, or dashboards—and R’s rich package ecosystem makes it easy. In this guide, you’ll start from zero to advanced techniques: environment setup, HTML/CSS fundamentals, basic rvest usage, HTTP control, dynamic content handling, proxy integration, parallel crawling, error resilience, and ethics.

Understanding HTML & CSS Basics

Before you can scrape data, you need to know how web pages are structured and how to pinpoint the elements you care about.

Why HTML & CSS Matter

Web pages are built from HTML—a hierarchy of elements (tags)—and styled or organized using CSS. To extract specific text, images, or tables, you tell your scraping tools exactly which elements to target via selectors.

Term Check

Element: An HTML tag, e.g. <p>, <h1>, <table>.

Attribute: Additional information inside a tag, e.g. class="price", id="main".

Selector: A pattern (CSS or XPath) that matches one or more elements on the page.

Example HTML

html

 

<div class="product">

  <h2 id="title">Widget A</h2>

  <p class="price">$10.00</p>

</div>

• The <div> groups a product.
• The <h2> has an id of title.
• The <p> has a class of price.

Common CSS Selectors

Class selector

css

 

.price       /* matches <p class="price"> */

ID selector

css

 

#title       /* matches <h2 id="title"> */

Nested selector

css

 

div.product > p.price  

/* matches any <p class="price"> directly inside <div class="product"> */

Simple XPath Example

xpath

 

//div[@class="product"]/p[@class="price"]

This finds any <p> with class="price" inside a <div> of class product.

Pro Tip

Install the SelectorGadget browser extension (or try CSS Diner) to click on elements and generate CSS selectors visually—no manual trial and error needed.

Step 1: Environment & Prerequisites

1. Install R & RStudio (or VS Code)

2. Create a new project

3. Install packages

r

 

install.packages(c(

  "rvest", "xml2", "httr2",

  "RSelenium", "chromote",

  "polite", "parallel", "Rcrawler",

  "tidyverse", "purrr", "stringr", "lubridate"

))

4. Load libraries once per script

r

 

library(rvest); library(xml2); library(httr2)

library(RSelenium); library(chromote)

library(polite); library(parallel); library(Rcrawler)

library(tidyverse); library(purrr); library(stringr); library(lubridate)

Core Tools & Packages Overview

You need the right packages loaded before writing any scraping code.

Task	Package(s)	Purpose
Static HTML scraping	rvest, xml2	Fetch and parse page content
JavaScript-rendered pages	RSelenium, chromote	Drive headless browsers for dynamic content
HTTP control	httr2	Custom headers, cookies, rate limiting
Parallel crawling	parallel, Rcrawler	Multi-core scraping, depth/pagination control
Data hygiene & export	tidyverse, jsonlite	Clean, transform, save to CSV or JSON
Polite scraping	polite, req_throttle	Respect robots.txt and throttle requests

Step 2: Basic Static Scraping with rvest

rvest makes fetching and parsing HTML trivial.

1. Fetch & Parse

r

 

url  <- "https://example.com/products"

page <- read_html(url)

2. Extract Elements

r

 

titles <- page %>% html_elements(".product-name") %>% html_text2()

prices <- page %>% html_elements(".price")        %>% html_text2()

df     <- tibble(title = titles, price = prices)

head(df)

Beginner: html_elements() returns a list; html_text2() cleans whitespace.

Professional: Chain pipes for readability; inspect intermediate objects with print().

Step 3: Pagination, Tables & Early Error Handling

Most sites spread data across pages and tables, and sometimes requests fail.

1. Scrape a Table

r

 

table <- page %>% html_node("table.inventory") %>% html_table(header = TRUE)

2. Loop Through Pages

r

 

pages <- sprintf("https://example.com/page/%d", 1:5)

safe_read <- possibly(read_html, otherwise = NA)

 

all_tables <- map_dfr(pages, ~ {

  pg <- safe_read(.x)

  pg %>% html_node("table.inventory") %>% possibly(html_table, otherwise = tibble())()

})

Beginner: map_dfr() binds rows and handles lists of tibbles.

Professional: Wrapping in possibly() ensures your loop continues even if one page errors.

Step 4: HTTP Control with httr2

Customize headers and throttle to avoid blocks.

r

 

response <- request("https://example.com/data") %>%

  req_headers(`User-Agent` = "R Scraper v1.0") %>%

  req_throttle(5 / 60) %>%    # 5 requests per minute

  req_perform()

page <- resp_body_html(response)

Beginner: Change User-Agent to mimic a browser and bypass simple bot filters.

Professional: Randomize delays:

r

 

Sys.sleep(runif(1, 1, 3))

Step 5: Dynamic Pages via RSelenium & chromote

Some sites render content client-side with JavaScript.

1. RSelenium (Headless Chrome)

r

 

rD    <- rsDriver(browser = "chrome", chromever = "latest",

                  extraCapabilities = list(chromeOptions=list(args="--headless")))

remDr <- rD$client

remDr$navigate("https://example.com/js-content")

Sys.sleep(4)  # wait for JS to load

html  <- remDr$getPageSource()[[1]] %>% read_html()

titles <- html %>% html_elements(".js-title") %>% html_text2()

remDr$close(); rD$server$stop()

2. chromote (Lightweight JS)

r

 

session <- ChromoteSession$new()

session$Page$navigate("https://example.com/js-content")

session$Page$loadEventFired()

html <- session$Runtime$evaluate("document.documentElement.outerHTML")$result$value

doc  <- read_html(html)

items <- doc %>% html_elements(".item") %>% html_text2()

Beginner: RSelenium automates what you’d do manually in a browser.

Professional: Pass proxy settings via Chrome arguments if needed:

r

 

ChromoteSession$new(

  browser_args = c("--proxy-server=http://user:[email protected]:8000")

).

Step 6: Integrating OkeyProxy & Rate-Limiting

Proxies keep your IP fresh, avoid geo-blocks, and distribute load. Sign up here and select a rotating residential proxy plan or a rotating datacenter proxy plan as needed.

1. Single-Request Proxy

r

 

proxy <- "http://user:[email protected]:8000"

resp  <- request("https://example.com") %>%

         req_proxy(proxy) %>%

         req_throttle(2 / 1) %>%  # 2 requests/sec

         req_perform()

page  <- resp_body_html(resp)

2. Automated IP Rotation

r

 

proxies <- c(

  "http://u:[email protected]:8000",

  "http://u:[email protected]:8000",

  "http://u:[email protected]:8000"

)

scrape_page <- function(url, proxy) {

  request(url) %>% req_proxy(proxy) %>% req_perform() %>%

    resp_body_html() %>% read_html() %>%

    html_elements(".data") %>% html_text2() %>%

    tibble(data = .)

}

results <- map2_dfr(pages, rep(proxies, length.out = length(pages)), scrape_page)

Professional: Rotate proxies round-robin to spread traffic and stay under per-IP limits.

Step 7: Scaling with Parallel & Rcrawler

Parallelization and automated crawling speed up large jobs.

1. Parallel Scraping

r

 

cl <- makeCluster(detectCores() - 1)

clusterEvalQ(cl, library(rvest))

out <- parLapply(cl, pages, function(u) {

  read_html(u) %>% html_elements(".product") %>% html_text2()

})

stopCluster(cl)

2. Automated Crawling with Rcrawler

r

 

Rcrawler(

  Website   = "https://example.com",

  no_cores  = 4,

  MaxDepth  = 2,

  FUN       = function(url, ...) {

    doc <- read_html(url)

    tibble(title = doc %>% html_element("h1") %>% html_text2())

  }

)

Professional: Monitor memory, checkpoint progress, and gracefully stop clusters on repeated failures.

Step 8: Errors, Cleaning & Selector Tools

Clean data and resilient scripts save time down the road.

1. Error Handling

r

 

safe_parse <- possibly(~ html_elements(read_html(.x), ".item"), otherwise = character())

results <- map(pages, safe_parse)

2. Data Cleaning

r

 

df <- df %>%

  mutate(

    price = str_remove_all(price, "[^0-9\\.]"),

    date  = parse_date(date, "%B %d, %Y"),

    text  = str_squish(text)

  )

3. Selector Discovery

Use SelectorGadget, DevTools, or CSS Diner to refine your CSS or XPath queries.

4. CAPTCHA & Bot Defenses

Introduce randomized delays, header rotations, or leverage headless browsers.

Tip: In your Mini-Project below, see how cleaning functions produce a polished CSV.

Ethics & Best Practices

Responsible scraping protects you and the sites you crawl.

1. Respect robots.txt:

r

 

session <- bow("https://example.com", delay = 2)

page <- scrape(session)

2. Check Terms: Look at the site’s Terms of Service.

3. Throttle Requests: Use req_throttle() or Sys.sleep().

4. Identify Yourself: Clear User-Agent unless anonymity is needed.

5. Privacy Compliance: Don’t scrape personal data without permission; follow GDPR.

6. Monitor & Log: Record successes, failures, and response times for debugging.

Putting It All Together: A Mini-Project Example

Scrape a paginated e‑commerce site, rotate OkeyProxy endpoints, clean data, and export a CSV:

r

 

library(rvest); library(httr2); library(tidyverse)

 

proxies <- c(

  "http://u:[email protected]:8000",

  "http://u:[email protected]:8000"

)

urls <- paste0("https://shop.example.com/page", 1:5)

 

scrape_page <- function(url, proxy) {

  resp <- request(url) %>% req_proxy(proxy) %>% req_throttle(1/1) %>% req_perform()

  doc  <- read_html(resp)

  tibble(

    title = doc %>% html_elements(".title") %>% html_text2(),

    price = doc %>% html_elements(".price") %>% html_text2()

  )

}

 

raw_data <- map2_dfr(urls, rep(proxies, length.out = length(urls)), scrape_page)

 

cleaned <- raw_data %>%

  mutate(

    price = str_remove_all(price, "[^0-9\\.]") %>% as.numeric()

  ) %>%

  drop_na(price)

 

write_csv(cleaned, "products.csv")

Conclusion

You now have a full-stack R scraping workflow—from rvest basics through JS rendering, OkeyProxy rotation, parallel crawls, and robust data cleaning. Happy scraping—and remember: scrape responsibly.

Unlock the full potential of your R scrapers—sign up for a free trial of OkeyProxy today. High quality and cheap rotating proxies to stay under rate limits and avoid blocks. See how easy it is to power fast, reliable, and ethical data collection at scale!