R - httr Web interaction

 

Interact with web

raw <- GET(url_string) – retrieve from server, status starting with 2/3 is fine, 4 your problem, 5, their problem.

POST(url_string, data) – send data to server

content(raw, as = “text”/”parsed” (default) ) from httr, read from GET result

http_error(raw) returns  TRUE if there’s an error, make error handling easier

GET(url, user_agent(“my@email.address this is a test”, query = params) user agent provides extra info for the webmaster, in case anything goes wrong. query helps add more params to the url more easily (rather than string concat or paste), param example = list (x = “asd”, y = “qwe”) -> url?x=asd&y=qwe .

paste() sep = “/”, glue strings together

http_type()

JSON

fromJSON(content(raw))

rlist package: list.select(json,var1,var2) collect var out of a json, list.stack() stack result from list.select into a dataframe

bind_rows, from dyplyr, turns list into df.

XML

xml2 package

read_xml()

xml_structure()

xml_find_all(xml result, XPATH “api/abc/xyz/rev”) extract all nodes matching the XPATH. Find all rev nodes regardless of path, use “//rev”. Each node is sith like <node> </node>.

xml_find_first, similar to find_all

xml_text, xml_double, xml_interger extract data out of a nodeset (the result of find_all)

xml_attrs or xml_attr , find all or a specific attribute of nodes

Web scraping

rvest package

read_html(url)

html_node(result above, xpath = “//node”)

html_text(node), html_attr(node, name = “”), html_name(node) name of the node …

html_table(), accepts a table node object and output a dataframe

html_nodes(test_xml, css = “..”), select by css, nodes return more than one result. “tag” selects by tag name. “.classname” selects by class name. “#id” selects by ID.

xml_text() pulls out the text (html) from xml response

 

Comments

Post a Comment

Popular posts from this blog

Jonas - Javascript

  variables – var abc = ”, – data types   Section 3 – How JS works Scoping When refer to a var that is not available in the current scope, the program will look to the outer scope (or even outer outer scope)   Section 4 – DOM Manipulation   classList, .add .remove classes .toggle, turn on/off the “active” class   Section 5 – Objects and Functions Everything is an object, check with  console.log () for __proto__ when we try to access properties or methods of an object it will look in its own scope first, then its proto parents, and so on. call, apply, bind, to change the  this  variable. And to pre-set some arguments of a function  
Read more

R - Supervised Learning

Image
  knn k nearest neighbors, k library(class) knn(training data, test data, labels), extra args: k = 2. prob = TRUE enables probability of each guess. Primitive bayes nrows(df), count rows subset(df, conditions/ col > 0 etc.), nrows this subset then do divisions Naive bayes model <- naive_bayes(y ~ x1 + x2 + … , data = df) y as explained by x1, x2 etc. predict(model , test_df) + args type = “prob” show posterior probability (instead of priori overall prob) Laplace correction:  add 1 to prevent the 0% probability affects all estimation Logistic Regression model <- glm(y ~ x1 + x2 …, data = df, family = “binomial”) predict(model, test_df, type = “response”) ROC curve to measure how well the model performs, AUC area under curve value from 0 – 1, 1 is best. library(pROC) ROC <- roc(actual var, predicted var) plot(ROC), auc(ROC) Decision Tree library(rpart) model <- rpart(y ~ x1 + x2 + …, data = df, method = “class”, control = rpart.control(cp = 0)) predict(m...
Read more

Consulting Interview

  Numerical skills – rather simple tips on quick estimation, rounding Top down approach is preferred Hypothesis – Always start with a hypothesis, then reason with the info to either prove it or disprove it. Don’t present info and a bunch of rationales before the conclusion, it’s just less clear to the listeners, especially C-level executives. Synthesis – Let’s do A. Because of B, C and D.
Read more