Access table in other than default scheme (database) from sparklyr

apache-spark, dplyr, r, sparklyr

Solution

You can either use a fully qualified name to register temporary view:

spark_session(sc) %>% 
  invoke("table", "my_database.my_table") %>%
  invoke("createOrReplaceTempView", "my_view")

tbl(sc, "my_view")

or use `sql` method to switch databases

spark_session(sc) %>% invoke("sql", "USE my_database")

and access table directly with `dplyr:tbl`:

tbl(sc, "my_table")

Problem

After I managed it to connect to our (new) cluster using `sparklyr` with `yarn-client` method, now I can show just the tables from the default scheme. How can I connect to `scheme.table`? Using `DBI` it's working e.g. with the following line: `dbGetQuery(sc, "SELECT * FROM scheme.table LIMIT 10")` In HUE, I can show all tables from all schemes. ~g

Original source