bigquery Repeated record added outside of an array

google-bigquery, scala

Solution

Found the problem.

It's something with scala-to-java collections conversions mess. I had to explicitly add the conversion using JavaConversions.seqAsJavaList and it magically started working

val row = new TableRow()
  .set("revisionId", s"revisionId$v")
  .set("executions", JavaConversions.seqAsJavaList(Seq(
    new TableRow().set("ts", 1L).set("price", 100.01).set("qty", 1000),
    new TableRow().set("ts", 2L).set("price", 100.02).set("qty", 2000),
    new TableRow().set("ts", 3L).set("price", 100.03).set("qty", 3000)
  )))

Problem

Error: ``` Exception in thread "main" java.lang.RuntimeException {"errors":[{"debugInfo":"generic::failed_precondition: Repeated record added outside of an array.","reason":"invalid"}],"index":0} ``` Language: Scala Gradle bigquery dependency: ``` compile "com.google.apis:google-api-services-bigquery:v2-rev326-1.22.0" ``` The code to generate table schema: ``` import scala.collection.JavaConversions._ val orderTableSchema = new TableSchema() orderTableSchema.setFields(Seq( new TableFieldSchema().setName("revisionId").setType("STRING").setMode("REQUIRED"), new TableFieldSchema().setName("executions").setType("RECORD").setMode("REPEATED").setFields(Seq( new TableFieldSchema().setName("ts").setType("INTEGER"), new TableFieldSchema().setName("price").setType("FLOAT"), new TableFieldSchema().setName("qty").setType("INTEGER") )) )) ``` The table is created successfully with correct schema for executions column as shown in BigQuery web ui: ``` revisionId STRING REQUIRED executions RECORD REPEATED executions RECORD REPEATED executions.ts INTEGER NULLABLE executions.price FLOAT NULLABLE executions.qty INTEGER NULLABLE ``` The code to insert data which fails: ``` import scala.collection.JavaConversions._ val row = new TableRow() .set("revisionId", "revision1") .set("executions", Seq( new TableRow().set("ts", 1L).set("price", 100.01).set("qty", 1000) )) val content = new TableDataInsertAllRequest().setRows(Seq( new TableDataInsertAllRequest.Rows().setJson(row) )) val insertAll = bigQuery.tabledata().insertAll(projectId, datasetId, "order", content) val insertResponse = insertAll.execute() if (insertResponse.getInsertErrors != null) { insertResponse.getInsertErrors.foreach(println) // this prints: // {"errors":[{"debugInfo":"generic::failed_precondition: Repeated record added outside of an array.","reason":"invalid"}],"index":0} // throw to just terminate early for demo throw new RuntimeException() } ```

Original source