Not able to do proper path monitoring or querying with Firebase and ElasticSearch (with Flashlight)

elasticsearch, firebase, heroku, json, node.js

Solution

Flashlight actually looks for requests at search/requests/$key, which should contain a doc with fields type, index, query. It puts a response at search/response/$key with the results. If you want to test by manually adding the keys, try putting the request under another node e.g. search/requests/testkey123.

Problem

I'm trying to integrate ElasticSearch wit Firebase. I'm using the Flashlight integration from Firebase to set ut all up. I have deployed the code to Heroku as described in the Github repo linked above. It works in the sense that when I insert a query objet to `/search/request/` I get a `/search/response` result. But the results are a bit messy, not correct. But I can't figure out what is wrong. This is what is defined in `config.js` where I define the paths to monitor: ``` /** Paths to Monitor * * Each path can have these keys: * {string} path: [required] the Firebase path to be monitored, for example, `users/profiles` * would monitor https://<instance>.firebaseio.com/users/profiles * {string} index: [required] the name of the ES index to write data into * {string} type: [required] name of the ES object type this document will be stored as * {Array} fields: list of fields to be monitored and indexed (defaults to all fields, ignored if "parser" is specified) * {Array} omit: list of fields that should not be indexed in ES (ignored if "parser" is specified) * {Function} filter: if provided, only records that return true are indexed * {Function} parser: if provided, the results of this function are passed to ES, rather than the raw data (fields is ignored if this is used) * * To store your paths dynamically, rather than specifying them all here, you can store them in Firebase. * Format each path object with the same keys described above, and store the array of paths at whatever * location you specified in the FB_PATHS variable. Be sure to restrict that data in your Security Rules. ****************************************************/ exports.paths = [{ path: "users", index: "firebase", type: "user" }, { path: "messages", index: "firebase", type: "message", fields: ['msg', 'name'], filter: function(data) { return data.name !== 'system'; } }]; ``` This is the structure of the user node in Firebase (it's located at the root). ``` "users" : { "userid123" : { "friends" : { "userid42" : 1 }, "recs" : { "-recid1234" : 0 }, "user_info" : { "createdAt" : 1475157596, "email" : "e@male.org", "name" : "Firstname Lastname Johnson", "profilePicture" : "png.jpg", "pushId" : { }, "username" : "userLooser" } }, "userid42" : { "friends" : { "userid123" : 1, "test1" : 1 }, "recs" : { "-recid5678" : 0 }, "user_info" : { "email" : "email@icloud.com", "name" : "Firstname Lastname", "phoneNumber" : "", "profilePicture" : "jpg.png", "pushId" : { }, "username" : "userName" } } } } ``` What I want to achieve is to search for username, and return the top results. All i really need in return is the username and user id's. But I get the response below, when I query with the object in the image above: ``` "search" : { "response" : { "index" : { ".priority" : 1.477326417519E12, "hits" : [ { "_id" : "test1", "_index" : "firebase", "_score" : 1, "_source" : { "friends" : { "test1" : 1 }, "recs" : { }, "user_info" : { "createdAt" : 1475157596, "name" : "Testbruker 1", "username" : "testuzer" } }, "_type" : "user" }, { "_id" : "userid123", "_index" : "firebase", "_score" : 1, "_source" : { "friends" : { }, "recs" : { }, "user_info" : { "email" : "", "name" : "Firstname Lastname", "phoneNumber" : "", "profilePicture" : "", "pushId" : { }, "username" : "profileName" } }, "_type" : "user" }, { "_id" : "userid42", "_index" : "firebase", "_score" : 1, "_source" : { "friends" : { }, "recs" : { }, "user_info" : { "createdAt" : 1475157596, "email" : "e@male.org", "name" : "Firstname Lastname Johnson", "profilePicture" : "", "pushId" : { }, "username" : "userLooser" } }, "_type" : "user" } ], "max_score" : 1, "total" : 3 }, "query" : { ".priority" : 1.477326417484E12, "hits" : [ { "_id" : "test1", "_index" : "firebase", "_score" : 1, "_source" : { "friends" : { "test1" : 1 }, "recs" : { }, "user_info" : { "createdAt" : 1475157596, "name" : "Testbruker 1", "username" : "testuzer" } }, "_type" : "user" }, { "_id" : "userid42", "_index" : "firebase", "_score" : 1, "_source" : { "friends" : { }, "recs" : { }, "user_info" : { "email" : "", "name" : "Firstname Lastname", "phoneNumber" : "", "profilePicture" : "", "pushId" : { }, "username" : "profileName" } }, "_type" : "user" }, { "_id" : "userid123", "_index" : "firebase", "_score" : 1, "_source" : { "friends" : { }, "recs" : { }, "user_info" : { "createdAt" : 1475157596, "email" : "e@male.org", "name" : "Firstname Lastname Johnson", "profilePicture" : "", "pushId" : { }, "username" : "userLooser" } }, "_type" : "user" } ], "max_score" : 1, "total" : 3 }, "type" : { ".priority" : 1.477326417503E12, "hits" : [ { "_id" : "test1", "_index" : "firebase", "_score" : 1, "_source" : { "friends" : { "test1" : 1 }, "recs" : { }, "user_info" : { "createdAt" : 1475157596, "name" : "Testbruker 1", "username" : "testuzer" } }, "_type" : "user" }, { "_id" : "userid42", // Same content as above }, { "_id" : "userid123", // Same content as above } ], "max_score" : 1, "total" : 3 } } } ``` Which: - A) Does not return the correct sorting. The first result didn't match my search query at all, it just looks like it returns all users in random order. - B) The response is really messy. under `/response` there is three nodes `query`, `type` and `index` all containing lot's of hits. So, anyone know what's wrong? How can I create a query, or craft the path monitor so the search will work? And maybe get a simpler response?

Original source