Understanding Elastic Search

elasticsearch

Solution

Please see below for answers to your points.

- Why do I need do to add number_of_shards and number_of_replicas to index creation? I did look here http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules.html but ironically it leaves those two settings out :/

You don't "have" to, but you probably should in especially will want to in production. The default is five shards and one replication.

The number of replications defined is just the number of times your entire index is replicated throughout all of the nodes in your elasticsearch cluster. Think of it as being multiple read copies of a RDBMS database (but in this case, we read and write all copies).

A shard is the number of times I split up, or shard, an index. So, I can have an index with a single shard, or I can have an index with multiple shards. This is similar in concept to sharding a RDBMS database by primary key, but not identical.

So, the total number of shards you will have in an index is the product of number_of_shards and number_of_replicas.

When you do a search, elasticsearch will distribute your search to all possible nodes containing the shards in your index and aggregagate the result for you. You can think of this as a map/ reduce where the map is sending the search out to each shard and the reduce is collecting the results.

Also, you can change the replication number_of_replicas at any time, but you can never change the number_of_shards. This must be set at index creation.

- How can I have 3 shards with 2 replicas? If the glossary is anything to go by shouldn't that be impossible considering that a shard is "is a single Lucene instance"?

I think the above mostly answers this, but it's important to remember that elasticsearch is primarily a distributed computing solution to search. We are splitting the work up to multiple shards and possibly machines.

- If I add more nodes later how can I change these values to span the new nodes?

Once the cluster is aware of another node in the cluster, no other action is needed by you. The settings propagate throughout the cluster on their own. In your above example of three shards and two replicas, if you had two nodes initially and added a third, each node will have on average two shards per node, this shard movement happens without your intervention (again, provided the cluster is aware of the new node)

- How does sharding work in ES?

See above

- How does replica sets work in ES?

See above

- How can I manage sharding? I understand it is auto join ( http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html#cluster-name ) but how do I define the difference between replicas and shards?

You don't have to "manage" it actively. As stated earlier, sharding and everything else you define at index creation, is propagated to new nodes within the cluster.

You define replicas and shards like this:

{
    "settings": {
        "index": {
            "number_of_shards": 20,
            "number_of_replicas": 1
        }
    },
    "mappings": {
        "some_type": {
            "properties": {
                "some_field": {
                    "type": "long"
                }
            }
        }
    }
}

- How can I manage replica sets? I.e. how do I add replicas, promote primaries etc?

You do that through the update indices API, documentation for this specific case is found on there site here:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-update-settings.html

I just noticed your edit, please see below:

- I am also unsure how auto-discovery works on a distributed network.

In the YML config file you set the unicast like this:

discovery.zen.ping.multicast.enabled: false
#discovery.zen.minimum_master_nodes: 3
discovery.zen.ping.unicast.hosts: ["ip.add.r.ess", "ip.add.r.ess"]

The middle setting is an important setting, but I commented it out here. That number should always be number of (master nodes / 2) +1. This is to avoid split brain situations. Generally I set all nodes to master eligible.

These settings are for unicast, which is what I think you are going for with your question and not multicast.

Problem

Original source

Related problems

• Shards and replicas in Elasticsearch