ES mapping that fits our use case

[Adiguptakn]

I have created a mappings file which fits our use case after we push the data into ES.

[MMAThijssen]

The mapping is looking nice. Good use of the type keyword for country, city, email and link. The number of shards and replicas is to be determined later on in the project?

[andrewsutjahjo]

There's a couple of things that might interfere here:
json in and of itself doesn't allow for comments. I'm not too sure how it would interact with the es-python package when adding a body though.

Also this doesn't have a clean way to interface with the rest of the codebase as it's written (I believe) to interact with Kibana.

The cleanest way to get a mapping would be to create a function
def get_mapping() -> dict:
which reads the mappings.txt file and turns it into a dict. That dict can then be picked up by whatever function that ultimately creates the index.

Same goes for the settings file

[andrewsutjahjo]

I'd also go for camelcase: contact_person as the rest is in camel case as well.

I'm also not too sure how ES would resolve dots in the naming of keys, but the usual way is:

"contact_person" : {
    "properties" : {
        "name": {"type": "text"},
        "email": {"type": "keyword"},
        "number": {"type": "long"}
    }
}

[Adiguptakn]

Elasticsearch translates fields using a dot notation into using the “properties” parameter behind the scenes. So,

      "contact_person.name": {"type": "text"}, 
      "cotact_person.email": {"type": "keyword"},
      "contact_person.number": {"type": "long"} 

would translate to

"contact_person" : {
    "properties" : {
        "name": {"type": "text"},
        "email": {"type": "keyword"},
        "number": {"type": "long"}
    }
}

[andrewsutjahjo]

There are some optimizations ES has for keywords called normalizers, which you can add into your settings

https://www.elastic.co/guide/en/elasticsearch/reference/current/normalizer.html

[Adiguptakn]

There's a couple of things that might interfere here:
json in and of itself doesn't allow for comments. I'm not too sure how it would interact with the es-python package when adding a body though.

Also this doesn't have a clean way to interface with the rest of the codebase as it's written (I believe) to interact with Kibana.

The cleanest way to get a mapping would be to create a function
def get_mapping() -> dict:
which reads the mappings.txt file and turns it into a dict. That dict can then be picked up by whatever function that ultimately creates the index.

Same goes for the settings file

Thanks for the review.
This mappings was specifically written to interact with Kibana. I will make the necessary changes and create a python function to read the mapping file and turns it into a dict.

[MMAThijssen]

It would be nice to add normalisers to the settings. The example Andrew mentioned shows a good example that makes sure that search queries are lowercase and without accent marks.

The format used in settings.txt and mappings.txt does not follow the format assumed by ES:

      "contact_person.name": {"type": "text"}, 
      "cotact_person.email": {"type": "keyword"},
      "contact_person.number": {"type": "long"} 

"type" is missing here.

[MMAThijssen]

It is not clear to me what file is. Is it settingsFile? The type of that isn't a dictionary but a string (of the path name / file name).

[MMAThijssen]

There are two typos: Properties for creating the index in ES.

[MMAThijssen]

I am not familiar with the method you open the file. What is the purpose of adding the '' in front of settingsFile? It seems redundant.

[MMAThijssen]

Same comment as above. If mappingsFile is meant by file, the type is string. In the code the content of mappingsFile is transformed to a dict.