Elasticsearch Index Generation with Java Custom Annotation in spring boot and Elasticsearch Java Client

8 min readDec 22, 2024

Hi everyone;

You choose to use Elasticsearch-Java-Client instead of spring-boot-starter-data-elasticsearch in your application. Especially if you use Spring Boot 2 version in your application, There are some problems with Spring Boot 2 version and Elasticsearch 8 version.So either choose Elasticsearch-Java-Client instead of spring-boot-starter-data-elasticsearch or choose to use Spring Boot 2, in this article I show you that integration of spring boot and elasticsearch java client.

Code repo : Github

Firstly, you can create a spring boot project or choose your existing spring boot project.

If you want to use docker compose in your project

version: '3.8'
 services:
   elasticsearch:
     image: docker.elastic.co/elasticsearch/elasticsearch:8.7.1
     expose:
       - 9200
     environment:
       - xpack.security.enabled=false
       - "discovery.type=single-node"
       - ELASTIC_USERNAME=elastic
       - ELASTIC_PASSWORD=xxxxxx
     networks:
       - es-net
     ports:
       - 9200:9200
     volumes:
       - elasticsearch-data:/usr/share/elasticsearch/data
   kibana:
     image: docker.elastic.co/kibana/kibana:8.7.1
     environment:
       - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
     expose:
       - 5601
     networks:
       - es-net
     depends_on:
       - elasticsearch
     ports:
       - 5601:5601
     volumes:
       - kibana-data:/usr/share/kibana/data
 networks:
   es-net:
     driver: bridge
 volumes:
   elasticsearch-data:
     driver: local
   kibana-data:
     driver: local

You can change username and password.

1.Add elasticsearch-java dependency in pom.xml

<dependency>
   <groupId>co.elastic.clients</groupId>
   <artifactId>elasticsearch-java</artifactId>
   <version>8.9.0</version>
   <exclusions>
    <exclusion>
     <groupId>jakarta.json</groupId>
     <artifactId>jakarta.json-api</artifactId>
    </exclusion>
   </exclusions>
  </dependency>

2. Create config class to integrate with elasticsearch

@Configuration
@RequiredArgsConstructor
public class ElasticJavaConfig {

    @Bean
    public ElasticsearchClient elasticsearchClient() {

        final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
        credentialsProvider.setCredentials(AuthScope.ANY,
                new UsernamePasswordCredentials("username", "password"));

        RestClientBuilder.HttpClientConfigCallback httpClientConfigCallback = new RestClientBuilder.HttpClientConfigCallback() {
            @Override
            public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
                httpClientBuilder.disableAuthCaching();
                return httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
            }
        };

        RestClientBuilder.RequestConfigCallback requestConfigCallBack = new RestClientBuilder.RequestConfigCallback() {
            @Override
            public RequestConfig.Builder customizeRequestConfig(RequestConfig.Builder requestConfigBuilder) {
                return requestConfigBuilder
                        .setConnectTimeout(4000)
                        .setSocketTimeout(60000);
            }
        };



        RestClientBuilder restClientBuilder = RestClient.builder(new HttpHost("localhost", 9200,"http"));
        restClientBuilder.setHttpClientConfigCallback(httpClientConfigCallback);
        restClientBuilder.setRequestConfigCallback(requestConfigCallBack);

        RestClient restClient = restClientBuilder.build();


        ElasticsearchTransport transport = new RestClientTransport(restClient, new JacksonJsonpMapper());
        return new ElasticsearchClient(transport);

    }
}

You can change username ,password, host and port .
If you want to use https, set https instead of http.

3. Create Elasticsearch index model class

public class ProjectElkModel {

     private String id;
     private String name;
     private LocalDateTime createdDate;
     private BigDecimal cost;
     private String manufacturer;



 }

Later, we will change it!
Note : In normally, if we use spring-boot-starter-data-elasticsearch, we write @Document annotation. But now, we use Elasticsearch Java Client. So we generate our custom annotation and when project runs, our index will be created.

4.Let’s create custom annotation named GenericElkIndex

@Documented
 @Target({ElementType.TYPE})
 @Retention(RetentionPolicy.RUNTIME)
 public @interface GenericElkIndex {

     String indexName() default "";
 }

Create a Custom annotation which has indexName property

5.Add this annotation to our index model class


 @GenericElkIndex(indexName = "project")          
 public class ProjectElkModel {

  private String id;
     private String name;
     private LocalDateTime createdDate;
     private BigDecimal cost;
     private String manufacturer;

 }

index name of model is “project”

6.We create a BaseClass for our all index model class.

public class BaseElkModel implements Serializable {

     private String id;

     public String getId() {
         return id;
     }

     public void setId(String id) {
         this.id = id;
     }

     public BaseElkModel(String id) {
         this.id = id;
     }
 }

Every model classes extend BaseElkModel parent class.

7.Change index model class

 @Data
 @EqualsAndHashCode
 @GenericElkIndex(indexName = "project")
 public class ProjectElkModel extends BaseElkModel {

     private String name;
     @JsonFormat(pattern = "yyyy-MM-dd'T'HH:mm:ss.SSS")
     private LocalDateTime createdDate;
     private BigDecimal cost;
     private String manufacturer;

 }

ProjectElkModel extends BaseElkModel.

8.We create a interface class (IElasticOperationService) for all elasticsearch operations.

Now this interface has 2 methods.

public interface IElasticOperationService {
      
      boolean checkIfExistIndex(String indexname)  throws IOException;

      String createIndex(String indexname)  throws IOException;
  }

checkIfExistIndex

check indexname exists according to given indexname parameter

createIndex

create index according to given indexname parameter

9. Create ElasticOperationService implements IElasticOperationService

@Service
 public class ElasticOperationService implements IElasticOperationService{

     private final ElasticsearchClient elasticsearchClient;

     @Autowired
     public ElasticOperationService(ElasticsearchClient elasticsearchClient, ObjectMapper objectMapper) {
         this.elasticsearchClient = elasticsearchClient;
         this.objectMapper = objectMapper;
     }

     @Override
     public boolean checkIfExistIndex(String indexname) throws IOException {

         BooleanResponse check =  elasticsearchClient.indices().exists(ExistsRequest.of(e-> e.index(indexname)));

         if(Objects.nonNull(check) && check.value())
         {
             return true;
         }

         return false;
     }

     @Override
     public String createIndex(String indexname) throws IOException {

         CreateIndexRequest req = CreateIndexRequest.of(c -> c
                 .index(indexname)
         );

         CreateIndexResponse createIndexResponse = elasticsearchClient.indices().create(req);

         return createIndexResponse.index();
     }
 }

Now it’s ready some operations for elasticsearch.

10. I have a listener to create all defined indexes when project runs

I create a spring event listener (ElkIndexCreatorListener) implements ApplicationListener<ApplicationReadyEvent>

@Component
 @Slf4j
 public class ElkIndexCreatorListener implements ApplicationListener<ApplicationReadyEvent> {

     private static final Logger LOG = LogManager.getLogger(ElkIndexCreatorListener.class);
     private final IElasticOperationService elasticOperationService;
     private final ConfigurableApplicationContext applicationContext;

     public ElkIndexCreatorListener(IElasticOperationService elasticOperationService, ConfigurableApplicationContext applicationContext) {
         this.elasticOperationService = elasticOperationService;
         this.applicationContext = applicationContext;
     }


     @Override
     public void onApplicationEvent(ApplicationReadyEvent event) {

         LOG.info("elk model scan listener starting");

         scanElkModelAnnotation();

         LOG.info("elk model scan listener ending");
     }

     private void scanElkModelAnnotation() {

         ClassPathScanningCandidateComponentProvider provider =
                 new ClassPathScanningCandidateComponentProvider(false);
         provider.addIncludeFilter(new AnnotationTypeFilter(GenericElkIndex.class));

         Set<BeanDefinition> beanDefs = provider
                 .findCandidateComponents("tr.salkan.code");
         List<String> annotatedBeans = new ArrayList<>();
         for (BeanDefinition bd : beanDefs) {
             if (bd instanceof AnnotatedBeanDefinition) {
                 Map<String, Object> annotAttributeMap = ((AnnotatedBeanDefinition) bd)
                         .getMetadata()
                         .getAnnotationAttributes(GenericElkIndex.class.getCanonicalName());
                 annotatedBeans.add(annotAttributeMap.get("indexName").toString());
             }
         }

         LOG.info("found total index count :" +  annotatedBeans.size());

         for(int i = 0; i < annotatedBeans.size(); i++)
         {
             if(!checkIfExistIndex(annotatedBeans.get(i)))
             {

                 try {
                     createIndex(annotatedBeans.get(i));
                 } catch (IOException e) {
                     LOG.error("hata",e);
                 }
             }
         }

     }

     private boolean checkIfExistIndex(String indexname) {

         try {
             return elasticOperationService.checkIfExistIndex(indexname);
         } catch (IOException e) {
             LOG.error("hata",e);
         }

         return false;
     }

     private String createIndex(String indexname) throws IOException {

         String createdIndex =  elasticOperationService.createIndex(indexname);

         return createdIndex;

     }

 }

When project runs, ElkIndexCreatorListener listens on onApplicationEvent method and runs scanElkModelAnnotation method.
scanElkModelAnnotation scans all class with annotated GenericElkIndex annotation and get indexName property value. These values are added.
These values are added to annotatedBeans list.
On loop, every value checks if index exists.Unless index exists, create index.

11. Run application

You can see logs

 t.s.c.j.e.l.ElkIndexCreatorListener      : elk model scan listener starting
 t.s.c.j.e.l.ElkIndexCreatorListener      : found total index count :1
 t.s.c.j.e.l.ElkIndexCreatorListener      : elk model scan listener ending

If you have many model classes which are annotated with GenericElkIndex, you can see, scanning index count are increasing.

12) Check Kibana

You visit your Kibana host.

Stack Management -> Index Management

Your model class is created.

13) Insert model to elasticsearch

Firstly create a Controller class and POST API

@RestController
 @RequestMapping("/elastic-client-service")
 @RequiredArgsConstructor
 public class ElasticJavaClientController {

     @PostMapping(value = "/saveModel")
     public ResponseEntity<String> saveModel(@RequestBody ProjectElkModel projectElkModel)
     {
         
         return new ResponseEntity<>(HttpStatus.CREATED);
     }
 }

Before depending service class which adds model to elasticsearch, in IElasticOperationService interface, I will create methods which adds model to elasticsearch.

I will create 2 generic method to insert my model to elasticsearch.

 <T extends BaseElkModel> String insertModelGeneric (T model) throws IOException;

 <T extends BaseElkModel> String insertModelGeneric (String id,T model) throws IOException;

First method, it is generic method which gives (T) model object. This object extends BaseElkModel. Because our model class extends BaseElkModel. In this line, I define model class types for my generic methods.

Second method, it is generic method which gives (T) model object and id .This object extends BaseElkModel. Because our model class extends BaseElkModel. In this line, I define model class types for my generic methods and also maybe you want to different generate id for your object.

Now impelementing our methods in ElasticOperationService class.

       @Override
       public <T extends BaseElkModel> String insertModelGeneric(T model) throws IOException {

           GenericElkIndex genericElkIndex =  model.getClass().getAnnotation(GenericElkIndex.class);

           if(Objects.nonNull(genericElkIndex))
           {
               String indexName =  genericElkIndex.indexName();

               IndexRequest<T> request = IndexRequest.of(i->
                       i.index(indexName)
                               .id(model.getId())
                               .document(model).refresh(Refresh.True));

               IndexResponse response = elasticsearchClient.index(request);
               return response.result().toString();
           }

           throw new IOException("Can not create document ID :" + model.getId());
       }

       @Override
       public <T extends BaseElkModel> String insertModelGeneric(String id, T model) throws IOException {
           GenericElkIndex genericElkIndex =  model.getClass().getAnnotation(GenericElkIndex.class);

           if(Objects.nonNull(genericElkIndex))
           {
               String indexName =  genericElkIndex.indexName();

               IndexRequest<T> request = IndexRequest.of(i->
                       i.index(indexName)
                               .id(id)
                               .document(model).refresh(Refresh.True));

               IndexResponse response = elasticsearchClient.index(request);
               return response.result().toString();
           }

           throw new IOException("Can not create document ID :" + id);
       }

Firstly, check model class’s annotation. If model class has GenericElkIndex annotation, get indexName from GenericElkIndex annotation. Our model has GenericElkIndex annotation and its indexName value is project.

Remember our model class

       @Data
       @EqualsAndHashCode
       @GenericElkIndex(indexName = "project")
       public class ProjectElkModel extends BaseElkModel

Later, Elasticsearch Java Client has IndexRequest. Set indexName, id and model to IndexRequest. After inserting, if you search your data you have to set Refresh.True.

Now I will depend IElasticOperationService for our controller class.

@RestController
       @RequestMapping("/elastic-client-service")
       public class ElasticJavaClientController {

           private final IElasticOperationService elasticOperationService;

           @Autowired
           public ElasticJavaClientController(IElasticOperationService elasticOperationService) {
               this.elasticOperationService = elasticOperationService;
           }

           @PostMapping(value = "/saveModel")
           public ResponseEntity<String> saveModel(@RequestBody ProjectElkModel projectElkModel)
           {

               try {
                   String response = elasticOperationService.insertModelGeneric(projectElkModel);
                   return new ResponseEntity<>(response,HttpStatus.CREATED);
               } catch (IOException e) {
                   return new ResponseEntity<>(HttpStatus.INTERNAL_SERVER_ERROR);
               }

           }
       }

14. Run application and test it

If you get error :

Java 8 date/time type `java.time.LocalDateTime` not supported by default: add Module “com.fasterxml.jackson.datatype:jackson-datatype-jsr310”

You can create custom LocalDateTimeSerializer and LocalDateTimeDeserializer classes.

LocalDateTimeSerializer class:

 public class LocalDateTimeSerializer extends StdSerializer<LocalDateTime> {

     private static final long serialVersionUID = 1L;

     public LocalDateTimeSerializer() {
         super(LocalDateTime.class);
     }

     @Override
     public void serialize(LocalDateTime value, JsonGenerator gen, SerializerProvider sp) throws IOException, JsonProcessingException {
         gen.writeString(value.format(DateTimeFormatter.ISO_LOCAL_DATE_TIME));
     }
 }

LocalDateTimeDeserializer class:

public class LocalDateTimeDeserializer extends StdDeserializer<LocalDateTime> {

     private static final long serialVersionUID = 1L;

     protected LocalDateTimeDeserializer() {
         super(LocalDateTime.class);
     }

     @Override
     public LocalDateTime deserialize(JsonParser jp, DeserializationContext ctxt)
             throws IOException, JsonProcessingException {
         return LocalDateTime.parse(jp.readValueAs(String.class));
     }
 }

Define them in your model class

 @JsonSerialize(using = LocalDateTimeSerializer.class)
 @JsonDeserialize(using = LocalDateTimeDeserializer.class)
 @JsonFormat(pattern = "yyyy-MM-dd'T'HH:mm:ss.SSS")
 private LocalDateTime createdDate;

If you can post model to elasticsearch successfully, you get “CREATED” response.Later, check our data in Kibana.