T01:28:25 INFO esdedupe Found 1,291 duplicates out of 168,842 docs, unique documents: 167,551 (0.8% duplicates)Д % | | 1/1291 T01:28:37 INFO esdedupe Deleted 2,582 documents (including shard replicas) T01:28:02 INFO esdedupe Found 1,321 duplicates out of 165,664 docs, unique documents: 164,343 (0.8% duplicates)Д % | | 1/1321 T01:28:15 INFO esdedupe Deleted 2,642 documents (including shard replicas) T01:27:33 INFO esdedupe Found 1,338 duplicates out of 162,882 docs, unique documents: 161,544 (0.8% duplicates)Д % | | 1/1338 T01:27:53 INFO esdedupe Deleted 2,676 documents (including shard replicas) T01:27:07 INFO esdedupe Found 1,544 duplicates out of 162,805 docs, unique documents: 161,261 (0.9% duplicates)Д % | | 1/1544 T01:27:25 INFO esdedupe Deleted 3,088 documents (including shard replicas) WARNING: Running huge bulk operations on Elastic cluster might influence performance of your cluster or even crash some nodes if heap is not large enough.ДЄ sliding window -w / -window could be used to prevent running out of memory on larger indexes (if you have a timestamp field): T03:42:01 INFO esdedupe Successfully completed duplicates removal. T20:01:37 INFO esdedupe Memory usage: 1.2GB T20:01:37 INFO esdedupe Scanned 3,579,286 unique documents T20:00:46 INFO esdedupe Scanned 2,861,671 unique documents T20:00:03 INFO esdedupe Memory usage: 695.0MB T20:00:03 INFO esdedupe Scanned 1,950,957 unique documents T19:59:16 INFO esdedupe Memory usage: 414.0MB T19:59:16 INFO esdedupe Scanned 987,892 unique documents T19:58:25 INFO esdedupe Building documents mapping on index: nginx_access_logs-2021.01.29, batch size: 10000

T19:58:25 INFO esdedupe elastic: es01, host: localhost, version: 7.6.0

'buckets_path': 'employee>distinct_sum_feedback.value' Loop through dictionary to get sum or count of feedback return sum ' Once again go through through all dictionary returned from each shard and create a single unique dictionary. Loop through all document and add unique identifierid and corresponding feedback value toД®xecuted once on each shard after document collection is completeД®xecuted once on the coordinating node after all shards have returned their results I am using scripted metric aggregation to create a dictionary of unique feedbackIdentifierI and feedback value.Д®xecuted prior to any collection of documents. Total sum is 10 and using distinct sum is 8 I have taken one employee with id 1 and three feedback (two duplicate for identifierId 1 and one for identifierId2).

Es deduplicator