Audit Logs not showing up (sharding limit reached)

  • 13 January 2023
  • 7 replies
  • 418 views
Andoni
Userlevel 3
Badge +8

Hello there,

Context

  • I’m currently working on a customer that has a fully on-prem A360 installation. This was installed like 2 years ago so Audit Log size is considerable (+500k logs).
  • Disk space (C drive) is more than free (+250GB free)

Problem

  • Recent Audit Logs not showing up upon “Audit Logs” tab
  • WebCR log explicitly states: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add [10] total shards, but this cluster currently has [992]/[1000] maximum shards open;]

I’ve thought about increasing sharding limit but this is not recommended by Elasticsearch in the mid-long term

Please, do not send me a link to some article that states that this is due to low disk space and changing “index.blocks.read_only_allow_delete” to false is the solution, because it is not (no single index has this setting set to true)

Thank you🤗

 

7 replies

Shyam Prakash
Badge

Hi @Andoni,

 

We are facing this similar issue. Is it resolved? Please let me know how it resolved.

rbkadiyam
Rank
Userlevel 5
Badge +14

@Andoni @Shyam Prakash 

  • stop elastic service
  • Rename data folder to data_bkp (C:\ProgramData\AutomationAnywhere\elasticsearch)
  • Restart the elastic service
Ramesh B Kadiyam
Shyam Prakash
Badge

@Andoni @Shyam Prakash 

  • stop elastic service
  • Rename data folder to data_bkp (C:\ProgramData\AutomationAnywhere\elasticsearch)
  • Restart the elastic service

 

@rbkadiyam It doesn’t worked. Got Generic server exception in CR. Audit log entries are not getting populated.

Andoni
Userlevel 3
Badge +8

Hi @Shyam Prakash . Let’s go point by point:

 

  • I do not support (at all) what @rbkadiyam suggests. I would not rename or mess up (at first) with those directories
  • @Shyam Prakash , which version of A360 are you using? Is it on-premises? How many nodes your deployment has?
  • According to AA support this issue has been fixed from A360 v27 on
Shyam Prakash
Badge

Hi @Andoni,

  • We are upgrading from A360 v.22 to v.27 (v.22 to v.25 and then v.25 to v.27)
  • This is on-premise 
  • We have single node
  • After upgrading to v27, last 3 months audit logs are not showing up and also new audit entries are not getting populated.

Tried below steps but it doesn’t worked out, not sure where I am going wrong

  1. https://apeople.automationanywhere.com/s/article/Automation-360-Elasticsearch-password-repair-steps
  2. https://docs.automationanywhere.com/bundle/enterprise-v2019/page/enterprise-cloud/topics/deployment-planning/on-prem-install/install-elasticsearch-credentials.html
  3. https://apeople.automationanywhere.com/s/article/A2019-Failed-to-connect-to-Elasticsearch-Server-error-in-Audit-logs
Shyam Prakash
Badge

@Andoni @rbkadiyam - It has been solved by freeing up the shards using API calls.

Andoni
Userlevel 3
Badge +8

Hello @Shyam Prakash,

 

I tried to answer you here these days but somehow the forum would not allow me…

(I opened a ticket to AA in regards to this and they explicitly told me that from version 27 onwards this was not an issue anymore. I did not believe it as I did not see any explicit note in the release notes stating this issue)

Yes, your solution is also correct, what I did is delete the replicas (as it is a single node deplyoment it has no sense to talk about replicas)

 

This is what solved my issue back then,

 

Sorry for late response and glad that you sorted it out buddy

Reply


Terms & ConditionsCookie settings