On-Premise ControlRoom Server sudden insuficent disk space

Hallo again,

I’ve found a folder causing the problem:

it’s the ...\Users\svc_AA\AppData\Local\Temp\newrelic-security-agent ……….. .jar files

What are these files and why aren’t they removed auutomaticly? Are the incorporated with AAe Java service somehow?

@MichalTurek 

Those files dont seem generated by Automation Anywhere. CR  creates log files under:

 C:\ProgramData\AutomationAnywhere (default)

Your enviroment are using  NewRelic for monitoring/securing  (third part agent ) and it is filling your drive.

Regards

Thanks @Raul Jaimes 

I’ve noiced that turning off the services:

Automation Anywhere Control Room IQ Bot Service

Automation Anywhere Control Room Robotic Interface Service

Automation Anywhere Control Room Discovery Bot Service 
Automation Anywhere Control Room Discovery Bot ML Service

Automation Anywhere Control Room Storage Service

prevented these files from being produced.

What is the mechanism for such behaviour?

greeting Michal

@MichalTurek  All AA services are self healed. i.e., if your windows or any server related issues including network causing issue will try to restart these services. If there is not enough space or heap memory available these services will be unable to start. 

Suggesting that you can remove older log files, dump files, older AA imported files. which will free up some space also check windows even listener logs to find out what causing this memory issues. 

I guess due to internal server issue keep dumps created which is going to pile up and consume all your disk space

I’ve noiced that turning off the services:

Automation Anywhere Control Room IQ Bot Service  - if you are not using IQ bot or document automation, turning off this service will not cause any issue

Automation Anywhere Control Room Robotic Interface Service - if you are not using AARI, it is ok to stop this service 

Automation Anywhere Control Room Discovery Bot Service 
Automation Anywhere Control Room Discovery Bot ML Service

Check if you have any other java running in Task manager and if those services have:

args:

-javaagent:/full/path/to/newrelic.jar

https://docs.newrelic.com/docs/apm/agents/java-agent/additional-installation/include-java-agent-jvm-argument/

Regards

Hi @Raul Jaimes 

Thanks for Your input, that’s what I found out:

The only Java tasks are these:

and all those are related to ...\Program Files\Automation Anywhere\Automation360\jdk11\bin  folder.

These two services doesn’t impact the server behaviour at all:

Automation Anywhere Control Room Discovery Bot Service 
Automation Anywhere Control Room Discovery Bot ML Service

Turning ON the service “Automation Anywhere Control Room Storage Service” results in increasing the CPU usage by est. 40%, but doesn’t trigger dump files generation.

The problem lies in those two services:

Automation Anywhere Control Room IQ Bot Service

Automation Anywhere Control Room Robotic Interface Service

turning them ON results in mass dump files reproducing such as:

“agent-bridge<digits>.jar”

“instrumentation<digits>.jar”

“newrelic-bootstrap<digits>.jar”

“newrelic-api<digits>.jar”

“newrelic-security-agent<digits>.jar”

What could be the issue?

@Raul Jaimes 

Im not so sure about that, becouse according to Complience document “OSS_Automation 360 Version .33 OSS Disclosure_2024 06 14” these OSS components comes with AAe360:

“New Relic Java agent bridge datastore API 8.8.1 : Apache License 2.0
New Relic Java Agent Jar 7.11.1 : Apache License 2.0
New Relic Java Agent Jar 7.4.0 : Apache License 2.0
New Relic Java Agent Jar 8.11.1 : Apache License 2.0
New Relic Java Agent Jar 8.5.0 : Apache License 2.0
New Relic Java agent Scala Weaver API 8.4.0 : Apache License 2.0
New Relic Java agent Weaver API 8.11.0 : Apache License 2.0
New Relic Java agent Weaver API 8.3.0 : Apache License 2.0
New Relic Java API Jar 8.10.0 : Apache License 2.0
New Relic Java API Jar 8.6.0 : Apache License 2.0
New Relic Java API Jar 8.8.1 : Apache License 2.0
newrelic-module-util-java 2.1 : Apache License 2.0”

Hello,

I faced the exact same error. I got a “Bad Gateway” error and found out about 120 GB of new relic .jar files.

Is there a solution to this problem ? I have deleted files and restarted the control room process, then restarted the virtual machine but the problem is still there

Hallo @Augustin 

for me turning off the services as mentioned above helped, but I still haven’t found out what is the real matter of this issue,

greeings

For me, as soon as I turn on the main service (“Automation Anywhere Control Room Service”) the files start being generated. And even if I turn it on knowing that files are being generated I still have a “Bad Gateway” error

@Augustin do You have everything with licensing sorted out properly? When have problems begun?

Has everything been working correctly up to now? Did You switch licensing?

There is no licensing problem AFAIK (I am not responsible for the licenses in my job, but it was far from expiring when I last checked).

I have recently started using AARI (a few weeks ago) and have done a lot of testing yesterday. Given the speed at which the c:\ drive is filling up, the trigger of that problem can’t have been before the week-end.

I can safely start the processes elasticsearch, compiler, caching, discovery (x 2), reverse proxy, messaging, storage service!-->

However, starting the processes main (the Control Room Service), iqbot & robotic interface service, it starts filling up the Temp folder with files!-->

I’m affraid i won’t help. You should rise a ticket probably, let us know what was the problem.

greetings

I have just read in the release notes of v.33 https://docs.automationanywhere.com/bundle/enterprise-v2019/page/v33-release-automation-workspace.html “ After you update to this release, when the .jar files are created by Newrelic in the Java TEMP folder, the older files are now cleared correctly. “

Is this the error we are talking about ?

Should I update from v.32 to v.33 ? (The control room is currently down, I absolutely would not like to upgrade in this condition)

Yeah maybe thats the solution,

After desperately searching for information I opened the file WebCR.log and found that the SQL password had expired. Changing the SQL password seemed to solve the error.

I do not know why that would generate a huge bunch of files and never notice me about that, but for now the problem seem corrected.

What a waste of time to not have useful error messages and notifications and having to search for it on our own, it is not the first time this kind of thing happens with AA.