Skip to main content
Question

On-Premise ControlRoom Server sudden insuficent disk space

Hallo,

I have a Control Room on-premise (rev.33) installed on 500GB disk space VM and it got a sudden disk overusage and now its 0GB space of 499GB available.

The curious thing that checking all the folders on a disk get me a 68GB of disk usage.

I found in the Monitoring logs below inputs:

“ERROR] 2024/07/09 08:46:14.746373 Error: File IO Fail 
Module: FileSystemLatencyMonitoring 
Method: getReadFileLatency 
Error Message: Error occured in Read file operation 
Underlying Error: EOF
pERROR] 2024/07/09 08:46:17.607714  ConfigChangeDetector : Execute : File .\config/cluster.properties doesn't exist
sERROR] 2024/07/09 08:46:24.746745 Error: File IO Fail 
Module: FileSystemLatencyMonitoring 
Method: getWriteFileLatency 
Error Message: Error occured in Write file operation 
Underlying Error: write C:\ProgramData\AutomationAnywhere\Server Files/A360Monitoring/FileSystemLatency/muysRutT2024-07-09-08-46-24_Temp.txt: There is not enough space on the disk.
ERROR] 2024/07/09 08:46:24.746967 Error: File IO Fail 
Module: FileSystemLatencyMonitoring 
Method: getReadFileLatency 
Error Message: Error occured in Read file operation 
Underlying Error: EOF
EERROR] 2024/07/09 08:46:44.755509 Error: File IO Fail 
Module: FileSystemLatencyMonitoring 
Method: getWriteFileLatency 
Error Message: Error occured in Write file operation 
Underlying Error: write C:\ProgramData\AutomationAnywhere\Server Files/A360Monitoring/FileSystemLatency/OoEzqwAt2024-07-09-08-46-44_Temp.txt: There is not enough space on the disk.
eERROR] 2024/07/09 08:46:44.755509 Error: File IO Fail 
Module: FileSystemLatencyMonitoring 
Method: getReadFileLatency 
Error Message: Error occured in Read file operation 
Underlying Error: EOF
eERROR] 2024/07/09 08:46:47.604181  ConfigChangeDetector : Execute : File .\config/cluster.properties doesn't exist”

Is there a possibility thart CR server files reserve a space for a temp files or something?

Or is it some VM settings to look for?

looking for clues,

greeting Michal

7 replies

M
Badge +8

Hallo again,

 

I’ve found a folder causing the problem:

 

it’s the ...\Users\svc_AA\AppData\Local\Temp\newrelic-security-agent ……….. .jar files

 

 

What are these files and why aren’t they removed auutomaticly? Are the incorporated with AAe Java service somehow?

Raul Jaimes
Userlevel 5
Badge +9

@MichalTurek 

Those files dont seem generated by Automation Anywhere. CR  creates log files under:

 C:\ProgramData\AutomationAnywhere (default)

Your enviroment are using  NewRelic for monitoring/securing  (third part agent ) and it is filling your drive.

 

Regards

M
Badge +8

@MichalTurek

Those files dont seem generated by Automation Anywhere. CR  creates log files under:

 C:\ProgramData\AutomationAnywhere (default)

Your enviroment are using  NewRelic for monitoring/securing  (third part agent ) and it is filling your drive.

 

Regards

Thanks @Raul Jaimes 

I’ve noiced that turning off the services:

Automation Anywhere Control Room IQ Bot Service

Automation Anywhere Control Room Robotic Interface Service

Automation Anywhere Control Room Discovery Bot Service 
Automation Anywhere Control Room Discovery Bot ML Service

Automation Anywhere Control Room Storage Service

prevented these files from being produced.

What is the mechanism for such behaviour?

greeting Michal

rbkadiyam
Rank
Userlevel 5
Badge +14

@MichalTurek  All AA services are self healed. i.e., if your windows or any server related issues including network causing issue will try to restart these services. If there is not enough space or heap memory available these services will be unable to start. 

 

Suggesting that you can remove older log files, dump files, older AA imported files. which will free up some space also check windows even listener logs to find out what causing this memory issues. 

 

I guess due to internal server issue keep dumps created which is going to pile up and consume all your disk space

 

I’ve noiced that turning off the services:

Automation Anywhere Control Room IQ Bot Service  - if you are not using IQ bot or document automation, turning off this service will not cause any issue

 

Automation Anywhere Control Room Robotic Interface Service - if you are not using AARI, it is ok to stop this service 

Automation Anywhere Control Room Discovery Bot Service 
Automation Anywhere Control Room Discovery Bot ML Service

 

Ramesh B Kadiyam
Raul Jaimes
Userlevel 5
Badge +9

@MichalTurek

Those files dont seem generated by Automation Anywhere. CR  creates log files under:

 C:\ProgramData\AutomationAnywhere (default)

Your enviroment are using  NewRelic for monitoring/securing  (third part agent ) and it is filling your drive.

 

Regards

Thanks @Raul Jaimes 

I’ve noiced that turning off the services:

Automation Anywhere Control Room IQ Bot Service

Automation Anywhere Control Room Robotic Interface Service

Automation Anywhere Control Room Discovery Bot Service 
Automation Anywhere Control Room Discovery Bot ML Service

Automation Anywhere Control Room Storage Service

prevented these files from being produced.

What is the mechanism for such behaviour?

greeting Michal

 

Check if you have any other java running in Task manager and if those services have:

args:

-javaagent:/full/path/to/newrelic.jar

https://docs.newrelic.com/docs/apm/agents/java-agent/additional-installation/include-java-agent-jvm-argument/

 

Regards

M
Badge +8

@MichalTurek

Those files dont seem generated by Automation Anywhere. CR  creates log files under:

 C:\ProgramData\AutomationAnywhere (default)

Your enviroment are using  NewRelic for monitoring/securing  (third part agent ) and it is filling your drive.

 

Regards

Thanks @Raul Jaimes 

I’ve noiced that turning off the services:

Automation Anywhere Control Room IQ Bot Service

Automation Anywhere Control Room Robotic Interface Service

Automation Anywhere Control Room Discovery Bot Service 
Automation Anywhere Control Room Discovery Bot ML Service

Automation Anywhere Control Room Storage Service

prevented these files from being produced.

What is the mechanism for such behaviour?

greeting Michal

 

Check if you have any other java running in Task manager and if those services have:

args:

-javaagent:/full/path/to/newrelic.jar

https://docs.newrelic.com/docs/apm/agents/java-agent/additional-installation/include-java-agent-jvm-argument/

 

Regards

Hi @Raul Jaimes 

Thanks for Your input, that’s what I found out:

The only Java tasks are these:

and all those are related to ...\Program Files\Automation Anywhere\Automation360\jdk11\bin  folder.

These two services doesn’t impact the server behaviour at all:

Automation Anywhere Control Room Discovery Bot Service 
Automation Anywhere Control Room Discovery Bot ML Service

Turning ON the service “Automation Anywhere Control Room Storage Service” results in increasing the CPU usage by est. 40%, but doesn’t trigger dump files generation.

The problem lies in those two services:

Automation Anywhere Control Room IQ Bot Service

Automation Anywhere Control Room Robotic Interface Service

turning them ON results in mass dump files reproducing such as:

“agent-bridge<digits>.jar”

“instrumentation<digits>.jar”

“newrelic-bootstrap<digits>.jar”

“newrelic-api<digits>.jar”

“newrelic-security-agent<digits>.jar”

What could be the issue?

M
Badge +8

@MichalTurek

Those files dont seem generated by Automation Anywhere. CR  creates log files under:

 C:\ProgramData\AutomationAnywhere (default)

Your enviroment are using  NewRelic for monitoring/securing  (third part agent ) and it is filling your drive.

 

Regards

@Raul Jaimes 

Im not so sure about that, becouse according to Complience document “OSS_Automation 360 Version .33 OSS Disclosure_2024 06 14” these OSS components comes with AAe360:

“New Relic Java agent bridge datastore API 8.8.1 : Apache License 2.0
New Relic Java Agent Jar 7.11.1 : Apache License 2.0
New Relic Java Agent Jar 7.4.0 : Apache License 2.0
New Relic Java Agent Jar 8.11.1 : Apache License 2.0
New Relic Java Agent Jar 8.5.0 : Apache License 2.0
New Relic Java agent Scala Weaver API 8.4.0 : Apache License 2.0
New Relic Java agent Weaver API 8.11.0 : Apache License 2.0
New Relic Java agent Weaver API 8.3.0 : Apache License 2.0
New Relic Java API Jar 8.10.0 : Apache License 2.0
New Relic Java API Jar 8.6.0 : Apache License 2.0
New Relic Java API Jar 8.8.1 : Apache License 2.0
newrelic-module-util-java 2.1 : Apache License 2.0”

Reply


Terms & ConditionsCookie settings