Excel Advanced data entry slows down majorly after a number of rows populated

Sounds like a low memory issue. Use the Task Manager and monitor your memory and even your CPU utilization.

Rather than reading the CSVs into memory (data tables), consider looping through the CSVs and putting the data into the Excel file at the same time.

I had thought the same, which is why I started setting up the test task to narrow things down.

The system memory doesn’t appear to be any bottleneck, it stays under 60%. Is there a limit for the Java application?

CPU hovers around 80-100 % while running the test with no slowdown. I’m also able to do other things while it runs.

I switched the loop over to directly reading the CSV session instead of a data table.

Doesn’t seem to be any change here.

I do agree that it seems to be a system limitation, but I haven’t been able to replicate it in my test task, so I am unsure what is causing it.

Your CPU running so high means it’s seriously struggling. I’ve only seen those CPU levels on Atom- or Celeron-based PCs. Thirty-two bit Java applications can’t access more than 4GB of RAM and will struggle at far lower levels than that.

For what it’s worth: The minimum system requirements for a bot runner are 8GB of RAM and a quad-core CPU. More is better, of course.

Processing the CSV directly should have a profound affect on memory unless these CSV files are very small when reading into the data tables (memory).

I’ll try running a similar automation that reads a CSV and writes an XLSX file on my impossibly slow Dell/Wyse 5470. This is a computer that was designed to be a dumb terminal and I’m running Windows 11 on it. Even the Bot Agent took about 5 minutes to install. It has a quad-core Celeron N4100 processor (no hyperthreading), 16GB RAM, and 512GB of SSD.

This poor computer is able to process a CSV with 5000 lines (3 columns) and write an XLSX file in approximately 14 seconds.

The machine has 32GB of RAM, and it doesn’t appear that the java app is using anywhere near that while running.

The CPU is an Intel i5-8250U which is 4 core 1.6GHz (3.4GHz max with the Intel “Turbo boost”)

Your results seem to align with my test task on the same machine that struggles with my main task.

It seemingly has something to do with the other sub task that gets called from my main one. (This also calls another sub task)

These tasks are shared tasks that set up constant variables like folder/file locations and which modules to run for the bot process.

After removing this for testing, the CSV to Excel process runs without any slowdown.

Any thoughts on this? It is behaving as though the subtasks don’t get unloaded from system resources after they are called.

That machine should be more than powerful enough. Good! One thing that’s not a question.

Sub-tasks do take a lot of resources to load and unload. They do release the resources after being called but it’s still having to load and unload as the parent bot runs. You can watch the memory usage over time and see the increase during load and decrease during unload. It is possible that something isn’t unloading properly. Maybe a global session?

Having a Task Bot: Run in a Loop, for example, will run very slowly as it’s having to load and unload resources repeatedly. Moving that loop to the child bot and executing it there is significantly faster.

The constant variables shouldn’t have any significant impact on performance. Neither would global variables if you use them.

Let me know if you see a memory leak or something and we can investigate.

Doesn’t sound like these should impact things then.

My sub tasks only read through an Excel config file at the start of the bot and return a set of variables.

So it isn’t calling a sub task each loop iteration thankfully.

The sub tasks shouldn’t have any globally defined sessions. That would be a deliberate choice and I don’t recall ever using that option. I double checked my subtasks and all the sessions are listed as local.

I even pulled in the same subtasks to my testing bot and had no slowdown.

I might be throwing in the towel on this one.

I moved over to a VM to run the bot and had no issues.

The fact that my brand new test bot with the same steps is behaving normally, but my checked-out version is having slowdown leads me to think there may be some caching issues with the downloaded resources locally.

I did delete all the cache folders and tested again, but didn’t seem to make any difference.

The slowdown happens consistently at row 162 column G.

This is midway through one of the CSV files, but not the largest one. It also doesn’t improve for the remainder of the bot.

It doesn’t make any sense to me since the same exact input data works on a newly created taskbot.

Edit:

I created another new task and copied the entire contents of the bot steps within my try/catch block to the new task.

Now I get the same slowdown, so there is something causing it, but I cannot figure out what.