Tech Breakdown is the opportunity to understand how developers are implementing Automation Anywhere through the lens of the value-added for their organization, and the technical solution that got them there. This Tech Breakdown was written by former A-Lister, RPA developer, and CoE lead Zeth Baker previously of Republic Airways. 

The COVID-19 pandemic has created a need for flexibility and speed in creating new automation solutions, solving for unforeseen business changes, and increasing our overall development velocity to support our operation. As a regional airline, Republic Airways operates flights 24/7, and every second matters in the operation to make sure our flights depart and arrive on time. If you have ever flown, you know how important it is that flights take off on time, as nobody wants to be delayed while trying to visit their family and friends, go on vacations, or business trips. The bots we build support that goal of helping our flights to take off on time, along with easing the burden and stress of our business partners performing repetitive tasks.

The goal was to create a bot that was flexible, modular, and scalable allowing it to be easily modified as the business needs changed during the pandemic and for the unforeseen future ahead. The first problem to solve was automating some administration of existing bots to allow for more time for new development. The second problem was finding a way to provide the business partners with an easy method to execute their bots ad-hoc without the need of contacting an administrator. From these needs, a new bot was born - affectionately named O.T.I.S. which stands for On Time Integrated Support. O.T.I.S. is helping manage Republic Airways existing 130 bots, automating about 50 processes on ten bot-runners with one user performing the development and administration of these bots within the organization. Now let’s dive into what O.T.I.S. has the capability of doing today which created a self-service model for our business partners:

• Generate and update credential vault passwords when application passwords expire

• Restart bot runner machines and re-connect to the control room

• Perform control room actions using the existing APIs based on a provided email subject:

  • Deploy a bot on demand

  • Schedule a bot for future date/time

  • Stop a bot in progress

  • Enable or Disable a bot schedule

  • Provide a status update of a bot

First, a dedicated bot runner was created to manage this process of routing requests; the single bot runner is using the local Automation Anywhere triggers within v11. This bot runner also has full access to the control room and all its APIs. Let’s now look at the functions a little more in-depth and see how O.T.I.S is delivering value for the organization.

Deploying, Scheduling, Stopping, or Managing Schedules

The Purpose

With so many uncertainties it was best if the business users did not have to learn a new application (using the Automation Anywhere Control Room).  Second, it offers the flexibility to run their bots when they were ready, as the bots no longer could run on their standard schedules due to the evolving needs and changes in the business environment. Third, we needed a way to optimize the existing bot runners so that processes were not queueing up on a single bot runner and that tasks were evenly distributed to speed up the deliverables for each bot. In turn, this provided a better ROI and decreased the infrastructure footprint needed for the bots.

The Tech Breakdown

First, an evaluation is performed based on the type of trigger and the user to determine if the user has access to perform actions via O.T.I.S. which is done using RBAC (Role-Based Access Control).  If the user has the correct roles, we proceed to the next step. If not, the user receives a notification letting them know that their request has been denied due to an access violation.  RBAC is important because we don’t want just any user triggering any bot without knowing the implications or if the bot is ready to run or not. In all these actions communication is also key to the business partners so no matter the outcome a notification is always sent to let them know the outcome of their request. Now O.T.I.S. will determine which action to perform using a combination of the system variables and defined trigger actions for each subcomponent.

• If the requested bot needs to run immediately then the following actions will be performed:

  • First, we query the control room to determine which bot runners this process has been successfully completed on in the past. This ensures the correct applications are available on the bot runner selected.

  • With a list of available bot runners recorded, it’s time to determine which ones don’t have a bot in progress or scheduled within the next hour. This creates the flexibility to manage bots across all bot runners to avoid idle bot runners’ machines.

  • If all the bot runners are being used, then based on the historical average completion time, the bot runner that is estimated to complete the earliest will be selected.

  • Finally, the requested bot is deployed using the Automation Anywhere Bot Execution Orchestrator APIs.

• If the requested bot needs to be scheduled for a future date or time, then a slightly different set of actions will be performed:

  • The control room will be queried for scheduled bots on the requested date/time provided in the user’s email using the bot scheduling API endpoint.

  • If the date/time window is available then the API process will begin, otherwise, the next best option will be calculated and sent to the user for approval. Once the approval email is received the API process will begin.

• If the user needs to manage an existing schedule of a bot such as enabling, disabling, or stopping an in-progress automation, then O.T.I.S. determines the correct action and which bot the user requested. For the actions listed in this section, the control room interface will be used via Google Chrome.

  • First, the bot will query the control room to get the specific activity ID of the bot to be canceled.

  • Once the activity ID has been acquired we can use that to format a URL to launch the correct webpage to cancel the activity. This helps ensure only one bot is being canceled and it is then recorded with the specific ID of that process.

  • If the user wants to enable or disable a schedule, the bot will open the scheduled activity page and using the Object Cloning command to extract the HTML properties in order to identify which record to open.

Overall this set of actions empowers the business partners to easily manage their bot schedules via triggers and bypass the need to learn how to use the control or submit a request to an admin to perform these actions. It saves time for both the business and IT which is a win for the organization. It also has opened new opportunities to build bots that need to be triggered on demand because there isn’t a pattern or consistency that a schedule will accommodate.

As an airline there are many factors that can interfere with the standard flight operations; one example of this is weather patterns like hurricanes, snow, heavy winds, etc. When Irregular Operations (IROPS) occur within the operation, there are bots that need to run more frequently to keep up with the demand. The business users can also use O.T.I.S. to temporarily overlay a secondary schedule in the control room for the bot to run more frequently.

For example, a bot currently runs at the top of every hour which is perfect for a normal day, but bad weather hits - now the data is behind schedule. A request is sent to O.T.I.S. and a secondary schedule is created based on the data provided from the user. The secondary schedule runs hourly, but 30 minutes past the top of the hour for only the next 12 hours. Now with the combination of the primary schedule and the secondary schedule for the next 12 hours, the bot is performing actions every 30 minutes to help the operation run more efficiently.

Credential, Bot Runner, & Error Handling Management

The Purpose

The administration time taken by automating credential management through password reset notifications, generating new secure passwords, and simultaneously updating the credentials for the bots to consume was taking the CoE away from additional automation development. Second, we wanted to automate the restart of the bot runner machines when deployment errors occurred related to control room connection or networking issues. Finally, we wanted to begin automating certain error handling operations that could provide additional stability for our bots.

The Tech Breakdown

Starting with credential management, login subtasks were created for all the applications bots were using as part of the standard development process. This way if something changed, only one bot would need to be updated for the login function while each bot that used that login function would work correctly without updating them individually.

• If a login subtask fails and the application reports a password has expired, that login subtask error handler triggers O.T.I.S. Upon being triggered, the data containing the application name and a config file is referenced to determine the password requirements for each application.

  • Using the config files enables a simplified method of keeping up to date with new applications or password requirement changes within existing applications.

• Once the pattern for the password is identified, a variable is provided to a script that generates a new password. Now that we have the new credential available, we can use the Automation Anywhere Control Room APIs to update the attribute for bots to consume moving forward.

• With the attribute updated, the application login sub-task is triggered to validate that the new password works successfully. Upon successful completion of the login, the new main bot is scheduled to be run again so that the process does not fall behind schedule.

Now, what about restarting those bot runner machines? We all know when one is unavailable the control room kindly notifies us of each failed deployment. ? To improve the overall uptime and decrease the number of deployment failures, all control room notifications are sent to O.T.I.S.

• If a notification is received of an unsuccessful deployment, the control room will be queried for that bot runner that failed to determine how many unsuccessful deployments have occurred on this bot runner for the day.

  • If this is the first failure, then O.T.I.S. will deploy the bot again using the logic from the earlier example to see if it was just a network issue.

• If the second deployment fails, a script is triggered to restart the bot runner remotely.

  • If the control room stays active, the caching services will automatically log the bot runner back into the control room.

  • After the script is completed, the control room devices API endpoint will be used to validate the bot runner is active and connected.

• If the bot runner was unable to connect, then a notification will be sent for the admin to manually connect to the bot runner.

What does the future look like for O.T.I.S.?

Here are a few ideas for what the future of O.T.I.S. might include:

• Integration with Microsoft Teams so users can trigger bots via Teams messages.
• Reviewing error handling notifications and logs to remotely run actions to fix bots if the first line of defense fails within the bot that is running.
• Automated CMDB management to display what applications the bots use and to be kept up to date without manual data entry.
• Automated bot deployment using the Bot Life Cycle Management APIs in coordination with our change control system of record.
• Adding all requests into IT Service Management system to be tracked.
• Many more ideas that we will keep a secret for the time being ?

Conclusion

The automation of all these functions is just one example of how Robotic Process Automation developers can take their own advice and automate some of their own repetitive processes within their realm of responsibilities. This also created more bandwidth to automate new processes, helping to address more of our digital transformation initiatives as an organization. Overall our goal as robotic process automation developers is to be advocates to help our business partners do their jobs as efficiently and accurately as possible. In the case of Republic Airways, it is also to get pilots & flight attendants to the correct airports on time, to make sure maintenance items on aircrafts are accurately documented, and to ensure that flights take off on time to get customers to where they need to go throughout the country.