The Responsive Automation Center (RAC) is a revolutionary way of providing support by leveraging digital technologies such as artificial intelligence (AI), machine learning (ML), and robotic process automation (RPA) to prevent and quickly recover from incidents and automate service request provisioning. Although the backbone of the RAC is advanced automation, somebody has to engineer the automation and deal with work that is not fully automated.
Meet the solutionists (sometimes called solution engineers). The solutionists are the people that make RAC automation work. In this article, we expand upon previous discussions of the RAC by focusing on the people domain, namely, the RAC solutionists – who they are, what they do on a daily basis, and what skills they require. We also explain why having fewer but more dynamic staff adds more value than having a larger but one-dimensional single point of contact Service Desk.
The Invisible Hand of the Responsive Automation Center
When the RAC is accomplishing its mission, users are unaware of its existence. In turn, RAC solutions engineers, far from being “ghosts in the machine,” actively work with digital technology – albeit, behind the scenes – to propel automation.
RAC Core Principle: Availability is king!
Who are Solutions Engineers and What Do They Do?
If automation is the central nervous system of the RAC – taking in environmental inputs, making sense of them, and responding to them – Solutions Engineers are its heart, ensuring that the RAC is meeting the goals of achieving high availability, automating solutions, preventing incidents, and minimizing human intervention. Solutions Engineers have five essential (and often overlapping) functions:
Propel Automation
In legacy support operating models, support starts when a customer reports an incident or submits a service request. In the RAC operating model, creating automated solutions is the first step. In other words, RAC solutions engineers proactively fine-tune monitoring and event management systems to prevent incidents (more on this below) and create automated workflows for all service requests that are able to be fulfilled with no human intervention.
Along these lines, RAC solutions engineers work to continually prune and fine-tune the service catalog. They create self-help tools like service catalog items and corresponding workflows. In this way, they support a dynamic service catalog where standard or “static” service offerings are supplemented with new combinations based on customer demand.
In addition to creating automated solutions, solutionists configure self-help wizards and knowledge articles to promote user self-service.
Resolve Complex Incidents
The quality of conformance to the RAC operating model is to proactively prevent 100% of incidents. Thus, human response to incidents is rare. Nevertheless, in a volatile world, it is not always possible to predict failure when complex multi-integrated systems are involved. When incidents do occur, they are complex. In these cases – more the exception than the rule – RAC solutions engineers perform reactive Incident Management to restore service quickly. The solutions engineers have the ability within the team to resolve virtually all these incidents without resorting to tiered escalations.
Rather than put in place a temporary fix or workaround, once the incident is resolved, solutionists quickly pivot from a reactive stance to a proactive approach to prevent the incident from recurring.
Prevent Recurrence of Incidents
The solutions engineer’s mantra is “A human should never touch an incident twice.” Whenever an incident occurs, RAC solutions engineers work to prevent its recurrence. Similar to what is often considered Problem Management, this involves technical skills, complexity thinking, and creativity.
RAC Core Principle: A human should never touch an incident twice
After diagnosing and understanding the root causes, solutionists achieve prevention through a combination of operating model activities such as engineering automated workflows, calibrating monitoring and event management tools and cloud-based monitoring and AI “training,” and coordinating with platform engineers, DevOps/DevXOps teams, and other solutionists.
Orchestrate Solutionists
Although RAC solutions engineers have significant technical and business prowess, they sometimes need to work with other solutionists such as platform engineers and DevOps teams to automate workflows, resolve complex incidents, and prevent incident recurrence. In this way, RAC solutionists play a key role in coordinating solutions from multiple teams and working across silos.
However, it is once again worth mentioning that the RAC does not engage in tiered escalations. Rather, interactions with other teams are coordinated. For rare incidents and urgent needs that cannot be automated or resolved by the RAC, a swarming approach with other teams may be used.
Optimize Service Awareness
RAC solutions engineers ensure that the health of services and associated components is continuously monitored. As such, they deploy and calibrate AI-enabled monitoring and event management tools and monitor activity and health of cloud-based solutions. Regarding the latter, although most of this is already being performed by vendors, RAC interfaces with cloud vendors when necessary.
Service Awareness means understanding the end-to-end health of services and their associated technical and nontechnical components at any given point in time. It also means knowing, in near real time, when the health of the service or the status of an underlying component has changed.
The overall goal is to be aware of service health at any point in time and optimize or perfect the ability to monitor health and prevent service degradation at the early symptom stage.
What are the Characteristics/Traits of the RAC Solutionist?
Being a RAC solutionist is not just about what you know and what technical skills you possess. It takes a special mindset and approach to work. Solutionists are to the core problem solvers. They thrive on complexity and are energized by solving the problems not solved. They abhor manual and repetitive work (i.e., toil) and recoil at the thought of putting in place temporary workarounds. Solutionists see new technology and ways of working not as things to cope with or get used to; but rather as something to embrace. The following traits characterize the ideal solutionist:
- Curious – They are interested is how things work and why something happens.
- Adaptable – They are flexible, amenable to changing how they work and to discontinuing processes that are no longer successful.
- Impatient – They are driven to resolve issues quickly.
- Technically handy – They are well-versed in technical matters and capable of learning new technology quickly.
- Analytical – They are able to quickly identify trends and routinely employ complexity thinking.
- Intrinsically motivated – Although reasonable pay, rewards, and recognition are still important, they are primarily motivated by solving problems.
In addition to these intrinsic characteristics, solutionists need to have the right skills. Far from being “jacks of all trades,” solutionists are deep subject matter experts in several areas. They are what is sometimes known as “M-Shaped” or “Comb Shaped” people.
M-Shaped (Comb-Shaped) People
The concept of “M-shaped people” has evolved from the earlier ideas of “T-shaped” and “Pi-shaped” people. “T-shaped” workers have deep subject matter expertise in one area and general knowledge in other areas. “Pi-shaped” workers have two areas of deep expertise as well as some complementary skills or knowledge in other areas.
However, in today’s rapidly changing economy and job market, it’s becoming more common for workers to develop skills and knowledge in multiple areas, making them “M-shaped” (sometimes called “Comb Shaped”). RAC engineers need to be “Comb Shaped” by possessing deep expertise in more than two domains, having complementary skills in adjacent domains, and general knowledge in several other areas, allowing them to be versatile and adaptable across a range of roles and industries.
The ability to learn quickly and develop new skills on the job is becoming increasingly important, as the pace of technological change and innovation continues to accelerate.
At first glance, RAC solutionists appear to be the like proverbial unicorns – nice in theory, but impossible to find. The reality is that as technology has become more democratized, it is easier than in the past to find workers with deep technical skills, and the issues the RAC deals with involve a systems thinking approach.
What Skills do RAC Solutionists Need?
Considering how central automation is to accomplishing RAC goals, it comes as no surprise that RAC solutionists need to have expert understanding and practice experience with several technologies. Beyond technology, Solutions Engineers need to be savvy with high velocity techniques and approaches to work and proficient at working with other technical experts and business customers. Let us discuss each in more detail.
Technology Skills
RAC solutionists need to have more than just technical know-how – they need to be expert in the ITSM platform, multiple automation tools, and digital technologies.
At a minimum, they should be expert in using a “single pane of glass” ITSM platform where incidents, requests, and changes are logged. Although it is unlikely that solutionists are platform engineers, knowledge of platform engineering and configuration for their organization’s ITSM tool is ideal.
The ITSM platform is also where the service catalog is housed, and solutionists must be able to update the service catalog with new offerings and new automated workflows. In conjunction with the ITSM platform and service catalog, solutionists must leverage robotic process automation (RPA). RPA enhances the service catalog by improving both efficiency and quality of service. For example, when an employee submits a request for a software installation, an RPA bot can be used to automatically install the software on the employee’s computer without the need for manual intervention from IT staff. This not only speeds up the request fulfillment process but also frees up RAC solutionists to focus on more complex tasks. RPA also plays a role in maintaining the Service Catalog by automatically updating it with new services and removing outdated services.
Beyond the ITSM platform, solutionists are expert in using and fine-tuning monitoring and event management tools to improve availability, reduce incidents, and achieve service awareness. The best in class monitoring and event management tools leverage artificial intelligence (AI) to track and analyze trends, proactively prevent incidents from occurring, and recover with lightning speed (within seconds or fractions of seconds) when failure does occur. Ideally, an organization would deploy one AI-enabled tool to monitor all infrastructure and systems. In reality, some industries rely on specialized systems and will still need to have multiple monitoring tools. Solutionists must be expert in both the specialized systems and the associated monitoring tools.
High Velocity Techniques and Approaches to Work
Maintaining high availability (near 100%) and low mean time to resolution (MTTR) are critical to RAC success. Likewise, when issues are encountered and new solutions identified, they must be built and deployed without delay. RAC solutionists need to have an advanced understanding of process design and associated techniques when creating new automated service request workflows.
While not a high velocity technique, solutionists also need to have deep analytical skills and the ability to understand data and trends. This is similar to problem management, which has a goal of preventing future recurrence of incidents. However, the focus is largely on proactive problem management (even beyond that) – that is, predicting and preventing failure before it happens. To be sure, AI-enabled monitoring and event management tools perform much of this work, but there are still cases where solutionists need to leverage their own judgement to supplement AI.
Although RAC solutionists themselves may not develop every solution, they need to understand the basics of agile product management and DevOps so that they are familiar with the work approaches used by other teams with whom they frequently collaborate. Additionally, they should have facility with site reliability engineering (SRE) techniques which focus on creating a resilient technology ecosystem that is able to quickly recover from failure.
Orchestration and Collaboration across Teams and with Vendors
In many cases, RAC solutionists operate independently to create automation solutions. However, some degree of collaboration with other, more technically advanced teams, is inevitable. RAC solutionists work from time to time across business units and with technical teams such as SRE, DevOps, and Platform Administrators. They sometimes play the role of business analyst and translate customer requirements to these teams or advocate for increased prioritization of software features or new products based on their understanding of user and customer demand.
In a RAC ecosystem, the Cloud is heavily leveraged for both software and infrastructure. RAC solutionists must have facility with cloud-based tools and working with Cloud Service Providers.
It is important, once again, to point out that although RAC solutionists play some role in orchestration and collaboration, we are not talking about serving as a single point of contact and we are not talking about tiered escalations. (We know – it will take a while to purge those concepts from what we consider to be best practice!)
The RAC Comb
Expanding upon the metaphor or the “Comb Shaped” person, we can create a “comb” tailored for the ideal RAC Solutions Engineer per the graphic below:
Why Less is so Much More
There is no set number regarding how many solutions engineers should staff the RAC. Of course, the precise number varies depending on the size, scope, and needs of your organization. However, it reasons that there should be considerably fewer solutions engineers than, for example, agents in a single point of contact (SPOC) Service Desk model. A good estimate is a 1:4 ratio. That is, for every four Server Desk agents in a SPOC model, there is just one solutions engineer in a RAC model.
RAC solutions engineers have a wider set of skills and deeper technical skills than support staff in years past. As previously suggested, greater skills, are harder to find, and those who possess them command greater salary. Even so, having fewer but more dynamic staff are better than larger but less skilled teams. Consider this: the average Service Desk Salary in the United States is approximately $40,000/year. Thus, an organization that requires a Service Desk staff of sixteen spends somewhere on the order of $640,000 in salaries alone. In a RAC support model, there are likely to be just four solutions engineers, though the average salary is 125% higher at $90,000/year. Even so, the total annual salary expense is $360,000 or about 44% less. What’s more is that the RAC model has achieved high service availability, prevented incidents, and dramatically improved the speed and quality of resolutions, all of which improves employee engagement and customer satisfaction.