By Dirck Lowe
If you are implementing or upgrading your Energy Trading & Risk Management (ETRM) system, now is the time to integrate DevOps automation. This approach is essential for both new implementations and those in a steady run and maintain state with their ETRM software. By adopting DevOps methodologies, including automation and standardization in development, configuration, and testing, operational teams can significantly enhance ETRM solutions.
Many ETRM system implementations still follow a traditional, monolithic waterfall approach. However, incorporating agile methodologies through DevOps can streamline processes, reduce implementation time, lower operational costs, and prepare for faster ETRM system upgrades. Are you ready to transform your ETRM landscape with DevOps expertise for quicker, more efficient upgrades?
In this article, we’ll explore the overarching benefits of a DevOps approach in ETRM, setting the stage for deeper dives into continuous integration/continuous deployment (CI/CD), release strategies, and more in upcoming articles.
Driving Value
Several factors drive the value of implementing a DevOps model, but the following four emerge as the most potent catalysts for impact.
Value to the Business Quickly 
DevOps is built for rapid development and more frequent releases, both custom solutions and core ETRM releases. Building development, configuration and automated testing shortens all upgrade cycles, especially testing. This model allows the business to react and adapt quickly to evolving markets and new business lines. Regulatory changes also drive change, for example, the periodic changing of Dodd-Frank requirements. Being able to quickly take advantage of new vendor functionality can keep a competitive advantage and allows the business to perform what they do best. In addition, you’ll stay on top of implementing vendor maintenance provided upgrades and enhancements for new functionality.
Lowering Operational Costs 
Engineering and application configuration standards provide faster identification and resolution of issues. One of the first approaches we recommend taking is building engineering standards for a common logging framework. ETRM systems generally provide APIs for this framework out of the box, or you may choose to add wrappers for additional telemetry. Common and consistent logging allows operational teams to trace the “deal lifecycle” and determine failure points, which is especially important when executing end-of-day processes. Development teams can leverage base development classes and libraries, allowing for quick deployments with confidence. This can also be a stepping stone into additional standards for external monitoring tools in the market.
Team Excitement 
Smaller agile teams working together provide stronger collaboration, which ultimately leads to success. An often-overlooked factor with DevOps is the pride that individuals take in their work, knowing very well, in certain scenarios, they could be supporting the solution longer term. Injecting a SRE (Site Reliability Engineer) into the team is a component many agile teams are starting to explore. The SRE can bridge the gap between the ETRM developers and the team responsible for the infrastructure. While not ideal, sometimes IT Infrastructure teams are distant from the project, and having SRE can provide the much-needed glue.
Faster and Cost-Efficient Upgrades
Gone are the multi-month (or even multi-year) upgrades. Doing more timely upgrades allows companies to stay in sync with vendor version and support timelines and ever-changing regulatory requirements. For many, establishing a monthly cadence of maintenance releases allows steps to be fine-tuned for a smoother major release or upgrade. Because DevOps focuses on leveraging engineering standards, documented NFRs (Non-Functional Requirements) and automated testing environment creation it further reduces both hardware and human time. The automation of repetitive tasks and tests limits the risk of human errors which leads us to……automation.
Automate, Automate, Automate!
Automation is at the heart of DevOps, allowing for manual and repetitive tasks to be carried out without human interaction. Everyone wins with automation, from the business being able to receive features and fixes faster to the operational teams spending less time tracking down the root cause of issues. The risk of human error is almost non-existent. This is an area, if just starting out with an implementation or upgrade, may require additional investment up front but the ROI should soon be realized. Automation plays an area in all aspects of DevOps including:
- Automation of Environments
- Automation of Code and Config components
- Automation of Testing
All three have significant benefits on the project to reduce implementation/upgrade time, but the largest benefit is post go live. The tools, standards and automation built during the project can be leveraged by operational and business teams to minimize risk and reduce maintenance costs.
Automation of Environments 
DevOps and the cloud go together. The beauty of leveraging the cloud, especially if already integrated into your landscape, is the tooling is likely in place and used in other areas of the organization. In addition, specialized ETRM expertise is generally not required for the underlying infrastructure. With the cloud, tooling can provide automation of new databases servers and schemas, scaling of application and servers and development/testing environments.
There is exponential benefit for self-service of this automation model as well. Think about a developer or business analyst working on a new feature and requires a “clean” environment. At the click of a couple of buttons through a simple GUI, they could spin up a new environment, have the latest core and custom code, defined datasets, test scripts and away they go! Architected with efficiency in mind, this can take minutes. Larger datasets may take longer to spin but having options (with ability to define the dataset imports) gives developers and functional resources control over their environments.
Automation of Code and Config Components
DevOps is built for code automation and our clients leveraging DevOps follow industry standard practices such as Sonar or other code check-in/validation utilities. However, it’s important to look at implementation holistically and not only a narrow view of development/code automation. Data and configuration are just as an integral part of the methodology, historically steered away from formal check-in/out procedures. ETRM Systems allow for static and reference data import and export using inherent and supported functions and APIs for moving configuration and data between environments. Adding version control to config files and data sets strengthens the quality of the implementation and limits manual errors of data and config entry into clean or empty databases, eventually becoming Production.
For new implementations, building configuration and reference data is part of any project. While some data is static enough not to change during the project, it’s still best practice to treat it as code and use version control. Typical examples are PZL (Pipeline, Zones, Locations), country codes, currencies, etc. Other data is more dynamic such as curves and counterparties. Following these standards allows for version control, especially useful for longer running projects where business counterparties change periodically during implementation, allowing for an export and import with a version number.
Proper hygiene and management are important for configuration management, just like development practices require. Gone are the days of managing “Gold” or “To-be” databases with Excel spreadsheets and asking functional users to track their changes. All config and code should follow standard development practices and use a code/config repository such as Visual SourceSafe or Git as an example (there are many repository/code management tools).
Automation of Testing
Automated testing warrants an entire article or white paper, but faster upgrades can’t be done without this critical component in place. Testing still follows standards such as integration, end-to-end, performance, UAT, etc. but much of the testing thoughts shift early in the build phase. As with typical DevOps tasks, each User Story or PBI should have some/all components of the following: Requirements, Acceptance Criteria, NFR’s, testing automation scripts/criteria. Building automated tests, associated to a User Story will eventually grow to an entire suite of manageable automated tests that can be executed during testing cycles, prior and post go-live. Ultimately, the flexibility of environment and config with DevOps will provide defined provisioned environments and artifacts for testing to be executed consistently every time. We’ll provide additional detail in the future regarding some of our automated testing best practices as well as standards followed in the industry.
Challenges
A DevOps model should produce positive benefits when the functional and technical teams collaborate with the single goal in mind. However, with any new ways of working, there are challenges that need to be addressed prior to any engagement.
Starting Out
Where to begin can sometimes cause “writer’s block” and derail a DevOps approach even before it starts. Change is difficult and it is easy to fall back to antiquated approaches. A simple approach is to start off like any ETRM project and have a general view/plan in place – can even start with a waterfall approach using the typical Analysis, Design, Build, Test, Prod approach. In many cases, starting with this approach and breaking down the known requirements will get teams thinking about sprints and work chunks. One of the most difficult elements is trying to break down work into manageable chunks for business analysts and developers on a sprint-to-sprint basis. It takes some initial time to adjust but once you’ve gone through a couple of sprints, finding logical work breaking points gets easier.
Initial learning Curve
DevOps is industry standard development practice but upskilling ETRM knowledgeable resources to take advantage of tools is important (the tooling these days makes for a fast-learning curve). Functional teams typically have a longer learning curve depending on how DevOps leveraging in the project. New steps for a functional resource may require exporting configuration in logical components or breaking down functional requirements into manageable and testable chunks. This takes practice. ETRM developers used to scripting solutions using ETRM provided API’s will need to shift to a development mindset using common libraries and classes for logging, telemetry requirements, etc.
Cultural Shift
This is one of the toughest challenges to mitigate and requires buy-in from all levels of the organization. Out of the gate, it’s important to establish the DevOps vision from the top down and continue to evangelize the model well into the implementation. This may be easier if an organization follows DevOps in other areas outside of ETRM. But there is no hiding that historically many ETRM implementations follow practices built before DevOps became the norm in the development world. One tip to help aid adoption is having project resources in leadership roles who ultimately will be responsible for supporting the system after it goes live. These resources become your strongest evangelists, and rightfully so.
Recruiting the Team
You may have ETRM knowledge in-house or leverage a knowledgeable third party, such as KWA, but at the end of the day, someone on the business and technical side of the house will own the solution. The simplest way to mitigate operational risk is to have institutional employees engage at all project levels. As noted above, using an SRE is one option and can be rewarding for individuals wanting to grow their ETRM career and hungry to learn more. Putting them in this role (with proper guidance and mentorship) will allow them to implement the ETRM system and then be responsible for critical support elements.
Conclusion
In conclusion, integrating DevOps automation into ETRM systems brings significant advantages. Organizations, however, must be aware of the inherent challenges. Investing in thorough training, fostering a culture aligned with DevOps principles, and securing top-level buy-in for a successful ETRM implementation is essential. The ultimate goal is to find the perfect balance between rapid deployment and sustainable solutions in the ETRM landscape. So, let’s reiterate the question – who doesn’t want quicker, more efficient ETRM system upgrades?
