At Pagero, we are spending lots of resources on improving tests, effectiveness of development and architectural changes to secure a modern and scalable platform for our customers. Even though this might seem at first glance to be an area mostly benefiting the developers of Pagero Online, there are also huge benefits for our customers. In this article, our CTO Bård Langöy tells you why.
Improving the support for business processes
Pagero Online is a mature SaaS platform that offers customers a great selection of services and a huge network reach and that has become one of the major providers of O2C and P2P services. We are constantly working on improving and extending the support for business processes and ensuring that our customers are compliant with laws and regulations in all countries where they operate. These are aspects of Pagero Online that our customers experience every day.
But Pagero Online is also a very technically focused product, and we are spending lots of resources on improving tests, effectiveness of development and architectural changes to secure a modern and scalable platform. Even though this might seem at first glance to be an area mostly benefiting the developers of Pagero Online, there are also huge benefits for our customers. This text will tell you why.
Great coverage on tests
Providing our customers with a high-quality service has been one of the most important tasks since Pagero Online was first developed. In order to build our users’ trust and security, we secure all services in Pagero Online with several different types of automated tests. Over time, as the need for manual verification is limited, these tests have also provided us with much faster delivery of new features.
“Pagero Online is a mature platform that has become one of the major providers of O2C and P2P services.”
Every service in Pagero Online is always verified through automated tests before it is approved for release. These tests cover everything from the service’s smallest parts to how the service works as a whole when receiving requests from other services or clients. In addition, every service that plays a part in the delivery of a business document (invoice, order, etc.) is also tested in larger feature tests in which many business documents are sent through the system, making sure that the handling of the document is consistent with the expected behaviour. The feature tests cover everything from the most common scenarios to less common corner cases.
Small quick steps
Even though we have secured our services with automated tests, we also want to introduce new features, or changes to the platform, in small steps so that the individual changes are not complex and thereby are easier to verify.
Smaller changes also mean faster delivery of new features to our customers. Customers will learn a new feature faster because it is easier for them to learn a smaller number of new features gradually instead of all at once. In addition, this fast delivery of small features to our customers gives us faster feedback from them on changes, improvements and new features needed.
“Providing our customers with a high-quality service has always been one of the most important tasks.”
We achieve this with Continuous Delivery, which means that instead of trying to release sets of changes every couple of weeks, we aim to release several times a day. New features are broken down into smaller working and functional parts and delivered if they pass all the tests.
One way of achieving the small quick steps discussed in the previous section is by splitting the platform into smaller services and not running everything as one large service. This means the area of responsibility for each service is smaller, which makes it easier to build and maintain it, thus reducing the time needed to deliver new features.
Introducing these small services also has several other benefits for our customers:
The platform does not need to be updated all at once. The individual services can be updated one at a time, reducing the need for scheduled downtimes.
Better runtime scalability allows the software system to grow with customers, providing a secure investment for companies.
It is possible to scale the number of services based on load, guaranteeing the best performance at all times.
This approach to software development is called „Microservices Architecture“ and can be described as a particular way of designing software systems as a collection of independently running services. New services can be introduced and existing services can be updated without affecting the currently running services. Services are developed by teams highly knowledgeable about the specific area the service relates to.
Automated Configuration Management
With several thousand customers on our platform and over a decade of experience running online services, we understand what is needed to provide our customers with a platform where they can feel secure. We want to have full control of all services and components running in our production environment, making it easier to stay updated with the latest security updates. In addition, we need to help bring about an environment where it is easy to audit changes in environments from a security and recovery perspective.
We achieve this by removing all the manual configuration and setup on our servers and machines. We treat infrastructure as code, where everything from a developer’s laptop to servers operating in the production environment is defined and set up using standardised files and tools to ensure that the setup is identical for all machines of the same type. This has a great number of benefits for developers, as well as being the basis for working with Continuous Delivery (described above).
“Services are developed by teams highly knowledgeable about the specific area the service relates to.”
It also provides our customers with a higher quality of services, since identical test and production environments make it easier to ensure that features are tested with the same conditions as in production.
Monitoring and statistics
One of the benefits to customers subscribing to our online services is that they do not have to maintain and monitor the services themselves. We want to provide our customers with a service where we constantly pinpoint areas for improvement by looking at trends and provide fast recovery if problems occur. In addition, we analyse features usage and improve features to better fit the way our customers actually use them.
Pagero Online runs with a modern, sophisticated monitoring framework. The monitoring framework handles everything from server hardware to heartbeats within particular services that show that they are alive. Even though every service in Pagero Online runs on at least two different machines, in order to secure maximum uptime, it is still important to detect and correct failure as soon as possible. Also, by viewing the monitoring statistics in a historical context it is possible to detect behavioural changes that might be the result of a security breach or incorrect logic within a service. In other words, the monitoring statistics play an invaluable part not only in planning future software and hardware resources but also in error detection.
The Pagero Online platform operates in over 50 countries all over the world and is reaching new countries ever more rapidly. This gives our customers huge growth potential in their electronic document delivery.
At the moment, we operate in countries where Friday and Saturday are considered the weekend instead of the more common Saturday and Sunday. Taken in combination with the different time zones, this puts great demand on the continuous availability of Pagero Online services.
It quickly became apparent to us that we had to either move our service windows to more obscure times or aim a bit higher and try to remove the service windows altogether. Since last year we have been implementing a zero-downtime policy which affects the way our developers work. The policy covers the following areas:
How services are designed
How service interfaces are updated
How services and databases are updated
How services are deployed to production
How many services and database instances must run as a minimum
How many data centres we have
Although notifications of service windows during larger monthly releases are still sent to our customers, Pagero Online now remains up and serving our customers during these updates.
“We have implemented a zero-downtime policy which affects the way our developers work.”
This policy gives our customers one great benefit: a platform that is always up and running.
When we develop new features in Pagero Online, we want to ensure that we build the features our customers want as well as prevent unwanted behaviour in our features affecting our customers. We want to be able to update existing or new features and test them out on a few selected customers without affecting the others. We also want alternative versions of a feature to be tested in order to see which version performs best in practical use.
We achieve this with a technique called “feature toggle”. A feature toggle is a way of controlling a certain feature in a service. The feature toggle is either turned on or turned off. The technique allows developers to release hidden features (toggled) so that they do not appear in user interfaces or background tasks. A team w