Drupal Monitoring with DRD

arocom GmbH has been operating its own monitoring system with Icinga for years. Although working with Icinga delivers the desired results for the most part, there are certain limitations and disadvantages resulting from its use.

For this reason, the search was on for a supplement or extension to the existing monitoring system. Since arocom GmbH only operates Drupal projects, attention was quickly drawn to an existing module for Drupal 6, 7 and 8, the Drupal Remote Dashboard, or DRD for short.

The main features of DRD could be found out quickly from the documentation, among other things, but the actual possibilities that this module brought with it, e.g. what the possibilities of expansion look like or how well this module could be integrated into the existing monitoring system of arocom GmbH, were initially difficult to assess.

Nevertheless, the signs were promising enough to discuss the possibilities of a Drupal monitoring system with DRD and Icinga as part of a bachelor thesis entitled "Monitoring, self-service and housekeeping with the Drupal Remote Dashboard".

Aims of the bachelor thesis:

• The conception of a cooperation between Icinga and DRD

• Setting up a Drupal 8 installation with DRD:

  • Analysis of the existing features

  • Creating the basis for later connection of live pages

• The development and implementation of a role and authorization system within this Drupal 8 installation

• The implementation of various new functionalities as well as the "contributing" of several of these functionalities.

This article summarizes the results of this work.

The Drupal Remote Dashboard

At its heart, DRD is a monitoring tool, similar to Icinga. The focus is therefore primarily on displaying various status information of the monitored site(s). For the most part, no user-defined information is obtained, but instead the information from the Drupal status report, the monitoring module, if activated, or other sources is read out.

From the area of classic monitoring, DRD currently offers functions such as the collection and display of status information or an overview of installed modules and their versions, according to its documentation, as already mentioned. If required and with the appropriate settings, it is also possible to have DRD update these modules and the site itself.

One of the most interesting functionalities, however, are the so-called "actions". "Actions" are configurable and executable code snippets that can be executed on monitored websites. These actions can be used, for example, to clear the cache of a page or change a user's password. In addition to monitoring, the automation of processes and the control of monitored pages are also elementary functionalities of DRD.

This exchange is made possible by a further module: The DRD Agent. In contrast to DRD, this module is not installed on the monitoring site, but on the monitored site. It contains all the code required on the monitored side, such as the logic of the individual "actions". In the event of a status request or the execution of an action, DRD contacts the DRD agent on the monitored page, which executes the corresponding code required and sends the result back to DRD, where the result can be displayed.

Advantages of DRD

Currently, the biggest disadvantage of using Icinga is its complexity and different nature compared to Drupal. Since arocom GmbH develops exclusively with Drupal, the expertise of the employees also tends to lie in this area. For these reasons, familiarization with Icinga is very time-consuming and writing extensions for this system always falls to the same employees, as only they are familiar with it.

A major advantage of using DRD is the existing Drupal environment, whose rules and conventions the employees are already familiar with and are therefore able to work with DRD more quickly and extend it if necessary.

The way in which the respective information is accessed is also simplified immensely when using DRD. The corresponding logic for the required information can be written using all of drupal's internal functions, which leads to solutions that are easier to understand and less prone to errors.

Basically, it can be said that the real attraction of DRD is its connection to Drupal, as it simplifies all processes from familiarization to use and expansion.

Results

The results of the individual objectives of this work are summarized below.

Cooperation between DRD and Icinga:

The introduction of DRD as part of the existing monitoring system is not intended to replace Icinga; instead, it is more of a supplement and an associated shift in task areas.

Previously, Icinga was responsible for all tasks from scheduling the corresponding requests to sending and executing the request to returning and displaying the response. Escalation based on various status information was also part of Icinga's tasks. The idea is now to let DRD take over the part that represents the sending, execution and return of the request. The following figure shows a conceptual diagram that describes the connection points between DRD and Icinga in more detail.

The information that DRD collects from its registered "hosts", i.e. servers, is sent back to Icinga instead of being evaluated directly in DRD. This information can be evaluated and displayed there.

The escalation routines mentioned above can also include the execution of a follow-up action, here the dashed arrow, which in turn is sent back to DRD.

The actual data exchange between DRD and Icinga should take place via "Drush" commands in the future. Each existing "action" and the retrieval of status information in DRD can be triggered by a "Drush" command. The return of these "Drush" commands on the DRD installation can be configured and read out by Icinga. In this way, all important information can be requested by Icinga, read out and returned in DRD and evaluated in Icinga.

Setting up a Drupal 8 installation with DRD:

Setting up a Drupal 8 installation with DRD did not cause any major problems. Other local Drupal installations on which the DRD agent was installed were mainly used to test the installation itself and the functionalities implemented, which are described in more detail below.

The aim here was not to connect the monitored live sites already, but to lay the foundations so that this connection could be made at a later date.

The results of the analysis have already been discussed in the chapter "Drupal Remote Dashboard".

Development and implementation of a role and authorization system:

Within the planned monitoring system with Drupal 8 and DRD conceptualized in this thesis, a large number of customers, users and websites will be found after the introduction into productive operation.

In order to ensure that the guidelines of the GDPR are complied with, a role and authorization system must be created that only allows access to or modification of data to persons who have the appropriate authorizations. At the same time, the design of this authorization system should always take into account the scalability of existing users and monitored websites, i.e. it should not lose functionality, robustness or usability for different numbers of these users and websites.

After analyzing the current authorizations that DRD allows its users to grant, the points that need to be improved in order to achieve the goals described above can basically be divided into two points.

DRD itself already defines a number of authorizations that currently restrict access to "entity types" such as a "domain", i.e. a page, based on operations such as "View" or "Edit".

There are currently two problems with this approach in DRD: Firstly, these authorizations can only be assigned based on roles. If you imagine a system in which there may be hundreds of users with different authorizations in the future, in the worst case you would have to define a new role for each user, which would lead to major problems both in terms of administration and clarity.

Secondly, these authorizations can only be restricted per "Entity", but not per "Entity Type". A user therefore either has access to all "domains" of a page or none at all. As the pages of different customers with different users are to be monitored in this system in the future, there must be a way to allow users access only to the domains to which they should have access.

The "group" module was ultimately chosen as the solution to this problem. "group" relies on so-called "groups", "entities" in which a number of additional "entities", e.g. also users of a page, can be referenced. Access to the "entities" of a "group" is only permitted to users who are themselves members of the same "group". In addition, there is an additional user and authorization management within "Groups" that only applies to "Entities" within a "Group". By default, however, the module does not offer any functionalities for user-defined entities. The logic that controls access to DRD entities such as cores or domains therefore had to be written by the user. To do this, the existing "AccessHandler" of DRD had to be partially overwritten, as there are otherwise no options for writing your own access logic based on "Groups".

All relevant data in DRD, such as the "domains" themselves, their individual status information or their servers are "entities". The solution for access to these "Entities" per individual "Entity" and per user is therefore that any number of users, as long as they have access to the same "Entities", are assigned to a "Group" together with these "Entities". Within this group, roles can be assigned to determine which authorizations users have for the individual entities. Only the admin of a site can create the corresponding groups. This approach ensures that each user only has access to the entities to which they should have access.

The authorizations for "Actions" have a similar problem to the authorizations for "Entities": Here too, the authorizations to execute a single "Action" can only be defined per role and not per user, which repeatedly leads to the problems already mentioned above if there are a large number of users.

The solution to this problem is based on the fact that users can be assigned multiple roles within Drupal. The authorizations of the individual roles that a user receives are added together and assigned to the corresponding user. If you create a new role for each existing action with a corresponding authorization for this action, each user can be assigned this role and thus the authorizations for individual actions in addition to their existing roles and authorizations.

The "group" module and the additional roles for the individual "actions" can now ensure that the GDPR requirements are met within the DRD installation.

Implementing various new functionalities and contributing several of these functionalities:

As part of this work, a number of custom functionalities were written and implemented, some of which were also part of a "contribution". A few selected functionalities are described below.

Generic command line commands:

At the beginning of the work, a collection of ideas with useful tasks to be implemented was created. When analyzing these available tasks, it became apparent that the solution to several tasks was based on "Drush" commands. So instead of adding a separate "action" for each individual request that only executes a "Drush" command, an option was implemented that allows a user to store any command line commands and execute them later. However, at the time of writing this article, this selection is limited to "Drush" and "Drupal Console". The following figure shows the form for creating such a command line command:

For this purpose, DRD forms for adding, editing and deleting such command line commands as well as a new "Action" have been added. When this "Action" is executed, a list of all existing command line commands is displayed, from which one can be selected for subsequent execution.

This functionality is part of a "Contribution". You can find the corresponding documentation here: https://www.drupal.org/project/drd/issues/2930229.

"Action" for searching for users:

Several tasks that were collected as part of the collection of ideas for this work require a user ID in order to be implemented reliably. However, the ID of a user can only be viewed by logging in with the appropriate authorizations on the monitored page.

To save time, as a basis for future implementations and as a general basis for obtaining information about the users of a page, a new "Action" has been implemented that allows a user to perform a search for users of the monitored page.

The current form of this action is shown in the following image:

The individual fields of the form represent the search parameters of a database query on the monitored page. The result is then displayed within the DRD installation.

This functionality is part of a "Contribution". The corresponding documentation can be found here: https://www.drupal.org/project/drd/issues/2976230.

Large Assets Monitoring:

"Assets" represent the files used on a page in the form of images, videos, audio files or other elements.

In order to get a better overview of the "assets" used and their used storage space, a new "action" has been implemented that allows a user to view and monitor the largest "assets" of a page in terms of used storage space.

The logic of this "action" is also based on various parameters that can be specified when selecting this "action". A database query is then performed again and the result is displayed in DRD. The following illustration shows an example of the result of such a query:

Here too, the focus is on laying the foundations. An extension of this functionality could, for example, offer the option of compressing files that are too large instead of just listing them as is currently the case.

The documentation for this work contains details of other implementations that have been realized, but these are not presented in detail here.

Conclusion

Even though a number of functionalities have already been realized and implemented, the system is still several working weeks away from a functioning productive operation. Especially with regard to the already existing Icinga, the current added value of the created monitoring system is therefore very clear.

However, the aim of this work was not to create a functional system with added value compared to the initial situation. Instead, the main goal is to create the foundations, i.e. to define and implement a robust basis on which to build further.

Much more important than a monitoring system that is already in productive operation are the foundations and experience that have been created and gathered and on which further functionalities can be implemented, which ultimately justify putting this system into productive operation.

In principle, however, it can be said that the necessary foundations have been laid in the course of this work. The role and authorization system plays a major role in this, as it ensures the general use of the system without violating the requirements of the GDPR. The concept for cooperation between Icinga and DRD is also already in place. Other implemented basics that were not covered in this article are forms and modules that enable the storage of variables and data and offer a simplified way to add new status information. Functionalities such as the generic command line commands also offer enormous added value, as this approach can be used to tackle and solve a whole range of problems.

In conclusion, it can be said that although the system as a whole is not yet ready for live operation, the individual components required are available.

The future viability of this monitoring system therefore stands and falls with the robustness and reliability of the individual components implemented as part of this work. After all, functionalities with real benefits and added value can only be successfully implemented, improved or expanded within a system that is characterized by these attributes.

Are you interested?

If you are still interested in this bachelor thesis and a Drupal monitoring system after reading this article, we invite you to read the entire bachelor thesis for yourself. If you have any further interest, technical details or other questions about this bachelor thesis, please feel free to contact info@arocom.de.