cloudlinux-os-plus

CloudLinux OS Shared Pro

[[toc]]

X-Ray

Description

:::warning Warning! X-Ray is available out of the box for cPanel, Plesk, and DirectAdmin. For other control panels you should implement integration as described here :::

X-Ray is a tool developed for website performance monitoring and performance issues detection.

X-Ray can gather and visualize information about top N slowest system functions, external requests, software modules and database queries of the client’s website.

Known limitations

X-Ray is not compatible with Opcahce JIT optimization. Once X-Ray tracing task is running for the site the Opcache JIT optimization will be disabled until tracing task completed.

Installation

Make sure you have CloudLinux OS Shared Pro subscription (only non-reseller accounts apply)
Make sure you have installed LVE Manager version 6.2 or later. You can install or update it with the following commands:
- installation
```
# yum install lvemanager
```
- update
```
# yum update lvemanager
```
X-Ray will be activated on all your servers during 4 hours. You will see the X-Ray tab in the LVE Manager UI.
For instant activation, run the following command:
```
# rhn_check
```
If the rhn_check command is not found, run the following command:
```
# yum install rhn-check rhn-setup
```
Then install the alt-php-xray package
- Via user interface
  - Go to the X-Ray tab.
  - Click Install to start installation.
- Via SSH by running the following command:
```
# yum install lvemanager alt-php-xray
```
After installation, use the Start tracing button to create your first tracing task for a slow site.

How to manage X-Ray

X-Ray provides several options for monitoring domain requests speed: Manual Tracing task, X-Ray Autotracing and Continuous tracing.

Manual Tracing task is a task that you can create for a specific URL to collect server requests. The task will end either after a specified number of requests to the URL or after a specified time (maximum after two days). It is not possible here to automatically email a report but it is possible to export the report in PDF and send to a user.
Autotracing task is a task that will be created automatically at the end of each day for slow URLs that were found during that day by the PHP Slow Site Analyzer (SSA). Need to be enabled separately, see How to enable X-Ray Autotracing.
Continuous tracing is a task that initiates daily hourly tracing requests for a specified domain and email a monitoring report. Continuous task can't stop automatically, you need to stop it manually. In fact, continuous task allows to automatically create a tracing task for each new day, with the ability to get a report for the past day.

Tracing tasks tab

The Tracing tasks tab contains a list of all tracing tasks created both manually and automatically via continuous tasks.

The Created column shows how a task was created – automatically (by continuous task) or manually.

Continuous tracing tab

:::warning Warning To use Continuous task, update your LVE Manager and alt-PHP-X-Ray packages to versions lvemanager-6.2.9-1 and alt-php-xray-0.2-1 by running the following command:

yum update lvemanager alt-php-xray

:::

The Continuous tracing tab contains a list of continuous tasks for which tracing tasks will be created automatically for a new day for a specific domain.

Managing tracing task

Creating a new tracing task

Go to the X-Ray tab
Click the Start tracing button to create a new task
In the opened popup specify a website URL to trace
Click the Run button
Tracing will run in the default mode. In the default mode X-Ray traces the first 20 requests for a specified URL

URL should be a valid URL of the domain which exists on the current hosting server. The URL field supports wildcard matching. To learn more about wildcard matching, click How to use special characters.
Advanced settings allow you to set an IP address and tracing options: by time or by number of queries.

Advanced settings

Client’s IP: it is an IPv4 address of a machine to trace. For example, if you have a production website that processes requests from different IP addresses and you do not want to add these requests to the tracing task. So, you can set a specific IP address and X-Ray will analyze requests only from this specific IP address. Record for
Time period: how much time X-Ray collect the requests (2 days max)
Requests: the amount of requests that X-Ray will collect

After creating, the task appears in the list of tracing tasks.

Viewing tracing tasks list

Tasks created Manually are simply tracing tasks.

Tracing status

A tracing task can have the following statuses:

Running – tracing is in progress
Stopped – tracing was stopped by administrator
On hold – the same URL already exists in the lists. Task processing will not start automatically. Administrator should start it manually.
Completed – period of time is finished or number of requests is reached.

Collected requests for tracing task

:::warning Warning! Collected requests are available in the UI for two weeks. :::

Click to open a list of collected requests.

Tracing tasks

The slowest request is highlighted.

Total displays how many requests were collected according to tasks requirements.
Pending displays how many of collected requests are not visible in the table yet.
Throttled displays the number of requests during the execution of which the LVE limits were exceeded.
Slow displays the number of requests lasting more than one second.

There are filters for the request types and the indicator of a filter used now.

If slow requests were not detected during the tracing task, the following is displayed. Here, you can also view all requests.

X-Ray collects the following data for each request:

Top issues – the slowest items of a request
Software modules/plugins by execution time (only for WordPress plugins)
Database queries by execution time
External requests by execution time
Other system functions by execution time

Software modules/plugins

The Software modules/plugins section displays the following data:

Software type – a type a module/plugin. For now, X-Ray can analyze only WordPress software
Software module – a name of the WordPress plugin
Duration – plugin execution time
Duration (%) – plugin execution time as a percentage of the total duration of the request

Database queries

The Database queries section displays the following data:

Query – the executed SQL-query
File – the file and the line of the executed query and backtrace
Software module – a WordPress plugin name from which the request was completed. If the request does not belong to any of the WordPress plugin, the name of the function that executed the given request is displayed
Calls – the number of identical SQL queries
Duration – execution time as a percentage of the total duration of a request and the function processing time (in brackets)

External requests

The External requests section displays the following data:

URL – the URL of the executed request
File – the file and the line of the executed request and backtrace
Duration – execution time as a percentage of the total duration of a request and the function processing time (in brackets)

System functions

The System functions section displays the following data:

Function – the executed function
File – the file and the line of the executed request
Duration – execution time as a percentage of the total duration of a request and the function processing time (in brackets)

Stopping tracing task

Click to stop the tracing task.

The tracing task status will be changed to Stopped. Data will not be collected anymore but you can see already collected information or continue tracing later by clicking .

Deleting tracing task

Click to delete the tracing task.

:::warning Warning! When you have deleted a tracing task, all collected data will be unavailable. :::

Managing continuous tasks

Creating a new continuous task

Click the Create continuous tracing button

Specify URL in the Domain field and email in the Email for reports field and click the Create button.

You can see a new task in the Continuous tracing tab in the X-Ray UI.

If you stop a continuous tracing task, a new task for the next 24 hours will not be created. The task for the current day will be finished at midnight and the report will be emailed.
If you delete a continuous tracing task, the task for the current day will be finished at midnight and the report will be emailed.

Viewing continuous tasks list

You can find a list of continuous tracing tasks in the Continuous tracing tab.

You can find automatically created tasks in the Tracing tasks tab marked as Automatically in the Created column.

The statuses for automatically created tasks are the same as for tracing task.

To view detailed info about an automatically created task, click . You will get requests grouped by hour.

Click to a group to open a list of the requests.

The following data is collected for each request:

Software modules/plugins by execution time (only for WordPress plugins)
Database queries by execution time
External requests by execution time
Other system functions by execution time

Stopping automatic tracing task

Stopping automatic tracing task (a part of continuous tracing task) affects only the automatic tracing task for the current day. A new task for the next day will be created at the end of the day.

To stop the continuous tracing task completely, see Creating a new continuous task, paragraph 4.

Deleting automatic tracing task

Deleting automatic tracing task (a part of continuous tracing task) affects only the automatic tracing task for the current day. A new task for the next day will be created at the end of the day.

To delete the continuous tracing task completely, see Creating a new continuous task, paragraph 5.

Continuous task daily report

Users get daily reports on their emails. An example of a report is shown below:
Click the link in the email to show the detailed report:
You can view requests grouped by hour:
You can also view the detailed information about request:

X-Ray Autotracing

X-Ray Autotracing automatically creates tracing tasks for slow URLs that were found during a day by the PHP Slow Site Analyzer (SSA).

:::warning Warning To use X-Ray Autotracing, update your alt-php-ssa and alt-php-xray packages to versions alt-php-ssa-0.2-1 and alt-php-xray-0.4-1 or higher by running the following command:

# yum update alt-php-ssa alt-php-xray --enablerepo=cloudlinux-updates-testing

:::

How to enable X-Ray Autotracing

To enable X-Ray Autotracing, run the following commands via SSH:

# /usr/sbin/cloudlinux-ssa-manager enable-ssa
# /usr/sbin/cloudlinux-autotracing enable --all

Check CLI documentation for a description of the /usr/sbin/cloudlinux-autotracing CLI utility.

Requirements

CloudLinux OS Shared Pro or CloudLinux OS Solo or CloudLinux OS Admin
alt-php-ssa > 0.2-1 version
alt-php-xray > 0.4-1 version
Enabled PHP SSA on the server

Autotracing Interface

A new tab for Autotracing tasks was added to the X-Ray UI:

Autotracing FAQ

Q: Why are the slow URLs in the Slow Site Analyzer report different from those on which the autotracing tasks were created?

A: Because the autotracing decision module uses rules and thresholds different from Slow Site Analyzer, which are configured by the CloudLinux team.

Q: How often autotracing tasks will be generated?

A: Once a day at the same time as a Slow Site Analyzer report.

X-Ray Smart Advice

Smart advice is a new feature of X-Ray that is designed to find and propose solutions to fix performance issues and speed up the performance of a sites.

:::tip Note Smart Advice will work only for CloudLinux OS Shared Pro servers with cPanel Control Panel for now.

At the moment, Smart Advise is focused only on WordPress sites. :::

:::warning Warning To use X-Ray Smart Advice, update your alt-php-ssa and alt-php-xray packages to versions alt-php-ssa-0.2-3 and alt-php-xray-0.5-1 or higher by running the following command:

# yum update alt-php-ssa alt-php-xray lve-utils lvemanager --enablerepo=cloudlinux-updates-testing

:::

How to enable X-Ray Smart Advice

:::warning Attention! If you'd like to try Smart Advice and AccelerateWP you should participate in the Beta tester program. To become a beta tester, please send your request at our Beta program page with the signup form here. Once you submit the request, we will send you a confirmation email with program details and terms of use. :::

Requirements

CloudLinux OS Shared Pro or CloudLinux OS Solo
alt-php-xray > 0.5-1 version
lve-utils > 6.3.2-1 version
lvemanager > 7.6.1-1 version
cloudlinux-site-optimization-module > 0.1-1 version

We Recommend:

Enable X-Ray Autotracing on the server
Use alt-php-ssa > 0.2-3 version

How X-Ray Smart Advice works

The main process of looking for advice is X-Ray tracing tasks. The best way for doing this is to enable X-Ray Autotracing. This will allow you to most effectively find Smart Advice for sites that have performance issues without your manual participation. You can find information on how to enable X-Ray Autotracing here. On the other hand, you can create tracing tasks manually or use continuous X-Ray but we suggest you use X-Ray Autotracing for this purpose.

:::tip Note Advice will not be generated by old tracing tasks. :::

While the tracing task is running, X-Ray will look for places where advice can be applied. New advice will be displayed on the Smart Advice tab.

After the X-Ray finds advice you will see new advice in the Review status on the Smart Advice tab. Then you may use the Details button to see which URLs were found by X-Ray that will be speed up by that advice and use Quick Action to enable advice for a site.

Example of details:

After you apply the advice by using Quick Action, the status will change to the Applied.

If a long time has passed since the last time the advice was updated for a site, it will be moved to the Outdated status.

:::tip Note New X-Ray task may update Outdated advice if X-Ray finds performance issues that may be fixed by that advice. Then the status of advice becomes Review. :::

If the process of applying advice fails you will see an error log with a detailed error, you may try to fix it manually and try again or contact our support team for help.

Example when an error appears during advice applying:

Smart Advice FAQ

Q: Why I can't see new advice on the Smart Advice tab?

A: For the generating of advice, it is necessary to run X-Ray tracing tasks, the best way to do it without manual interaction is to use X-Ray Autoracing. You can find more information on how to enable X-Ray Autotracing here.

Useful Smart Advice CLI commands

:::warning Warning! This is an experimental feature. The backwards compatibility is not guaranteed. :::

To obtain the full list of advice generated for your server use the following CLI command:

cl-smart-advice list

For each advice in the list the CLI command returns the following information:

metadata, which includes information about username, domain and website for which the advice is issued
advice information:
- its identifier id
- its type
- its status -- review or applied
- if advice is Premium is_premium
- other internal informational fields

The example output is given below:

{
  "data": [
    {
      "created_at": "2023-04-11T07:33:48.191870+00:00",
      "updated_at": "2023-04-11T07:33:48.191870+00:00",
      "metadata": {
        "username": "user16",
        "domain": "user16.com",
        "website": "/wordpress"
      },
      "advice": {
        "id": 23484,
        "type": "OBJECT_CACHE",
        "status": "review",
        "description": "Turn on Object Caching",
        "is_premium": true,
        "module_name": "object_cache",
        "subscription": {
          "status": "no",
          "upgrade_url": null
        },
        "total_stages": 0,
        "completed_stages": 0,
        "detailed_description": "To improve site performance, enable the Object Caching We recommend applying the advice if you see it frequen
tly for the most valuable URLs of your site."
      }
    },
    {
      "created_at": "2023-04-11T07:33:48.297784+00:00",
      "updated_at": "2023-04-11T08:51:42.362117+00:00",
      "metadata": {
        "username": "user16",
        "domain": "user16.com",
        "website": "/wordpress"
      },
      "advice": {
        "id": 23485,
        "type": "SITE_OPTIMIZATION",
        "status": "applied",
        "description": "Turn on AccelerateWP feature",
        "is_premium": false,
        "module_name": "accelerate_wp",
        "total_stages": 0,
        "completed_stages": 0,
        "detailed_description": "To improve site performance, enable the AccelerateWP optimization feature. We recommend applying the advice if you see it frequently for the most valuable URLs of your site."
      }
    }
  ],
  "result": "success",
  "timestamp": 1681203110
}

End-user X-Ray plugin

:::warning Warning To use the end-user X-Ray plugin, update your LVE Manager and X-Ray packages to the lvemanager-6.3.9-1 (or later) and alt-php-xray-0.3-1 (or later) by running the following command:

# yum update lvemanager alt-php-xray

:::

How to enable/disable the end-user X-Ray plugin

You can hide or show the end-user X-Ray plugin icon by ticking or unticking the proper checkbox in the LVE Manager.

Go to LVE Manager → Options Tab → User interface settings.

:::tip Note The X-Ray plugin icon in the end-user interface is hidden when the checkbox is ticked. :::

How to manage the end-user X-Ray plugin

The web interface of the end-user X-Ray plugin is almost the same as the X-Ray administrator interface.

But there are some differences and they are described further.

End-users can create tasks only for their domains from the drop-down list:
To specify URL or wildcard, end-users should use the input field next to the domain:

You can read about all other basic interface elements and managing tracing tasks in the Managing tracing task section.

:::tip Note Tracing tasks created by an end-user will also be displayed in the administrator interface and administrators can manage the end-user's tasks the same way as they manage their own. At the same time, tasks created by the administrator or other end-users will not be displayed in the UI of the current user. :::

End-user X-Ray plugin limitations

The end-user X-Ray plugin does not support creating continuous tasks.
The end-user has a limit of tracing tasks running at a time. Before starting the next task, the end-user should wait for the completion of the previous ones or forcefully stop the running ones. Otherwise, the user will get the next error:

:::tip Note The current limit is one tracing task per user. :::
The administrator and the end-user can’t run the tracing task for the same Domain/URL at the same time. Once, the administrator started a specific tracing task, the end-user will not be able to duplicate it. And the same is true for the administrators – they will just see the running task for the specific domain and see the notification that they're trying to create a tracing task with a duplicated URL.
If continuous tracing is enabled for the domain, the end-user will not be able to create a new task for this domain because the same rule works - it will be a duplicate of the existing tracing tasks. The next warning will appear:

To solve this, the existing running tasks for the same Domain/URL should be stopped or completed. You can find more details about this in the FAQ.
If a user's tracing task was created for a domain which is using the FPM handler there's an additional limitation. To avoid frequent reloads of the particular FPM service, Start tracing , Stop tracing or Continue tracing action would be blocked in case if the latest reload of a corresponding FPM service was done less than 1 minute ago.
If a user gets such an error message - it means that 1 reload in 1 minute for a particular FPM service has been already done. Just try performing the same operation once again in a while.

X-Ray automated throttling detection

:::tip Note CPU throttling detection is available since alt-php-xray-0.3-2 and lvemanager-xray-0.5-2.

IO/IOPS throttling detection is available since alt-php-xray-0.3-7 and lvemanager-xray-0.7-1.

kmod-lve-2.0-23 (and later) for CloudLinux OS Shared 8 or CloudLinux OS Shared 7 hybrid
kernel-1.5-58 (and later) for CloudLinux OS Shared 7 or CloudLinux OS Shared 6 hybrid

are also required to utilize the feature of IO/IOPS throttling detection. :::

:::warning Warning X-Ray automated throttling detection feature is not supported for CloudLinux OS Shared 6 :::

The X-Ray automated throttling detection system checks if the account exceeds LVE limits by CPU or by IO/IOPS during the HTTP request execution. Requests with exceeded LVE limits are indicated in both X-Ray Administrator and X-Ray User plugins.

If CPU limiting was detected for a particular request, it is indicated in the X-Ray UI that the system itself has slowed down the request processing due to CPU throttling and this is apparently not a performance issue in the PHP code.

If limiting by IO and IOPS in total was detected for a particular request, it is indicated in the X-Ray UI in the same manner, except for the cause of slowing down the request -- IO throttling.

The case of both limiting for the request is also possible.

Requests with exceeded LVE limits are also marked in the request detailed view.

Requests with exceeded LVE limits are marked in the PDF report as well.

X-Ray client

X-Ray client is a PHP extension named xray.so. It analyzes the processing time of the entire request and its parts and then sends the data to the X-Ray agent.

List of supported PHP versions

The list of currently supported PHP versions:


ALT PHP:	EA PHP:	Plesk PHP	DirectAdmin PHP	Other panels PHP
alt-php54 alt-php55 alt-php56 alt-php70 alt-php71 alt-php72 alt-php73 alt-php74 alt-php80 alt-php81	ea-php54 ea-php55 ea-php56 ea-php70 ea-php71 ea-php72 ea-php73 ea-php74	php54 php55 php56 php70 php71 php72 php73 php74	php54 php55 php56 php70 php71 php72 php73 php74	54 55 56 70 71 72 73 74

:::warning Warning! php-zts and custom PHPs, rolled in selector, are not supported :::

Functions that X-Ray client can hook

Database queries

Functions from the MySQL extension:
- mysql_query
- mysql_db_query
- mysql_unbuffered_query
Functions from the MySQLi extension:
- mysqli_query
- mysqli::query
- mysqli_multi_query
- mysqli::multi_query
- mysqli_real_query
- mysqli::real_query
Functions from the PDO extension:
- PDO::exec
- PDO::query
- PDOStatement::execute

External requests

Function curl_exec

System PHP functions

It may be any PHP system function which can be related to a PHP engine or other PHP extension, for example fopen() or json_encode(). A list of these functions can be found here.