# Introduction {% hint style="info" %} End of life for Flopsar 3.0. This product is no longer supported. {% endhint %} ## What is Flopsar® Flopsar is a *Fault Detection and Diagnosis* software for Java systems developed by [Flopsar Technology](http://flopsar.com). Its main goal is to detect errors and problems in your Java systems and help to find the root cause quickly. What is the most important thing in systems maintenance? Systems maintenance relies on making the systems highly available and responsive. This implies that we must maintain the number of outages as low as possible and the time needed to resolve any issues must be as short as possible. However, if the worst happens and the system fails, besides the fact it failed we need to know why. Knowing why is not sufficient if finding the cause can take a whole day or even longer. These are the main problems that need to be addressed. Unlike other APM-like tools, Flopsar does not require users to predict what may go wrong in their systems. Moreover, we do not lose information about your applications processing by probing or collecting only some vague averages and aggregated data. Flopsar is a highly customizable and extensible product. It is framework and application servers agnostic and all it needs is just JVM. Flopsar has more key features that help users to maintain their systems better. Its area of application is not only limited to systems maintenance, it is also used in applications development processes successfully. Its features can be extended by using plugins, which allow users to perform extra processing at runtime, with no application source code changes. {% hint style="info" %} For older releases documentation go to [legacy.flopsar.com](http://legacy.flopsar.com) {% endhint %} *** Release date: March 2020 # Versioning Every version label is composed of three numbers `X.Y.Z`: `X` Major version number: major changes, e.g. architecture change. `Y` Minor version number: minor changes, e.g. new features. `Z` Small version number: bugs fixes. {% hint style="info" %} Versions `X.Y` are equivalent to `X.Y.0` {% endhint %} Do not mix binaries form different releases, since they are not compatible. You can mix only those components, which have the same both major and minor release numbers. {% hint style="danger" %} Incompatible components are unable to work together. Any connection between two incompatible components is dropped. {% endhint %} # Licensing Flopsar is proprietary software and you need a valid license to run it. The license is a `license.key` file, which should be obtained separately. There are some restrictions on the Flopsar usage depending on the license type you purchased. {% hint style="success" %} Flopsar Lite does not require any license to run. {% endhint %} You can always display information about your license by executing the command: ``` $ flopsar info --license license.key ``` # Supported Platforms and Requirements ## Agent | AGENT FILE | SUPPORTED JVM VERSION | | -------------------------- | --------------------- | | `flopsar-agent-N.jar` | 1.6 | | `flopsar-agent-N.jar` | 1.7 | | `flopsar-agent-N.jar` | 1.8 | | `flopsar-agent-N-jvm9.jar` | 9 | | `flopsar-agent-N-jvm9.jar` | 10 | | `flopsar-agent-N-jvm9.jar` | 11 | {% hint style="success" %} The agent is framework and application servers agnostic, so no detailed list of supported environments is needed. {% endhint %} ## Workstation `Workstation` requires `JVM 1.8u131+`. It can be run only on those platforms which are supported by [JavaFX](http://docs.oracle.com/javase/8/javase-clienttechnologies.htm). {% hint style="warning" %} `Workstation` does not support JVM 9+. {% endhint %} ## Flopsar Server Flopsar Server supports only 64 bit GNU/Linux systems with the kernel at least 2.6.33. # Distribution Packages Flopsar distribution binaries include: `flopsar-server-X.Y.Z.deb` - Flopsar Server installation package. `flopsar-server-X.Y.Z.rpm` - Flopsar Server installation package. `flopsar-server-X.Y.Z-Lite.deb` - Flopsar Lite Server installation package. `flopsar-server-X.Y.Z-Lite.rpm` - Flopsar Lite Server installation package. `flopsar-agent-X.Y.Z.jar` - Flopsar Agent for JVM 8 and older. `flopsar-agent-X.Y.Z-jvm9.jar` - Flopsar Agent for JVM 9 and newer. `flopsar-fdbc-X.Y.Z.jar` - Flopsar Database API. `flopsar-workstation-X.Y.Z.zip` - Flopsar Workstation application. `setup-flopsar-workstation-X.Y.Z.exe` - Flopsar Workstation install the application for Windows. `SHA256.asc` - PGP signed file of all binaries checksums. In order to verify the signature use our [PGP public key](http://flopsar.com/pgp). {% hint style="success" %} All the files can be downloaded from our [release site](http://bin.flopsar.com). {% endhint %} ## Verifying Packages In order to verify `SHA256.asc` file, you must first import our public key. You can do this by executing the following command: ``` $ gpg --keyserver pgp.mit.edu --recv-keys 4E1CBB4012E01B7A ``` and verify the file itself: ``` $ gpg --verify SHA256.asc ``` If the file is valid, you should see the output: ``` gpg: Good signature from "Flopsar Technology (Flopsar Technology Secure Key) " ``` # Release Notes Flopsar 3.0 is a brand new release. It has been redesigned from the ground up. No migration mechanism from the older Flopsar version is available. # Flopsar Lite Flopsar Lite is free of charge version of the Flopsar software. It does not require the license file to run. The lite version has the same features as the paid one, however with some limitations. * It can be run only in the standalone mode. * It purges all the collected data every three hours. * It supports only a single `Agent` and a single `Workstation` client. {% hint style="info" %} The difference between the lite and standard versions lies only in the server component. The remaining components stay the same for both versions. {% endhint %} You can either install the server package on your own machine or use a [Docker image](https://hub.docker.com/r/flopsar/server-lite). # Architecture Fundamentals Flopsar consists of three main components: * Agent * Server * Workstation `Agent` is installed on JVM applications and produces data, `Server` collects and serves data, and `Workstation` provides access to the collected data. The entire Flopsar environment can be installed in one of the two architectures: [standalone](/3.0/administrator-guide/architecture-fundamentals#standalone-architecture) or [distributed](/3.0/administrator-guide/architecture-fundamentals#distributed-architecture). ## Standalone Architecture In this architecture, both manager and database are bundled and make use of the same working directory. The architecture is easier to manage than the distributed one and is sufficient for most of the installations. If you experience any performance issues with your installation or have any other reason not to use the standalone architecture then consider using a distributed one. ![Standalone Architecture](/files/-Lwmv8j5IqYyvOgtOBSl) In order to install Flopsar environment in the standalone architecture, you must install `Server` in the [standalone mode](/3.0/administrator-guide/server#standalone-mode). ### Networking There is only a single TCP server exposed by `Server`. Any client connects to this TCP server. This means the following connections must be allowed: * outgoing connections from agents to the server, * outgoing connections from workstations to the server. ## Distributed Architecture In this architecture manager and database operate separately. They can be both installed either on a single machine or separate ones. ![Distributed Architecture](/files/-Lx0zZduE4IZW5D7Vqm6) ### Networking In this architecture, `Manager` and `Database` instances expose their own TCP servers. This means the following connections must be allowed: * outgoing connections from agents to the `Manager` server, * outgoing connections from agents to the `Database` servers, * outgoing connections from workstations to the `Manager` server, * outgoing connections from workstation to the `Database` servers, * outgoing connections from databases to the `Manager` server. # Installation Before you install the software, make sure you meet the following requirements: * You have at least a few GB of storage for `Server` data. * Your networking environment does not block TCP connections between components. ## Server There are two packages available for installation: `flopsar-server-X.Y.Z.deb` and `flopsar-server-X.Y.Z.rpm`. For Debian based distributions you can install it by executing the following command: ``` # dpkg -i flopsar-server-X.Y.Z.deb ``` For Red Hat based distributions you can install it by executing one of the following commands: ``` # rpm -ih flopsar-server-X.Y.Z.rpm # zypper install flopsar-server-X.Y.Z.rpm # yum install flopsar-server-X.Y.Z.rpm ``` {% hint style="warning" %} You will not be able to run Flopsar unless you accept the [Flopsar End User License Agreement](http://flopsar.com/EULA.txt) (EULA). In order to accept the license, you have to set the environment variable `FLOPSAR_EULA_ACCEPT` to `YES`. {% endhint %} You can verify if the Flopsar has been installed successfully by executing the command: ``` $ flopsar ``` You should see the Flopsar banner similar to the one below: ``` ******************************************************* * ______ _ ____ _____ _____ _____ * * | ____| | / __ \| __ \ / ____| /\ | __ \ * * | |__ | | | | | | |__) | (___ / \ | |__) | * * | __| | | | | | | ___/ \___ \ / /\ \ | _ / * * | | | |___| |__| | | ____) / ____ \| | \ \ * * |_| |______\____/|_| |_____/_/ \_\_| \_\ * * * ******************************************************* Version: 3.0.0 Build: 3.0.0 Released on: (release date) Copyright (C) 2012-present by Flopsar Technology http://flopsar.com ******************************************************* ``` {% hint style="info" %} For either `standalone` or `manager` modes you must have a valid license file inside your environment directory. For `database` mode, the license file is not required. {% endhint %} ### Installation Example In this example, we use the standalone architecture. 1. Create a new environment: ``` $ flopsar init /opt/flopsar-server ``` 2. Copy the license file: ``` $ cp license.key /opt/flopsar-server/ ``` 3. Accept the EULA: ``` $ export FLOPSAR_EULA_ACCEPT=YES ``` 4. Start the server: ``` $ flopsar start /opt/flopsar-server ``` ## Agent In order to install `Agent`, follow carefully the instructions below: 1. copy the agent file `flopsar-agent-X.Y.Z.jar`to your server machine. 2. create a directory (we call this directory as Flopsar home directory) and `ext` subdirectory for the agent data. 3. [configure](/3.0/administrator-guide/agent#configuration) the agent. 4. optionally put all your agent [plugins](/3.0/developer-guide/untitled) into the `ext` directory. 5. restart your application. When the agent starts for the first time it does not have any configuration deployed. It just registers itself to the manager, after a successful connection, and does not instrument anything. ### Installation Example In this example we assume there is already a `Server` instance running and listening on `A.B.C.D:port` socket address. 1. Create a Flopsar home directory along with the `ext` directory: ``` $ mkdir /opt/flopsar $ mkdir /opt/flopsar/ext ``` 2. Copy the agent file: ``` $ cp flopsar-agent-X.Y.Z.jar /opt/flopsar/ ``` 3. Copy some plugins: ``` $ cp my-plugin.jar /opt/flopsar/ext/ ``` 4. Update JVM start options: ``` -Dflopsar.id=MyApp \ -Dflopsar.home=/opt/flopsar \ -Dflopsar.manager=A.B.C.D:port \ -javaagent:/opt/flopsar/flopsar-agent-X.Y.Z.jar ``` 5. Restart your application. If the installation was successful, a log file `/opt/flopsar/MyApp.0.log` should appear. # Agent `Agent` is a core component. It runs on a JVM and its main goal is to collect data from applications running on the JVM and the JVM itself. Each agent is attached to a JVM. It initiates a single two-way TCP connection to `Manager` and the second one-way TCP connection to `Database`. The manager address is specified in the agent configuration options while the database address is retrieved from the manager. ![Agent Networking](/files/-LwmmZser9zafcAN4jeZ) ## Configuration Agents are configured by their runtime options. ### Mandatory Options * **`-Dflopsar.id`** Primary identifier of a JVM instance. It should consist of only word characters and have no more than 62 characters. If you provide a name consisting of more characters, the agent will truncate the name. * **`-javaagent`** Path to your agent file `flopsar-agent-X.Y.Z.jar`. * **`-Dflopsar.manager`** Socket address of the Flopsar TCP server. Agents initiate connections to this server and the connections are maintained until the agents are shutdown. It should be specified in the form: `host:port`. * **`-Dflopsar.home`** Path to the Flopsar working directory. Log files are stored in this directory. {% hint style="info" %} The Flopsar working directory must contain the`ext` subdirectory. This is the place you put all your plugin classes. {% endhint %} The above options must be set otherwise, the agent will not work. For example, your configuration could look like this one: ``` -javaagent:/path/to/flopsar-agent-*.jar \ -Dflopsar.id=MyAgent \ -Dflopsar.manager=172.16.0.1:9000 \ -Dflopsar.home=/path/to/some/dir ``` ### Additional Options There are some additional options, which are not mandatory. **`-Dflopsar.ext.packages`** Comma-separated list of packages, which all formatter classes come from. If not specified or empty, it contains only `org.flopsar.ext` package by default. Classes from these packages will be loaded by the agent from the `ext` subdirectory. {% hint style="info" %} No class coming from any of the specified packages will be instrumented since the agent assumes these are plugin classes. {% endhint %} {% hint style="warning" %} Do not put your plugin classes into packages, which have the same root package as your instrumented application, e.g. if all your application classes belong to `com.foo.bar.*` package, make sure your plugin classes do not. {% endhint %} **`-Dflopsar.cl`** Comma-separated list of class loader classes, which should be instrumented. You can use this option if you want to instrument some class loader, which is not automatically instrumented by the agent. This can be useful if the agent cannot load some classes by already instrumented class loaders. This option is empty by default. **`-Dflopsar.id.dyn`** Secondary identifier of a server instance. If you use this option, you must set its value `V` to only another JVM option, i.e. the agent will take the value of this option and execute the method: ```java java.lang.String getProperty(V, null) ``` The output of the method will be the effective value of the secondary identifier. If you use both identifiers, primary and secondary, the agent will combine them into a single, composite identifier. They will be concatenated by an underscore character. The composite identifier’s length is up to 62 characters. **`-Dflopsar.errline`** If you want the agent to skip line numbers (in stack traces) in the instrumented methods, set this option value to `false`. The default value is `true`. **`-Dflopsar.logsize`** Total maximum size in MB of log files. By default, this value is set to 100. By setting this option you specify how much space log files can use. **`-Dflopsar.qsize`** Total maximum size of the agent outbound queue size in MB. By setting this option you specify how much memory the agent can use for the queue. **`-Dflopsar.qratio`** Ratio of agent outbound queue size to the maximum JVM memory size in percentages. By default, this value is set to 20. By setting this option you specify how much memory the agent can use. If you specify both this option and `-Dflopsar.qsize`, the latter takes precedence. **`-Dflopsar.debug`** This option enables/disables additional logging for class loading. If you have problems during instrumentation regarding class loading, you can enable this feature to track, which class loader loads classes. The default value is `false`. For example, your extended configuration could look like this one: ``` -javaagent:/path/to/flopsar-agent-VER.jar \ -Dflopsar.id=MyAgent \ -Dflopsar.manager=172.16.0.1:9000 \ -Dflopsar.home=/path/to/some/dir \ -Dflopsar.logsize=200 \ -Dflopsar.qsize=400 \ -Dflopsar.errline=true ``` There are various places where you can add your custom JVM options, it depends on the Java software you use. Please, refer to your Java software documentation for more details about adding extra JVM options. {% hint style="warning" %} Please, make sure that every agent uses its own, unique identifier. If there are multiple agents with the same identifiers and they connect to the same manager, their connections will be dropped by the manager every time a new connection attempt appears from any of the agents. {% endhint %} {% hint style="info" %} If you plan to install agent in OSGi environments, make sure to add `com.flopsar` classes to the OSGi container class path. Please, refer to your OSGi environment documentation for details. {% endhint %} ## Data Collecting Considerations Although the agent is designed to collect as much data as possible, we are not able to guarantee that all the instrumented method calls data will be collected. It is because the agent priority is not to crash the application it runs on. That has some serious implications on the agent design and operation. We just favor stability and reliability over data completeness. Each call of an instrumented method, symbol, or metric value is represented as a serialized entity called *packet*. As soon as the packet is created it is ready to be sent to a database. In order for the packet to be sent, it must be first put in the outbound queue. This is the point where the agent decides whether put it or not. The packet is put in the queue if and only if the connection to the database is established and there is enough space in the outbound queue (the size of the outbound queue can be customized). Otherwise, the packet is dropped. This scenario holds for each packet. # Server Server is one of the key Flopsar components, which is responsible for managing the entire Flopsar environment. Depending on the mode the server is run, it consists of one or two components. The server can operate in one of the three modes: **`manager`** This mode makes the server run as a manager only. **`database`** This mode makes the server run as a database instance. **`standalone`** This mode makes the server run as both manager and database ([standalone architecture](/3.0/administrator-guide/architecture-fundamentals#standalone-architecture)). ## Configuration Configuration is defined in `config.json` file under `FLOPSAR_WDIR/conf` directory. {% tabs %} {% tab title="Standalone" %} {% code title="config.json" %} ``` { "server" : { "host" : "*", "port" : 9000 }, "archive" : { "enable" : false, "policy" : "purge", "scheduled on" : { "hour" : 3, "minute" : 0 }, "retention" : 72 } } ``` {% endcode %} * **`server/host`** : IP address, which the server listens on. If `*` is specified, the server will listen on all available addresses. * **`server/port`** : TCP port, which the server listens on. * **`archive/enable`** : if set to `true`/`false`, the archiving procedure ie enabled/disabled. * **`archive/policy`** : if set to `archive`, all the data will be archived. If set to `purge`, the data eligible for archiving will be deleted. * **`archive/scheduled on/hour`** : hour of the day at which the archiving procedure will start. * **`archive/scheduled on/minute`** : minute of the hour at which the archiving procedure will start. * **`archive/retention`** : data retention period in hours. {% endtab %} {% tab title="Manager" %} {% code title="config.json" %} ``` { "server" : { "host" : "*", "port" : 9000 } } ``` {% endcode %} * **`server/host`** : IP address, which the server listens on. If `*` is specified, the server will listen on all available addresses. * **`server/port`** : TCP port, which the server listens on. {% endtab %} {% tab title="Database" %} {% code title="config.json" %} ``` { "server" : { "host" : "*", "port" : 9000 }, "manager" : { "host" : "host:port", "port" : 9000 }, "redirect" : { "host" : "*", "port" : 9000 }, "archive" : { "enable" : false, "policy" : "purge", "scheduled on" : { "hour" : 3, "minute" : 0 }, "retention" : 72 } } ``` {% endcode %} * **`server/host`** : IP address, which the server listens on. If `*` is specified, the server will listen on all available addresses. * **`server/port`** : TCP port, which the server listens on. * **`manager/host`** : IP address, which the manager listens on. The database will connect to this address. * **`manager/port`** : TCP port, which the manager listens on. * **`redirect/host`** : IP address, which the database reports to the manager. * **`redirect/port`** : TCP port, which the database reports to the manager. * **`archive/enable`** : if set to `true`/`false`, the archiving procedure ie enabled/disabled. * **`archive/policy`** : if set to `archive`, all the data will be archived. If set to `purge`, the data eligible for archiving will be deleted. * **`archive/scheduled on/hour`** : hour of day at which the archiving procedure will start. * **`archive/scheduled on/minute`** : minute of hour at which the archiving procedure will start. * **`archive/retention`** : data retention period in hours. The `redirect` option is useful when you have your database behind a firewall and no `Workstation` can connect to the `server` socket address. If this is the case, you just need to specify the `redirect` option, which will be used by `Workstation` instances to connect to the `server` socket address. ![Redirect Usage](/files/-LyB1qIb-Exu04bdXuWw) {% endtab %} {% endtabs %} ## Local Documentation Flopsar executable usage is also documented in man pages. You can access these pages calling: ``` $ man flopsar ``` ## Manager Mode Manager is responsible for managing the entire Flopsar environment. When Flopsar runs in this mode, it means the [distributed architecture](/3.0/administrator-guide/architecture-fundamentals#distributed-architecture) is in use. ### Command Line Interface The manager is managed by means of CLI subcommands of `flopsar` executable. #### Creating Manager Environment In order to create a new `Manager` environment execute the following command: ```bash $ flopsar create --mode=manager MANAGER_DIR ``` where `MANAGER_DIR` is a directory, which the environment is created. #### Starting Manager In order to start the manager execute the command below: ``` $ flopsar start MANAGER_DIR ``` #### Rerouting Agents Every agent needs to know, which database it should connect to. In the distributed architecture, there can be multiple databases running independently. In order to attach an agent to a particular database, you must execute the following command: ``` $ flopsar agent reroute --pattern=PATTERN --agent-type=java --database=host:port FLOPSAR_DIR ``` This command will reroute Java agents, which names match the `PATTERN` to the database listening on `host:port`. #### Removing Agent Data If you want to remove some agent from Manager, you just execute the command: ``` $ flopsar agent rm --pattern=PATTERN --agent-type=java FLOPSAR_WDIR ``` This command will remove all the data for any Java agent, which name matches the specified `PATTERN`. {% hint style="info" %} You cannot remove any agent data while the server is running. {% endhint %} ## Database Mode Server runs as a database in this mode. ## Plugins `Server` provides a [plugin](/3.0/developer-guide/server-plugins) mechanism, which allows to replace and extends Flopsar functionalities. {% hint style="success" %} Currently, only `Manager` instances support plugins. {% endhint %} In order to install a new plugin, you need to execute the following command: ``` $ flopsar plugin add --name=PLG_NAME --lib=PLG_LIB --conf=PLG_CONF FLOPSAR_WDIR ``` where: * `PLG_NAME` : user-defined plugin name, * `PLG_LIB` : absolute path to the plugin library file, * `PLG_CONF` : optionally, a plugin configuration file name. {% hint style="warning" %} Please note, if your plugin has a configuration file, this file must be in your current working directory which you execute the `plugin add` command from. {% endhint %} You can always print a list of all the installed plugins by executing the following command: ``` $ flopsar plugin list FLOPSAR_WDIR ``` In order to uninstall a plugin, you just need to execute the command: ``` $ flopsar plugin rm --name=PLG_NAME FLOPSAR_WDIR ``` where `PLG_NAME` is the name of the plugin you want to uninstall. ### Example Suppose, we have the following resources: a plugin file `myplugin.so` and its configuration file `myplugin.conf` in a current working directory, which is, for example `/opt/flopsar`. Flopsar manager installation is in `/opt/flopsar/flopsar-server` directory: ``` $ pwd /opt/flopsar $ ls flopsar-server myplugin.so myplugin.conf ``` In order to add the plugin, we need to execute the following command: ``` $ flopsar plugin add --name=myFancyPlugin --lib=/opt/flopsar/myplugin.so \ --conf=myplugin.conf flopsar-server ``` If the plugin was successfully installed, we should see the following message: ``` Plugin "myFancyPlugin" added successfully. ``` If you do not see the above message, an error must have occurred. In order to get more details, add an extra option `--verbose` to the command, and execute it again. Now, a new log file `out.log` should be created with more information about the issue. # Data Archiving Data archiving is a feature, which helps you to keep your database volume under control. If enabled, it runs every day at the scheduled time, compresses part of the data, and moves the compressed data to the `/storage.db/arch` directory. After this operation, your database volume is smaller. First of all, in order to enable the archiving feature, you must set `archive/enable` option to `true` in the [configuration file](/3.0/administrator-guide/server#configuration). To make the archiving procedure start at the `HH:MM` time, you must set the option `archive/scheduled on/hour` to `HH` and `archive/scheduled on/minute` to `MM`. When the time comes, the database process will spawn a new archiving process, which does the job. When the archiving procedure starts, it must know which data it should archive. This is managed by `archive/retention` option. The archiving procedure always produces a single file. The file is stored in the directory `/storage.db/arch` and its name is of the form `flopsar.arch-YYYY-MM-DD_HHMM`, where `YYYY-MM-DD_HHMM` is a date time of the procedure beginning. Besides the output file, there is a log file `flopsar.arch-YYYY-MM-DD_HHMM.log` created in the same directory. The log file contains some information about the archiving operation. If you want to only keep your database volume under control and you are not interested in old data, then you can set the option `archive/policy` to `purge`. This policy makes the database delete the old data. {% hint style="info" %} Data Archiving feature is available only for `Database` instances. {% endhint %} ## Database Dump If you want to dump your database data on demand, you can do it by executing the following command: ``` $ flopsar arch dump --from=START_DATE --thru=END_DATE --agents=AGENTS_REGEX FLOPSAR_WDIR ``` where: * `START_DATE/END_DATE` : data time range. * `AGENTS_REGEX` : regular expression for agents pattern. {% hint style="warning" %} You cannot dump the data when the database is running. {% endhint %} ## Extracting Database Archive If you want to extract the archived file, you just need to execute the following command: ``` $ flopsar arch extract ARCH_FILE ``` This command will extract the specified `ARCH_FILE` file in the current working directory. A new directory will be created with the extracted database data. Additionally, a log file `flopsar-arch-extract.log` will be stored with the extraction procedure logs. # Authentication and Authorization ## User Any user has the following attributes: * username * password * permissions * list of banned agents ### Getting Users Information In order to print all users, execute the following command: ``` $ flopsar auth info FLOPSAR_WDIR ``` In order to print detailed information about a particular user (e.g. user1) execute the command: ```bash $ flopsar auth info --username=user1 FLOPSAR_WDIR ``` ### Adding/Deleting Users In order to add a new user `user1`, execute the following command: ```bash $ flopsar auth add --username=user1 FLOPSAR_WDIR ``` In order to delete the user `user1` execute the following command: ```bash $ flopsar auth rm --username=user1 FLOPSAR_WDIR ``` ## Authentication Flopsar supports two types of authentication providers: internal (local) and external. ### Internal This is a default authentication system used by Flopsar. All the users information is stored in `Manager`. The authentication itself is performed internally, i.e. is not delegated to any other external authentication system. This means, that users password are stored encrypted in `Manager`. In order to change a password for `user1` user, you must make use of the following command: ```bash $ flopsar auth mod --username=user1 --password FLOPSAR_WDIR ``` ### External This type delegates the authentication to other, third-party authentication systems. In order to enable it, you need to implement a specific `Server` [plugin](/3.0/developer-guide/server-plugins#authentication-plugin) and [install](/3.0/administrator-guide/server#plugins) it on the Manager. ## Authorization {% hint style="warning" %} No matter which authentication type you choose, the authorization is always performed in the Manager. That means all users must exist in the Manager. {% endhint %} #### Permissions The following permissions are available: | Symbol | Description | | ------ | ----------------------------------- | | c | User can edit agents configuration. | | p | User can see method parameters. | | o | User can see agents online data. | | q | User can query agents directly. | In order to set the permissions for the user `user1`, execute the command: ```bash $ flopsar auth mod --username=user1 --perms=perms_vals FLOPSAR_WDIR ``` where `perms_vals` are the permissions symbols prefixed with `+` or `-` depending on whether you want to add or revoke the permission. For example, if you want to allow `user1` to see methods parameters and agents online data, execute the following command: ```bash $ flopsar auth mod --username=user1 --perms=+po FLOPSAR_WDIR ``` In order to revoke those permissions, execute the command: ```bash $ flopsar auth mod --username=user1 --perms=-po FLOPSAR_WDIR ``` #### Banned Agents A user can see all data from any agent by default. In order to prevent a user (`user1`) from seeing particular agents, you need to execute the following command: ```bash $ flopsar auth mod --username=user1 --ban=agents_pattern FLOPSAR_WDIR ``` where `agents_pattern` is a regular expression for agents' names. In order to unban some agents, execute the command: ```bash $ flopsar auth mod --username=user1 --unban=agents_pattern FLOPSAR_WDIR ``` ## # Logging Each Flopsar module (application) uses logging levels. The levels are defined as follows: **`finest`** : most detailed logging. It should not be used during normal, production operation since it can degrade the application performance significantly. Use only in cases when you suspect some problems with the application, want to trace how it processes requests, requested by the Flopsar Technology Support to help investigate some problems, or any other problems. It prints lots of diagnostic information, which might not be covered by this manual. **`finer`** : less detailed logging than `finest`. Used to log some additional information about the application operation and data processing. **`fine/debug`** : more detailed logging than `info`. Used to log some additional information about the application operation and data processing. **`info`** : default level used in applications. Used only to log some runtime information to verify a configuration and the like. You should use this level on normal, production operation. **`warn`** : used in cases when some error occurs but with no impact on the application stability and data integrity. **`error`** : normally, there should be no error messages in the log. The level is used in cases when some error occurs. These errors impact the application operation stability and data integrity but not all of them. The messages are informative, so you should know whether you can ignore it or not. Any system calls and external libraries errors are also logged in this level. **`fatal/severe`** : there should be no such messages in the log. Messages of this level are only printed in cases when some fatal error occurred and the application cannot run any longer. For example, some misconfigured options can result in printing fatal messages in the log. Please, read them and fix your configuration. The messages are usually very informative. ## Agent By default, the agent always starts with `finest` logger level and after 2 minutes it is switched to `info` automatically. You can always change the level in runtime using `Workstation`. ### Messages Each logger message entry has the following structure: ``` DATE [LEVEL] Message content ``` where: `DATE` : current date, `LEVEL` : current logger level, `THREAD` : current Java execution thread name. ## Server Server stores its logs in the `FLOPSAR_WDIR/logs` directory in `flopsar.log` file. The log file does not roll over so its size must be controlled manually. {% hint style="info" %} `Server` module uses only `debug`, `info`, `warn`, `error`and `fatal` levels. {% endhint %} # Docker Flopsar Docker images can be found at [Docker Hub](https://hub.docker.com/u/flopsar). The following images are published: * [Flopsar Server Lite](https://hub.docker.com/r/flopsar/server-lite) - this is a [Lite](/3.0/overview/flopsar-lite) version. ## Demo We share simple demo images so that you can try Flopsar out. In order to run the demo, you need to follow the instructions below: 1. Download the demo script from our [GitHub](https://github.com/flopsar/docker/tree/master/demo3) repository. 2. Download `Workstation` binary from our [releases site](http://bin.flopsar.com). 3. Change `VERSION` variable in the demo script to make it [compatible](/3.0/overview/untitled) with the `Workstation` version. 4. Execute the demo script and connect to the Flopsar server at address `localhost:9000` # Workstation ## Installation Flopsar Workstation is a GUI client of the Flopsar environment. Installation is straightforward, just copy the `flopsar-workstation-X.Y.Z.zip` file to your machine and uncompress it. Next, run a workstation script from `bin` directory. {% hint style="warning" %} You should have your Java environment set before you try to run the workstation application. {% endhint %} ## Accessing Flopsar Environment When you start the workstation a login window should appear. You can either access a remote Flopsar sever or a local Flopsar database storage. ### Remote Access If you want to access a remote Flopsar server fill out the form below and press `CONNECT` button. If you run the workstation for the first time, the `Connection ID` combo box is empty. In order to connect to your manager instance, you must specify some connection identifier (a label) and input the manager socket address in the form `host:port`. You must also provide your credentials and then press the `CONNECT` button. ![Remote Access](/files/-LwI93KzZQ3PGjUdVEVt) {% hint style="success" %} Default credentials when using Flopsar Internal Authentication are as follows: the username is **admin** and the password is **flopsar**. {% endhint %} ### Local Access In order to access a local Flopsar database, click `SWITCH TO LOCAL` link and fill out the form below: ![Local Database Access](/files/-LwI8z0bZymYD2WMUufW) You have two options to choose from: `Storage` : points to the database storage directory `Archive File` : points to the archive file (either dump or ordinary archive file). In this case, the archive file will be extracted and then accessed. Next time you access the storage you just need to select `Storage` option and point to the extracted directory. ## Settings `Workstation` stores its settings locally, on the machine it runs on. It makes use of [Java Preferences API](http://docs.oracle.com/javase/7/docs/technotes/guides/preferences/) to store its configuration. The physical location of the settings depends on a platform the workstation runs on. When you exit the application, it stores its current settings. ### General ![General Preferences](/files/-LwOj8iXrZcbdNSRabcY) By default, all the workstation logs are stored in a user home directory. This can be changed in the preferences window above, where you can set both the logger level and the location of the logs. You can also check if there is a new Flopsar version available. In order to check it, just press the `Check now` button. If you select `Check automatically`, then whenever you start the workstation, it will check for updates automatically. ### Connections Your manager connections information is also stored in the settings. You can either edit or delete these connections in the preferences window below. ![](/files/-LwOkHdf2NbME7pqWAjz) ### Permissions This view presents a list of all [permissions](/3.0/administrator-guide/authentication-and-authorization#permissions). A green circle denotes if the corresponding permission is granted. ![List of user's permissions](/files/-LyTD6d66sTGJj36p1iM) #### Permissions Implications The following features are hidden or disabled when permissions are not granted: ***Configure Agents*** * Green plus button is not visible in the [Attached Agents](/3.0/user-guide/agents-configurations#attaching-agents) tab of the active configuration view. * `Profile` submenu of the context menu is disabled in: * [root call execution stack view](/3.0/user-guide/execution-stack-interpretation#context-menu) * [root call stack spectrum view](/3.0/user-guide/execution-stack-interpretation#context-menu-1) ***View Parameters*** * Parameters details are not displayed. * Characters `******` are displayed in the [parameters column](/3.0/user-guide/data-browsing#browse-result-view). ***View Online Data*** * [Monitoring tab](/3.0/user-guide/online-monitoring#settings) is not visible in the preferences window. ***Query Agents*** * [Properties](/3.0/user-guide/agents-status#jvm-properties), [Packages](/3.0/user-guide/agents-status#jvm-packages), [Class Loaders](/3.0/user-guide/agents-status#jvm-class-loaders), [Agent Logs](/3.0/user-guide/agents-status#agent-logs) and [State](/3.0/user-guide/agents-status#jvm-state) tabs are not visible. * [Threads dump](/3.0/user-guide/online-monitoring#agents) button is not visible. # Agents Profiles ## Profiles View Agents profiles are managed from `Workstation` and stored in `Manager`. The profiles view is divided into two parts * list of profiles * selected profile details ### List of Profiles This is a view where you can see a list of all the available profiles. Here you can add, import, and delete profiles. Each list entry (representing a profile) has a label, number, and a trash icon. The label is a profile name, the banner shows how many agents are attached to the profile. The trash icon is used to remove the profile. ![Profiles List](/files/-Lzg5PYRCoLrSBR72Bic) There is a toolbar at the top with a search text entry and two icons, which provide import and add profile features: 1. This action will create a new profile by importing it from a file. You can import profiles from *2.3, 2.4* and *3.0* Flopsar versions. 2. This action will create a new profile. ## Profile Details In order to display the profile details, you need to double click the desired profile in the list. In general, there are two types of profiles: *draft* and *active*. ### Draft Profile View Draft profile is used for editing. Each profile is composed of rules and options. The following options are available: `cpu time` : enables/disables a CPU time metric, `threshold` : the lowest value of instrumented methods execution time (in milliseconds) above which the method calls will be reported, `jmx period` : how frequently (in seconds) agents should collect JMX data. ![Draft Profile: Instrumentation Rules](/files/-Lx5qF6Z52wor7BoEcfw) ![Draft Profile: MBeans](/files/-LzvT141NPJuQNVtwq6f) The `Instrumentation Rules` tab contains a list of [instrumentation rules](/3.0/user-guide/agents-configurations#instrumentation-rules). These rules can be enabled or disabled using a green switch by each rule. The `MBeans` tab contains a list of [JMX MBeans entries](/3.0/user-guide/agents-configurations#jmx). #### Instrumentation Rules Context Menu ![Instrumentation Rules Context Menu](/files/-LzvOs-JE0XjHFZhaezO) | Menu Item | Description | | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | *Create Similar...* | This action allows for creating a new rule as a clone of the selected rule. It is enabled only for rules with formatters. | | *Formatters* | This submenu allows for creating either [Maven](https://maven.apache.org/) or [Gradle](https://gradle.org/) project template with all the formatters declarations. You can use it to implement formatters. | | *Remove Selected Rules...* | This action allows to remove the selected rules. | #### MBeans Context Menu ![MBeans Context Menu](/files/-LzvT_mA1Qod801UYtkS) | Menu Item | Description | | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | *Transform to Pattern...* | Transforms a selected MBean `ObjectName` by replacing it with a pattern specified in the [pop-up form](/3.0/user-guide/agents-configurations#transforming-mbean-objectname). | | *Clear Similar* | Reduces the number of MBeans by [transforming](/3.0/user-guide/agents-configurations#transforming-mbean-objectname) their `ObjectName`. | | *Remove Selected MBeans...* | Removes selected MBeans from the list. | | *Remove All MBeans...* | Removes all MBeans from the list. | #### Profile Actions There are four toolbar actions available in the draft mode: 1. This action will save the profile in `Manager`. 2. This action will activate the profile, which means it will save and deploy the profile to all the attached agents. 3. This action will create a new profile by cloning the current one. 4. This action will export the profile to an XML file, which can be later imported. ![Draft Configuration Toolbar](/files/-Lzg2WOkpGGAgYf1MjXZ) ### Active Profile View This view is basically read-only, i.e. you cannot edit any rules or change any settings. However, only in this view you can attach or detach agents. You can do it in the `Attached Agents` tab. ![Active Profile View](/files/-LwOkmULhH5ABXEoSVWY) #### Profile Actions There are two toolbar actions available in the active mode: 1. This action will create a new profile by cloning the current one. 2. This action will export the profile to an XML file, which can be later imported. ![Active Profile Toolbar](/files/-Lzg4XQhOkT7E6YJrXvQ) ## Instrumentation Rules The fundamental feature of `Agent` is instrumentation. This feature makes use of [java.lang.instrument](https://docs.oracle.com/javase/8/docs/api/java/lang/instrument/package-summary.html) framework. A user profile describes what and where to instrument. However, there are some classes which are always excluded from instrumentation: ```java com.flopsar.* org.objectweb.* javassist.* org.apache.bcel.* jdk.internal.org.objectweb.asm.* $* ``` {% hint style="info" %} All exception classes are ignored and not instrumented. {% endhint %} When the agent receives a profile to load, it first parses it and then sorts its instrumentation rules. If there are any changes in the profile that requires configuration reloading, the agent reloads those classes that have been instrumented earlier and those which are required by this new profile. This classes flow is passed through the profile filters. There are a few filters operating on two levels: *class* and *method* ones. The first filter excludes all the classes from packages, which match the internal exclusion set. Next, it goes through the second, user-defined, exclusion class filter. At this point, we have all the classes we want to instrument. In the next step, all the methods from each, selected class are passed through the method filters. At this point, a class can also be excluded from the profile if any of its methods do not satisfy the filters conditions. In the end, we obtain a set of methods that are selected for instrumentation. ![Classes and methods filtering](/files/-LyB57Be8VRA69BKQMhO) The methods instrumentation is performed in a specific order. First, rules with custom formatters are applied, then rules with standard formatters, and finally rules without formatters at the end. For example, if there are two method rules, one instrumenting with a custom formatter and the other one instrumenting only `public` methods and our method is both `public` and its signature matches the one specified in the first rule, then the rule with the formatter will be applied. ![Instrumentation Order](/files/-LqhKY6Z4J4huoyxhfnI) ### Rule View Each instrumentation rule is represented as a record in the rules list. ![Instrumentation Rule View](/files/-LzfVlLLx6OmFdOM23Y_) Each rule has a green or red banner. If the banner is green, the rule is inclusive. If it is red, the rule is exclusive. The text in the banner represents the type of the rule. The following types are available: * `CLASS` : the rule applies to classes. * `INTERFACE` : the rule applies to interfaces. * `ANNOTATION` : the rule applies to class annotations. * `SUPER CLASS` : the rule applies to super classes. * `EXCEPTION` : the rule applies to exception classes. * `METHOD` : the rule applies to method names. * `STANDARD` : the rule applies to methods with standard formatters. * `FORMATTER` : the rule applies to methods with custom formatters. Each instrumentation rule is either enabled or disabled. When a rule is disabled, it is ignored by agents. In order to disable the rule, you just need to toggle the green switch, to remove the rule you must click the trash icon at the end of the record. In order to edit the rule, just double click the record. All these operations are available only in the draft profile mode. ### Managing Rules There are several ways of creating new rules. You can either create a rule from scratch or clone a rule from another one. In order to clone a rule, right click the rule you want to clone and select `Create Similar...` item from the context menu. Note, you can clone only rules with formatters. In order to create a rule from scratch, use `+` buttons. If you want to remove rules, you can either click the trash icon on the rule record or select some rules and select `Remove Selected Rules...` from the context menu. ### Inclusive Rules Inclusive rules can be defined by clicking the green add button. There are several ways of specifying which class and method to instrument. ![Inclusive Rules Menu](/files/-Lx62Mcu04qZP9rXXnaa) #### Classes In order to add a class level rule, select `Classes` from the green button menu. You must specify which classes you want to instrument. If you want to match classes by their names, you should select the `Class name` radio button. If you want to instrument classes, which extend some super class, you should select `Super class name`. In this case, you need to specify the fully qualified super class name. If you want to instrument classes which implement some interface, you should select `Interface name`. You must then specify a fully qualified name of the interface. Finally, if you want to instrument classes, which are annotated by some annotation, you should select `Annotation name`. You must then specify a fully qualified name of the annotation. ![Classes Form](/files/-LyDrGYqlq47Mf_IMwm4) #### Classes from packages If you want to instrument classes from particular packages, just select `Classes from packages` from the green button menu. This operation will retrieve available packages from all the attached agents. You can add rules by double-clicking selected packages. ![Class from packages Form](/files/-LyDraE9mIQ_VZwp27f8) #### Method by access modifiers In order to add a method level rule, select `Method by access modifiers` from the green button menu. You can specify methods for instrumentation by their access modifiers. Just select the modifier so that those methods are instrumented, whose modifiers match one of the selected. ![Method by access modifiers Form](/files/-LyDrv2QQzv0Tp7b1DDJ) #### Method with exception You can also specify methods by exceptions they throw. Just select `Method with exception` from the green button menu and specify the exception class. ![Method with exception Form](/files/-LyDsrx27DDL_QniKd-T) #### Method with formatter If you want to instrument methods with formatters, you should select `Method with formatter` from the green button menu. There are some input fields that need to be filled. In the `CLASS NAME` field you can specify the exact class, your instrumented method should belong to. If you do not want to specify the class, then you should select `Any class` checkbox. {% hint style="info" %} Please note that any method, which matches the specified pattern, in any class will be instrumented with the formatter if you select `Any class` checkbox. {% endhint %} In the second field (`METHOD NAME AND SIGNATURE`), you must specify the exact method signature you want to instrument. You should ignore argument names and specify fully qualified class names of all objects. In the `OPTIONS` section you have two additional options to select. If you want to ignore the configuration threshold, select `Ignore threshold` checkbox. If you want to instrument a method on exit, select `Instrument on exit` checkbox. Now, if you want to report only parameter values without any preprocessing you can now click `Save` button and you have your rule defined. However, if you want to add some custom processing you should fill out `FORMATTER` section. First of all, you must specify the formatter signature. There are two ways of specifying the formatter signature. The first one is to specify it manually by entering the class and the method name of your formatter. The second one is used when you already have some implemented formatters. You just need to click `Browse` button and select a jar file with your formatters. The application will scan the jar and populate `IMPORTED FORMATTERS` combo box with all the found and valid formatters. Select one of the formatters, click `Save` button and you have your formatter rule created. ![Formatter Form](/files/-LzgBgtYvztJwHwNCx1q) ### Exclusive Rules Exclusive rules can be defined by clicking the red add button. There are several ways of specifying which class and method cannot be instrumented. ![Exclusive Rules Menu](/files/-Lx632TzjGBQxUw4YlU2) #### Classes In order to exclude some classes, you must specify the pattern for fully qualified class names. If you specify an asterisk at the end, the rule will match every class, whose fully qualified name starts with the specified pattern. Otherwise, the exact match will be checked. ![Classes Form](/files/-LyDvtNLiLpvWL-2m-GP) #### Methods If you want to exclude methods, you must specify a pattern for the methods name. The pattern should be a valid regular expression. ![Methods Form](/files/-LyDwLtQfChjESiW2uVL) #### Ignore Exceptions Normally, if an exception is thrown in an instrumented method, it is reported along with its stack trace. If you do not want to report it, you must define a rule for it by specifying a fully qualified class name of the exception. ![](/files/-LzCn_cZyW0GMAr0Edhh) #### Hot Methods This feature allows you to record instrumented methods executions. The Hot Methods feature is very useful, when you tune your profile for minimal performance impact. In this way, you can easily detect the most frequently executed methods and exclude them from instrumentation, unless you really need to instrument them. You can also exclude classes using this feature. You can use the feature only when you have some agents attached to your profile. In other words, agents are able to report their hot methods if and only if they instrumented these methods earlier. That means, you cannot use this feature when you create a new configuration and the profile has not been deployed yet. You must first deploy the profile, attach some agents to it, and then edit the profile again. Only then, this feature will be available to you. In order to retrieve hot methods, click `Collect` button in the top-left corner. This will trigger an operation of retrieving hot methods from all the attached agents. ![Hot Methods Form](/files/-LyDxbnmcOUlXc0eSEY0) All the retrieved methods will populate the table. The results are always grouped by classes, methods, classes & methods or agents. You can exclude classes or methods only from views grouped by either `classes` or `methods` respectively. Views grouped by `classes & methods` and `agents` are only for information purposes. The table column `DROPPED` shows how many of the calls were dropped due to the threshold value for methods duration. For `agents` view, the table has two additional columns: `CYCLES LEFT`: how many more times the calls statistics will be collected. When this value is zero, no more calls statistics will be collected. `COLLECTION TIME`: time when the statistics collection was initiated. You can reset the results anytime and start to collect new results by clicking `Reset` button. This will clear all the previous results and re-enable the Hot Methods feature on all the attached agents. ## JMX Rules Configuring JMX for agents relies on specifying `ObjectName` patterns and the data collecting frequency. In `jmx period` field you can specify how frequently agents must report data. In order to add some patterns, you must click the Manage MBeans menu. {% hint style="info" %} In order to add new `ObjectName` patterns, you must have some profile deployed and some agents attached. {% endhint %} Adding new patterns requires you to deploy some profile and attach some agents in the first place. MBeans can be imported only from running agents, so there must be some agents attached to the profile. ![Import MBeans](/files/-Ly4JANK0a3sZ0KItcJh) If the list is empty, click `COLLECT` button in the top-right corner. This operation will collect all the available ObjectNames from all the attached agents. In order to add a selected record, either double-click the record or use the list context menu. When you are done with importing ObjectNames, click `Close` button. Now, you should have the selected records added to the MBeans list. ### Transforming MBean `ObjectName` Usually, most of ObjectNames are specific to the servers they come from. Their names contain server-specific data such as IP addresses, ports, etc. That means there are ObjectNames that are available only on particular servers. In order to make them available on most of the servers and simplify the profile, we need to modify their ObjectNames. This can be achieved by clicking `Transform to Pattern` menu item in the list context menu. ![MBean Tranformation Form](/files/-Ly5AH3t85LD8uSwP51g) In order to illustrate the transformation procedure see the picture below. There are multiple ObjectNames, which are very similar and the only key they differ is the name. By transforming the `name` value to `*` for just one of these similar records, we can easily simplify the entire set of similar records into a single `ObjectName`. That is exactly how the transformation procedure works. ![MBeans Transformation](/files/-LzfP1l0mbvQ6-Ft2iIP) ## Profile Activation When your profile is ready to be activated, you should first save it by clicking the `Save` button. This operation will only persist the profile in `Manager`. During activation operation, all the attached agents are notified about it and the profile is uploaded to them. The agents will check whether there are any changes between their current profile and the new one. If agents detect instrumentation rules changes, they will reload their profile. The reloading operation duration can vary and it strongly depends on the profile and application itself. It can take from a few seconds to a minute. {% hint style="danger" %} Profile reloading is a heavy operation and will degrade your application performance temporarily. It can even halt the entire application for the time of reloading operation. Do not reload a profile when your application is under heavy load. {% endhint %} ### Attaching Agents When your profile is activated, you can attach agents to it. You can do it by clicking the green plus button in the `Attached Agents` tab. ![Attach Agents Tab](/files/-M2cmpmzIDEAhB3SO7pI) # Agents Status ## Agent Details The view is divided into panes with either text information or a graph on them. All the information in this view is reported by the agent. The reported data are collected since the agent started and reset every time the agent restarts. ![Agent Details](/files/-Lz6mcn5TgA2SMtTT0Qf) The following details are reported in the view. {% tabs %} {% tab title="AGENT" %} `Version` : version of the agent. `Connected From` : source socket address of the agent. It shows from which socket address the agent connects to manager. `Started At` : agent startup time. It shows when the agent started. `Uptime` : agent uptime time. It shows how long the agent is up. `Last update` : agent last report time. It shows how much time has passed since the last agent report received by `Workstation`. {% endtab %} {% tab title="CONFIGURATION" %} `Current` : current configuration. It shows the configuration the agent is attached to. `Deployed At` : agent configuration deployment time. It shows the last time the configuration has been deployed to the agent. {% endtab %} {% tab title="SETTINGS" %} `Database` : current database. It shows the database the agent sends its data to. `Hot Methods` : Hot Methods feature. It shows whether the feature is enabled or disabled. `CPU Time` : CPU time feature. It shows whether the CPU time measurement is enabled or disabled. {% endtab %} {% tab title="PACKETS TOTAL" %} `Sent` : Number of sent packets. Each method call, symbol or some metric value is represented as a single packet. These packets are sent to a database. This shows the total number of such packets sent to the database from the moment of the agent start. If the agent is attached to another database, this value will be reset. `Dropped` : Number of dropped packets. It shows the number of packets that have been dropped inside the agents. There can be a few reasons which make the agent to drop packets. See [Data Collecting Considerations](/3.0/administrator-guide/agent#data-collecting-considerations) for details. `Saturation` : Packets saturation. It shows the ratio of the number of received packets to the total number of packets. This total number is a sum of dropped and received packets. The value is displayed in percentages. If the value is 100% it means that all the packets, that have been collected on the agent, have been received by the database. If this value is low, it means that a large number of packets is dropped for some reason. `Size Sent` : Data size sent. It shows how much data has been sent to the database. This value is counted from the moment of the agent start. If the agent is attached to another database, this value will be reset. {% endtab %} {% endtabs %} Additionally, there are two graphs that present current number of instrumented methods and the current outbound size of data sent from the agent. ## Agent Logs In the `Agent Log` tab, you can control the current agent log level and monitor the number of logged messages with respect to their log level. The number is counted from the moment the agent starts. If you want to change the current log level, just click the level in the white box and select a new level. If you want to see recent agent logs, you can retrieve them from the agent by clicking the green download button. It will retrieve up to 1 MB of recent logs. ![Agent Logs](/files/-Lx2RO2pGc1abG47e1J5) You can change the logger level by selecting the level from the `current level` tile. ## JVM State This view presents CPU and memory usage graphs of the selected JVM. {% hint style="info" %} CPU usage graph is supported only for GNU/Linux platforms. {% endhint %} ![JVM State](/files/-Lx2Ss1Knkvj8KaT5yBm) ## JVM Properties In this view you can retrieve JVM system properties. ![Exemplary JVM System Properties View](/files/-Lx2TIqMVcksIMwZg9QD) ## JVM Packages This feature allows you to see what packages are available in your application. It is useful if you want to know what packages are available in your Java environment so that you can add some class rules to your configuration. The packages are reported up to the third level. ![](/files/-Lx2TRgS51Gt_RsaXd-h) ## JVM Class Loaders You can retrieve a list of class loaders detected by `Agent`. The result is presented in the form of a table. Each reported class loader can be in one of the four available states: * **`PENDING`** : to be instrumented, * **`NO NEED`** : has been checked and no instrumentation is needed, * **`SKIPPED`** : has not been checked and will not be instrumented, * **`INSTRUM`** : has been instrumented. ![](/files/-Lx2TbtkOxcSj9ptvpx0) # Data Browsing Workstation allows you to browse all the collected data. In order to browse the data, you need to fill out the form below. ![Data Browser Form](/files/-LwNxVsKh4CzzAs0jT2_) You must specify at least a pattern for agents and the time range. Additionally, you can search for methods by parameters. In `KEY` field, you must specify an exact key name for the parameters. In `VALUE` field, you must specify a phrase to search for. How the phrase will be matched depends on the option selected below the field. If you select `contains` option, only those methods will be returned, which parameters value contain the phrase. If you select `exact` option, only those methods will be returned, which parameters value equal to the phrase. Finally, if you select `regex` option, the phrase will be regarded as a regular expression and only those methods will be returned, which parameters value match the regular expression. By default, the parameters are searched only for root calls. If you want to search for parameters in every method call of the root call execution stack, then you need to select `DEEP SEARCH` option. ## Browse Result View When you click `SEARCH` button, you will get a similar view to the one below. ![Exemplary Browse Data Result](/files/-Ly97BVGVk3PhNe1A2Dz) The table is composed of the following columns: * `AGENT` : name and type of the agent, * `THREAD` : name of the thread the call was executed within, * `METHOD` : method name and signature, * `PARAMETERS` : method parameters preview, * `TIME` : time stamp of the end of the method execution, * `EXCS INSIDE` : number of exceptions thrown within the method execution tree, * `CPU TIME` : percentage of duration the method spent on CPU, * `DURATION` : method execution duration, * `CALLS` : total number of reported methods execution within the method execution tree. Whenever you click a record in the table, the details of the selected method are displayed on the right panel. The details cover: * `CLASS` : fully qualified class name the selected method is member of, * `METHOD` : method name and signature, * `TIME` : time stamp of the end of the method execution, * `DURATION`: method execution duration, * `CPU TIME` : time the method spent on CPU, You can see any [record details](/3.0/user-guide/execution-stack-interpretation) by double clicking it. ### Parameters List Additionally, when the selected method has parameters, they are displayed in the `PARAMETERS` list. The list records contain both `KEY` and `VALUE`. When you select a list record, its value is displayed in the text area below the list. The list has a context menu, which contains several actions you can perform on a selected parameter. ![Parameters List Context Menu](/files/-LzqkqMzF7foK68sqLNq) | Menu Item | Description | | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | *Search similar* | Use this action if you want to search for method calls, which have a similar value to the selected parameter. This action will bring up the Data Browser form, prefilled with the agent and parameters fields. | | *Reformat* | Use this action to format the selected parameter value in the text area view. You need to specify a separator, which will be used to break lines in the text area. | # Online Monitoring Flopsar allows you to observe your application data processing online. This feature is disabled by default and you need to enable it first. You can do this by selecting [`MONITORING`](/3.0/user-guide/online-monitoring#settings) tab in the Preferences window. ## Settings In order to enable the monitoring feature, you need to toggle on `Online monitoring enabled` switch. The online monitoring has a notification feature, which is turned on by default. The notification is an audible bell, which is fired whenever certain conditions are fulfilled. You can either use a default alarm sound or use your own. In order to use a custom alarm sound, click the `Browse...` button and select a file with the sound. You can then check your sound by clicking the speaker button. If you do not intend to use an audible alarm, you can turn it off by unselecting the `Enable` option. If you do so, only the visual notification will be used. ![Monitoring Settings](/files/-LwrxjnXrVQN15DdUrz_) You can define multiple online views by clicking green `+` button on the list above. If you want to open a view, you just need to double click the desired view record. ## Online View ![Online View](/files/-LwryCY2zjTmcFhc_afD) Basically, this view presents methods execution time for the last 10 minutes and it is updated every 3 seconds. The view is divided into four sections. The top section contains three curve graphs, which represents the following metrics: * `cyan graph` : total number of collected root calls, * `orange graph` : current number of threads, * `pink graph` : number of thrown exceptions, propagated to root calls. The other three stack area graphs represent the number of incomplete root calls. Each graph corresponds to a user-customizable duration level. The graphs are scaled with respect to their maximum value. There are three levels: * `green` : number of root calls, which duration is not greater than `LEVEL 0` value, * `yellow` : number of root calls, which duration is greater than `LEVEL 0` value and less than `LEVEL 1`, * `red` : number of root calls, which duration is greater than `LEVEL 1` value. {% hint style="info" %} Please note, at least one method call from a root call must end and be sent to the server so that Flopsar can report the root call current execution time. {% endhint %} You can customize each online view settings by clicking [`SETTINGS`](/3.0/user-guide/online-monitoring#settings-1) item in the context menu. ### Top Grid Bar ![Top Grid Bar](/files/-M3v-XVAwyLA3KbVUl2V) There is a grid bar at the top of the online view. Each grid cell has a color, which corresponds to one of the levels. The cell color depends on the level that has the biggest number of root calls. ### Context Menu The online view has a context menu, which provides several actions. ![Online View Context Menu](/files/-LzrXF3MPMbbC9H0K14A) | Menu Item | Description | | ------------------ | --------------------------------------------------------------------------------------------------------------------- | | *Details...* | This action will display a [detailed data view](/3.0/user-guide/online-monitoring#online-view-details) of each agent. | | *Settings...* | This action will bring up the online view settings pane. | | *Save image as...* | This action will export the online view to an image and save it to a selected file. | ### Online View Details This view presents a table of all currently monitored agents. ![Online View Details](/files/-LzrXegkdyVzoHW0_AE0) The table contains the following columns: * `AGENT` : agent name, * `THREADS COUNT` : total number of threads running in the corresponding JVM, * `LOAD` : current total number of methods calls, * `EXCEPTIONS LOAD` : current total number of exceptions, * `LEVEL COUNT` : total number of methods calls per `LEVEL`. ### Options In this view, you can change some online view settings. In order to change the window title, you need to update `WINDOW TITLE` text field. To change background and grid color, click the color picker to choose a color. You can also specify values for `LEVEL 0` and `LEVEL 1`. {% hint style="info" %} `LEVEL 0` value cannot be less than 3 seconds. {% endhint %} In `ALARM` section you can define two rules, which manage when the alarm should go off. If you disable the audible alarm in the Preferences window, the alarm will be only visual. ![Online Options](/files/-LycUOhO9liCLoUJH7s2) In order to delete the window, just click `DELETE` button. ### Agents In the AGENTS tab, you have a list of all available agents to monitor. In order to add an agent, just select a check box next to the agent. There is also an icon at the end of agents records. If you click the icon, you will get a [threads dump](/3.0/user-guide/online-monitoring#threads-dump) from the corresponding JVM. Note, the agent must be connected to your server to be able to retrieve the dump. ![Agents List](/files/-LzrWqtBnAu5PnfnUdsd) ## Threads Dump This query retrieves a threads dump from the selected agent. The result header contains the following metrics: * `STARTED` : total number of threads created and also started since the JVM started, * `PEAK` : peak live thread count since the JVM started, * `TOTAL` : total number of threads, * `DAEMON` : current number of live daemon threads. ![Exemplary Thread Dump](/files/-LyYV-OVaKPXEvVJJRu2) # Data Inspector This is a special view, which makes an analysis of your application performance much easier. In order to run it, you need to fill out the form below. Besides the methods execution analysis, you can also add some key-value metrics, which will be presented in the same resulting view. This is the fastest way to analyze your application since all you have to do is specify a pattern for agents and a time range. ## Search Form ![Data Inspector Form](/files/-LwI9uRvIw_q7A5SGS8k) ### KV Metrics This is a list of all stashed KV metrics, that will be used by the Data Inspector. You can access this list by clicking `KV METRICS` link on the search form. Each record in the list has a check box at the beginning which enables/disables the metric. If the metric is disabled, the Data Inspector result will not contain the metric graph. `Absolute` check box toggles between `Relative` and `Absolute` graph type (it has the same meaning as in [custom KV graphs](/3.0/user-guide/custom-key-value-metrics#absolute)). Finally, if you want to remove a metric, just click the trash icon. ![KV Metrics](/files/-Lzv17D_i6lT0TTpEbym) ## Analysis Result As a result of the data inspector, you will get a result similar to the one below. This is a list of previews of partial results. If you click the expand icon in the top-right corner of a preview, you will get a detailed view of the selected preview. ![Sample Analysis Result : part 1 of 2.](/files/-LyUlCewbCiUU53i1OxD) ![Sample Analysis Result : part 2 of 2.](/files/-LyUlFyDW1FxbTf-KACc) ### Index The index is composed of two graphs. The upper one presents exceptions density and the lower one presents mean off-cpu. ![](/files/-LyUzBAb2CMiht0RnhJ7) ### Calls Grid This view presents a grid of cells, which aggregates some specific values. In order to choose which values should be displayed, you need to select one of the options in `SELECTED VIEW` section. This can be [`Calls`](/3.0/user-guide/data-inspector#calls), [`Exceptions`](/3.0/user-guide/data-inspector#exceptions-1) or [`Off-CPU`](/3.0/user-guide/data-inspector#off-cpu). In the `Agents` section you get a list of all agents matching your query pattern along with their number of calls share. Each cell represents a collection of methods calls. Its width represents a time range, which is calculated as a quotient of the query total time range and the number of cells in a single row (which is 20). The cell height represents a duration range, which is specified in the `CELL HEIGHT` field. You can change this value (in milliseconds) by updating the field and clicking `Reload` button. There are two special rows in the grid, the bottom and top ones. The bottom row aggregates all methods calls, which duration is less than `CELL HEIGHT` value. For any next row, the duration is added to the previous row until the last one. The top row aggregates all the remaining methods calls. Whenever you move over a cell, its details are displayed below the agents' list. #### Calls Each cell contains a number in its center. This number value represents the number of methods calls aggregated in the cell. The cell background opacity is a qualitative representation of a ratio of the number of methods calls in the cell and the total number of methods calls. ![Calls Grid](/files/-M-K2lwnWhwo_rxHszIQ) #### Exceptions ![Exceptions Grid](/files/-M-K2FE30zJLyRx7gvge) #### Off-CPU ![Off-CPU Grid](/files/-M-K2JsUZz8_r5e8YYVB) ### Exceptions Exceptions view presents a list of exceptions thrown in all methods called satisfying the conditions specified in the [search form](/3.0/user-guide/data-inspector#search-form). However, these are not all exceptions thrown in the methods, it is only a subset of them. Suppose we have a method `A` execution stack similar to the one below. ![Sample Method Execution Stack](/files/-LzutAmIe6vEBWt7mfhm) Each rectangle represents a method call. If the background color of a rectangle is red, it means that an exception has been thrown. If the background is green, no exception has been thrown. In this particular example we can see there are four exceptions thrown in this stack in total, and all of them have been handled in the root method `A`. However, only two exceptions will be reported, the ones thrown in `B` and `F` methods. If you want to know how particular exception occurrences are distributed over time, just select the exception from the table and `Occurrences Time Histogram` item in the context menu. You will get a bar chart with the selected exception occurrences over time. You can click the bars to see more details. If you want to see where the selected exception come from, just select `Affected Calls` item from the context menu. ![](/files/-LyUldyutA7ymarBWyl-) # Custom Key-Value Metrics This view allows for browsing any key-value metric data collected by agents. In order to draw a graph for a selected metric, you must fill out the form below. ## Search Form First, you need to specify a pattern for agents, a time range, and a metric key. You can either specify the metric key manually or select one from the available metrics by clicking [`AVAILABLE KV METRICS`](/3.0/user-guide/custom-key-value-metrics#kv-metrics-tree) link. ![Key-Value Metric Form](/files/-LwNx5V_dyDaS_keX4Go) ### KV Metrics Tree This popup view contains a list of all available KV metrics in a form of a tree. You can always refresh the available metrics view by clicking `Refresh tree` button. ![Available Metrics](/files/-Ly3zEQ4UOeug9ydWNXP) ## KV Graph If you click `SEARCH` button, a graph will be drawn for the selected metric. ![Exemplary key-value metric](/files/-Ly461iYt4IM60cksQXx) If you want to preserve the graph, you can detach the window by clicking the icon in the top-right corner of the window. ### Context Menu Each KV graph has a context menu with several actions. ![](/files/-LzuxLMNH4YvIKOzcLgm) #### Agents This submenu allows for selecting from which agents data should be visible. #### Absolute This option allows us to toggle between `Absolute` and `Relative` type of graph. `Absolute` graph means that raw data points are displayed, while `Relative` graph means that data points are displayed as a difference between two consecutive values. *Use `Relative` option if, for example, your KV metric is an additive one, i.e. its value always grows and you want to observe how much it increases at each collection time.* #### Add to Inspector This action adds the [KV metric](/3.0/user-guide/data-inspector#kv-metrics) to the [Data Inspector](/3.0/user-guide/data-inspector). #### Save image as... This action exports the graph to an image and saves it to a selected file. # Method Execution Stack When you double click some selected record, you will get a window with the record details. The window presents a tree representing an execution stack of the selected method call. This tree view is self-explanatory but there can be situations where this view can be a bit confusing. This can happen when the displayed execution stack does not exactly correspond to the execution flow you see in your application source code. In order to understand why this situation takes place, you must understand how the [execution stack tree is built](/3.0/user-guide/execution-stack-interpretation#stack-intepretation). {% hint style="success" %} *Root Call* is the very first element in the tree. {% endhint %} Whenever you double click on any record of the [table of methods](/3.0/user-guide/data-browsing#browse-result-view), you will get the selected method execution tree. ![Exemplary Execution Method Stack](/files/-Ly9FJ6eKve-YNyri50T) The detailed view of a method execution tree has three tabs: * `STACK TREE` : tree view, * `STACK SPECTRUM` : list of all methods composing the tree, * `FLAME GRAPH` : [Flame Graph](http://www.brendangregg.com/flamegraphs.html) representation of the method execution tree. ## Stack Tree This view presents a method execution stack as an ordered tree of method calls. The tree table is composed of the following columns: * `COMPONENT` : method name and signature, * `EXCS INSIDE` : number of exceptions thrown within this method execution branch, * `CPU TIME` : percentage of duration the method spent on CPU, * `DURATION` : method execution duration. Each cell has a colored background, which represents the duration share with respect to the root element duration value, * `TIME` : time stamp of the end of the method execution, * `PARAMETERS` : method parameters preview. Whenever you click a record in the table, the details of the selected method are displayed on the right panel. The details cover: * `CLASS` : fully qualified class name the selected method is member of, * `METHOD` : method name and signature, * `TIME` : time stamp of the end of the method execution, * `DURATION`: method execution duration, * `CPU TIME` : time the method spent on CPU, Additionally, when the selected method has parameters, they are displayed in the [`PARAMETERS`](/3.0/user-guide/data-browsing#parameters-list) list. If you want to know the exact location of a method call within the execution stack, you can select the [`Elements Position`](/3.0/user-guide/execution-stack-interpretation#elements-position) from the [context menu](/3.0/user-guide/execution-stack-interpretation#context-menu) on the tree view. Next time, you expand a branch on the execution stack you should see the positions of methods. The position is determined by two numbers, the first one denotes the sequence number within the stack and the second one denotes the stack depth. ![Exemplary Execution Stack with Elements Position](/files/-Ly9G1wJxVpfPjXeka2i) ### Context Menu There are several actions you can perform on the stack tree view. The actions are available from the context menu. ![](/files/-M234VYWcGBaWsUIImeD) #### Get Class Bytes... This action allows for downloading a class file of the selected method so that you can decompile it in any decompiler of your choice. {% hint style="info" %} Note, the agent, where the class comes from, must be connected to your `Server` in order to download the class. {% endhint %} #### Profile This submenu allows to quickly add a new rule to the same profile, the corresponding agent is attached to. The submenu contains the following actions: * `Add with Formatter...` : creates a new rule with a formatter. The create rule form is prefilled with the method and class data, * `Exclude Class` : creates a new exclusion rule, which makes the class excluded from instrumentation, * `Exclude Method` : creates a new exclusion rule, which makes the method excluded from instrumentation. #### Tree View This submenu allows you to filter out methods from the tree. The submenu contains the following options: * `Complete` : no method is filtered out (default). A complete tree is presented, * `Critical` : filters out all methods but the one with the biggest duration value at any branch, * `Exceptions` : filters out methods, which do not have exceptions thrown within their branches. #### Find a Duration Gap Finds a method in the tree, which has the largest value gap between its duration and all of its methods it invokes directly. If `Ignore Leaf Calls` option is selected, this action ignores methods, which do not invoke any other methods. #### Expand All This action expands the entire tree. #### Elements Position This action makes the method elements position data visible. #### Export... Exports the entire tree to XML and saves it to a selected file. ## Stack Spectrum When you select the `STACK SPECTRUM` tab, you will get a spectrum view. This view presents a *spectrum* of the execution stack, in other words, it shows a collection of all the components which constitute the stack along with their calls and exceptions statistics. ![Execution Stack Spectrum](/files/-Ly9Gf3G8AqsXhaM8kWE) ### Spectrum Element Each record in the spectrum represents a unique method, i.e. the record identifies a method by its origin class and signature. If there are multiple calls of the same method in the tree, the number of times it is executed is presented on the right of the record entry. Each record has a colored background, which represents a ratio of a number of the corresponding method executions to the total number of all calls. Each record has also a banner with a number inside. This banner presents how many exceptions is thrown by the corresponding method. If you click on the record with a number in the banner, a popup window will appear with a list of all the exceptions thrown in the represented method. ![Exceptions List for a Spectrum Element](/files/-LzqeDL_Xdb5yTex4l8v) If there is more than one exception in the exceptions list, you can display a stack trace for any of the exceptions by selecting them from the `EXCEPTION CLASS` combo box. ### Context Menu There are several actions you can perform on the stack spectrum view. The actions are available from the context menu. ![Context Menu](/files/-M234ItrypFbBmSYmbRj) Both [`Get Class Bytes...`](/3.0/user-guide/execution-stack-interpretation#get-class-bytes) and [`Profile`](/3.0/user-guide/execution-stack-interpretation#profile) items have the same meaning as in the stack tree. ## Flame Graph When you select the `FLAME GRAPH` tab, you will get a view similar to the one below. ![Flame Graph](/files/-Ly9HEYvJ_xAuig1fxza) ## Stack Interpretation In the picture you can see four, sample execution stacks of some `A` method call. ![Execution Stack Assembly](/files/-Ly9HbvOvi6WCxIFp61N) The first `a` view shows how the real execution stack looks. This is a real execution flow that takes place in your application. The second `b` view, presents the same stack instrumented by your agent. Depending on your configuration, the instrumentation may not cover the entire stack. The empty, grey boxes represent missed, not instrumented methods. As you can see, at this very point your instrumented execution flow is not complete. Moreover, in your application runtime there can be situations when some of the instrumented method calls will not be reported (refer to [Data Collecting Considerations](/3.0/administrator-guide/agent#data-collecting-considerations) for details). The empty, red boxes in the third `c` view represent this situation. Finally, what you actually observe in workstation is the last `d` view. This view presents an assembled stack with missing calls ignored. Now you know how to interpret the execution stack in workstation and why the resulting stack can differ from the original one. # Agent API If you want to provide your own, non-JVM, data, and store them in the Flopsar environment, you can do it using the [Agent API](https://api.flopsar.com/agent/3.0/). This functionality is embedded in the Flopsar Agent. In order to make use of the API, you must add `flopsar-agent-X.Y.Z.jar` as a dependency to your project. # Agent Plugins An agent can be used standalone or it can be extended by means of Flopsar Extensions. Flopsar Agent is a powerful engine, which is extensible by plugins. The extensions are some extra jar files, which consist of user-defined POJO classes. The jars can be then included in the agent deployment. The plugins enable to extend `Agent` capabilities. We provide a mechanism that you can use to extend your application diagnosis and trace your data flow. It makes Flopsar an open tool, that can be extended. The plugin allows for processing data inside instrumented methods. There are two types of formatters, which can be used: `standard` and `custom`. ## Standard The `standard` formatter is embedded in `Agent`. This formatter will execute the following method on each argument it gets: ```java public static java.lang.String valueOf(java.lang.Object); ``` and return a result in a form of concatenated pairs of parameter identifiers and the corresponding `valueOf` function result. The following parameters are returned (in order of appearance): * `THIS` : If the instrumented method is static, this parameter is `null`, otherwise a class instance the method is executed on. * `ARGS` : List of all arguments of the instrumented method. * `RET` : Only if the instrumented method does not return void and the instrumentation is performed at the method exit. ## Custom If you are not satisfied with standard formatters and you want to extract some additional information (or make some preprocessing) you must implement your own, `custom` formatter. In order to take advantage of this feature, all you have to do is set up a configuration, create a simple POJO class and implement a single method (called *formatter*): ```java public static java.lang.String myFormatter(java.lang.Object[] args) ``` {% hint style="warning" %} You can give any name to the above method but the signature of the method must be preserved. Otherwise, it will not work. {% endhint %} The `args` argument of the formatter is an array of the instrumented method arguments delivered to your formatter. These are your original application objects, not clones. When you set up the configuration correctly, the agent will inject your formatter code into the instrumented methods during the instrumentation. It can be injected in one of the two places, i.e. at the beginning or the end of the method. Note, your formatter code is executed inside your instrumented method, so make sure your formatter code generates minimal overhead. The argument `args` is an array of the following objects (in order of appearance): * **`args[0]`** If the instrumented method is static this element is `null`, otherwise it is a reference to a class instance the instrumented method is executed on. * **`args[1..N]`** List of all arguments of the instrumented method in the same order as they appear in the method signature. These are the references to your original application objects, not some *clones*. * **`args[N+1]`** If the instrumented method returns void this element is `null`, otherwise it is a reference to the object instance this method returns. ***Please note, this argument is passed to the formatter if and only if the instrumentation is performed at the end of the method***. where `N` is the number of the instrumented method arguments. When you set up the configuration correctly, the agent will inject your formatter code into the instrumented methods during the instrumentation. It can be injected in one of the two places, i.e. at the beginning or the end of the method. Note, your formatter code is executed inside your instrumented method, so make sure your formatter code generates minimal overhead. {% hint style="warning" %} If you crash your application because of badly written formatters, do not blame us. Formatters code can be written by anyone, that is why they are not supported by Flopsar Technology. {% endhint %} In fact, it is up to you what extra features the agent will have. You can even extend your agent installation to cover a distributed transaction tracing. {% hint style="success" %} Use `U+001E` as a separator in your formatters. Otherwise, the result will not be formatted properly in `Workstation`. {% endhint %} Your result should always consist of pairs of keys and values. `U+001E` separates each entry in the result, which means you should always have an odd number of separators. The resulting string size cannot be greater than `8192 B`, otherwise it will be truncated. There are some tips, you should take into account when writing formatters: * Design and implement formatters with performance in mind. If your formatters code performs poorly, obviously the instrumented methods will perform poorly as well. * Use `try-catch` block to protect your application from any errors you can make in your formatter implementation. Otherwise, any errors inside the formatter can interfere with your application processing. * Do not modify arguments delivered to your formatter since they are your genuine application objects. Make sure that whatever you do with them will not cause any problem to your application flow. * Try not to use reflections, unless there is no other way. * Try not to create too many new objects, otherwise you risk increased GC overhead. * Deploy your formatter classes into separate jar files. This can save you a lot of trouble when dealing with class loading issues. ### Example Suppose, you have some method: ```java public FooBar foo(foo.bar.Object1, foo.bar.Object2, ...); ``` and you are interested in some additional information extracted from the first and second arguments. Your formatter implementation can look like the one below: ```java public static java.lang.String myFormatter(java.lang.Object[] args){ final char SEPARATOR = 0x1E; try { foo.bar.Object0 _this = (foo.bar.Object0)args[0]; foo.bar.Object1 obj1 = (foo.bar.Object1)args[1]; foo.bar.Object2 obj2 = (foo.bar.Object2)args[2]; /* implement your logic here to produce string output. */ String output1 = some_logic1(obj1); String output2 = some_logic2(obj2); return "Object1"+SEPARATOR+output1+SEPARATOR+"Object2"+SEPARATOR+output2; } catch(Throwable ex){ return "Error"+SEPARATOR+ex.getMessage(); } } ``` {% hint style="danger" %} No formatter implementation is supported by Flopsar Technology. {% endhint %} {% hint style="info" %} Please refer to our [GitHub repository](https://github.com/flopsar/formatters) for a sample implementation of plugins. {% endhint %} # Server Plugins Flopsar provides a few functionalities implemented as plugins. These implementations can be easily replaced with user-defined implementations. In this version of Flopsar only `Manager` supports plugins. There is one plugin currently available: *authentication*. Flopsar provides a C API for writing custom implementations of plugins. In order to write a plugin, you need to include in your project the main header file `plugin.h` , which can be downloaded from our [GitHub repository](https://github.com/flopsar/plugin-api). This file contains prototypes of functions to implement. The function below is the most important one and must be implemented. This function is called only once when the application loads the plugin. ```c int plugin_init(Plugin *p, const char *p_conf); ``` When you call the function above you will be provided with two arguments. The `p` argument is an opaque type representing your plugin. You should keep it since it is required when you register your plugin hooks. The `p_conf` argument is a path to your plugin configuration file. The second argument is provided only if you specify it during the plugin [installation](/3.0/administrator-guide/server#plugins). {% hint style="warning" %} Do not change the name of the function above. It must stay `plugin_init`, otherwise a plugin will not be loaded. {% endhint %} ## Authentication Plugin If you want to use your own authentication system, you must implement a plugin in order to be able to authenticate your users against this system. There is a default implementation of the authentication plugin available at [GitHub](https://github.com/flopsar/plugin-ldap). Use it as an example to implement your own. # Database API Flopsar provides [API](https://api.flopsar.com/fdbc/3.0/), which you can use to access your collected data. This API enables local access to the database. This implies you cannot access a database remotely. In order to make use of the API, you must add `flopsar-fdbc-VER.jar` as a dependency to your project. # Troubleshooting ## Manager stops periodically I see the following message in the `Manager` log file: ``` [FATAL ][PID] Flopsar cannot run any longer. No license control file found. Shutdown initiated. ``` `Manager` creates a license lock file in `/tmp` directory. If this file is deleted, then `Manager` stops. It occurs on Red Hat-like distributions usually. In order to prevent the operating system from deleting this file, you can create a file `/usr/lib/tmpfiles.d/flopsar.conf` with the following contents: ``` x /tmp/*.flp ``` {% hint style="info" %} Please, refer to `tmpfiles.d(5)` man page for details. {% endhint %} ## I see DROPPED in the Parameters column in `Workstation` This means that instrumented JVM drops parameters due to memory pressure. Either, increase memory limits for your application or decrease instrumentation. # Third-party Components A list of third-party components used by Flopsar software. | Component | Version | License | | --------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------ | | [ASM](https://asm.ow2.io/) | 7.3.1 | [3-Clause BSD License](https://asm.ow2.io/license.html) | | [LZ4](https://lz4.github.io/lz4/) | 1.9.2 | [License](https://github.com/lz4/lz4/blob/dev/lib/LICENSE) | | [JFoeniX](http://jfoenix.com/) | 8.0.9 | [Apache License Version 2.0](http://jfoenix.com/documentation.html#license) | | [Apache Commons Math](http://commons.apache.org/proper/commons-math/) | 3.6 | [Apache License Version 2.0](https://www.apache.org/licenses/LICENSE-2.0.html) | # Bug Reporting Whenever you want to submit a bug report, please attach the following information: * Affected Flopsar version * Description of the problem you are experiencing If this information is not sufficient, you will be asked to provide additional information, e.g. specific logs, screenshots, etc. # Introduction {% hint style="info" %} End of life for Flopsar 4.0. This product is no longer supported. {% endhint %} ## What is Flopsar® Flopsar is a *Fault Detection and Diagnosis* software for Java systems, developed by [Flopsar Technology](http://flopsar.com). Its main goal is to detect errors and problems in your Java systems and help to find the root cause quickly. What is the most important thing in systems maintenance? Systems maintenance relies on making the systems highly available and responsive. This implies that we must maintain the number of outages as low as possible and the time needed to resolve any issues must be as short as possible. However, if the worst happens and the system fails, besides the fact it failed we need to know why. Knowing why is not sufficient if finding the cause can take a whole day or even longer. These are the main problems that need to be addressed. Unlike other APM-like tools, Flopsar does not require users to predict what may go wrong in their systems. Moreover, we do not lose information about your applications' processing by probing or collecting only some vague averages and aggregated data. Flopsar is a highly customizable and extensible product. It is framework and application servers agnostic. Flopsar has more key features that help users to maintain their systems better. Its area of application is not only limited to systems maintenance, it is also used in applications development processes successfully. Its features can be extended by using plugins, which allow users to perform extra processing at runtime, with no application source code changes. # Versioning Every version label is composed of three numbers `X.Y.Z`: `X` Major version number: major changes, e.g. architecture change. `Y` Minor version number: minor changes, e.g. new features. `Z` Small version number: bugs fixes. {% hint style="info" %} Versions `X.Y` are equivalent to `X.Y.0` {% endhint %} Do not mix binaries from different releases, since they are not compatible. You can mix only those components, which have the same major and minor release numbers. {% hint style="danger" %} Incompatible components are unable to work together. Any connection between two incompatible components is dropped. {% endhint %} # Licensing You can run Flopsar without a license, but with some restrictions: * Only one agent can send its data to the server If you want to connect more agents, then you need to purchase the license. # Supported Platforms and Requirements ## Agent Agent library is distributed in two versions, for JVM 6-8 and 9-16. ## Workstation The workstation is a desktop application and is distributed as a Windows installation file and zipped application binaries. The Windows distribution unit does not require any additional runtime to install and run. The zipped binaries require the Java 11 runtime along with the JavaFX libraries. ## Server The server is distributed as a Docker image. # Distribution Packages Flopsar distribution includes: `Flopsar Server` - Flopsar Server Docker Image. `flopsar-agent-X.Y.Z.jar` - Flopsar Agent for JVM 8 and older. `flopsar-agent-9-X.Y.Z.jar` - Flopsar Agent for JVM 9 and newer. `flopsar-workstation-X.Y.Z.zip` - Flopsar Workstation application. `setup-flopsar-workstation-X.Y.Z.exe` - Flopsar Workstation install the application for Windows. `SHA256.asc` - PGP signed file of all binaries checksums. In order to verify the signature, use our [PGP public key](http://flopsar.com/pgp). {% hint style="success" %} All the files can be downloaded from our [release site](http://bin.flopsar.com). {% endhint %} ## Verifying Packages In order to verify `SHA256.asc` file, you must first import our public key. You can do this by executing the following command: ``` $ gpg --keyserver pgp.mit.edu --recv-keys 4E1CBB4012E01B7A ``` and verify the file itself: ``` $ gpg --verify SHA256.asc ``` If the file is valid, you should see the output: ``` gpg: Good signature from "Flopsar Technology (Flopsar Technology Secure Key) " ``` # Architecture Fundamentals Flopsar consists of three main components: * Agent * Server * Workstation ![Architecture](/files/-MUT51AaJkuylkA4Etpf) The agent is a jar file that is installed on JVM running the user application. Every agent belongs to some application. The agent is attached to the JVM and produces data. The data are sent to the server, which collects and serves the data to the workstation clients. ## Networking There are two TCP servers exposed by the server. The default ports used by the servers are: * **9000** - this server handles all the connections from agents. * **9443** - this server handles connections from the workstation clients. The connections to this server are encrypted. {% hint style="warning" %} If you change the ports, make sure your clients connect to their corresponding servers otherwise, the environment will not work. {% endhint %} In this networking architecture the following connections must be allowed: * outgoing connections from agents to the server, * outgoing connections from workstations to the server. ## # Installation ## Server ### Docker The server is shipped as a Docker image. To install the Flopsar server, you only need to execute the command or use `docker-compose.yml` file: {% tabs %} {% tab title="Command" %} ```bash $ docker run -d \ -p 9000:9000 \ -p 9443:9443 \ -e ACCEPT_EULA=y \ flopsar/server:4.0.0 flopsar start ``` {% endtab %} {% tab title="docker-compose.yml" %} ```yaml version: '3' services: flopsar-server: image: flopsar/server:4.0.0 command: flopsar start ports: - "9000:9000" - "9443:9443" environment: - ACCEPT_EULA=y ``` {% endtab %} {% endtabs %} This will create a new container and start the server. By default, the server listens on two ports: 9000 and 9443. The first port is used by agents and the second one by workstations. It is recommended to attach a volume for the server data. You can do this by executing the command: {% tabs %} {% tab title="Command" %} ```bash $ docker run -d \ -p 9000:9000 \ -p 9443:9443 \ -v :/opt/flopsar \ -e ACCEPT_EULA=y \ flopsar/server:4.0.0 flopsar start ``` {% endtab %} {% tab title="docker-compose.yml" %} ```yaml version: '3' services: flopsar-server: image: flopsar/server:4.0.0 command: flopsar start ports: - "9000:9000" - "9443:9443" environment: - ACCEPT_EULA=y volumes: - :/opt/flopsar ``` {% endtab %} {% endtabs %} By default, the server uses `/opt/flopsar` for its data directory. This can be changed by setting the `FLOPSAR_HOME` environment variable, e.g.: {% tabs %} {% tab title="Command" %} ``` $ docker run -d \ -p 9000:9000 \ -p 9443:9443 \ -v :/flopsar \ -e FLOPSAR_HOME=/flopsar \ -e ACCEPT_EULA=y \ flopsar/server:4.0.0 flopsar start ``` {% endtab %} {% tab title="docker-compose.yml" %} ```yaml version: '3' services: flopsar-server: image: flopsar/server:4.0.0 command: flopsar start ports: - "9000:9000" - "9443:9443" environment: - ACCEPT_EULA=y - FLOSPAR_HOME=/flopsar volumes: - :/flopsar ``` {% endtab %} {% endtabs %} ### Standalone The server component can also be installed manually, if Docker is not an option. In this case, first you must install the [Swift runtime](https://swift.org/download/). Obviously, you can install the server only on those platforms, that are supported by the Swift runtime. To install the server manually, follow the instructions below: * download the Swift runtime * install all the [dependencies required](https://swift.org/getting-started/) by the runtime * unpack the Swift binaries into the system root directory: `$ sudo tar xzvf -C / --strip=1` * copy the `flopsar` binary to your machine ## Agent To install the agent, follow carefully the instructions below: 1. copy the agent file `flopsar-agent-X.Y.Z.jar`to your server machine. 2. create a directory (we call this directory a Flopsar home directory) and `ext` subdirectory for the agent data. 3. [configure](/4.0/administrator-guide/agent#configuration) the agent. 4. optionally put all your agent plugins into the `ext` directory. 5. restart your application. When the agent starts for the first time it does not have any profile assigned. It just registers itself to the server after a successful connection, sends its status data, and does not instrument anything nor send any data. {% hint style="info" %} The agent must have a [license assigned](/4.0/managing-licenses) to send its basic data and a [profile](/4.0/profiles) to send the instrumented data. {% endhint %} ## Workstation If you are on a Windows platform, just download a Windows installation file and install the workstation. For other platforms, download the zip file with the workstation application. Since the application is distributed without Java runtime, you will need to install both [JRE 11 and JFX](https://www.azul.com/downloads/zulu-community/?version=java-11-lts\&package=jre-fx) before you start the workstation. ## Quick Start To quickly set up and start your environment follow the instructions below: 1. Download, [install](/4.0/administrator-guide/installation#server) and start the server 2. Download, install the agent, and start your application 3. Download, install and start the workstation 4. Log in to the workstation in [Admin Mode](/4.0/administrator-guide/server#admin-mode) and [register a license](/4.0/administrator-guide/workstation) 5. Log out and log in to the workstation again in normal mode 6. [Assign the license](/4.0/managing-licenses) to your applications 7. [Create a profile](/4.0/profiles#adding-profiles), add some rules, and [assign one of your applications](/4.0/profiles#assigning-applications) to the profile 8. Finally, [activate the profile](/4.0/profiles#profile-activation). # Agent ## Configuration Agents are configured by their runtime options. ### Agent Identifier Every agent must have a name. Since the agent is attached to a JVM, which runs an application instance, its name must include both the application and instance name, separated by a dot, i.e.:`myApp.myInstance` denotes that the agent is attached to the `myInstance` instance of the `myApp` application. If you do not follow the rules of the naming convention, the resulting agent identifier will take its default values: `generic.autoid`, where `autoid` is automatically generated unique identifier. {% hint style="info" %} It is recommended to give unique names to agents. If some of the agents have the same name, the data collected from these agents will be associated with this name and there will be no way to tell which data come from which agent. In other words, you will lose the information about the source of the collected data. {% endhint %} ### Mandatory Settings {% hint style="info" %} All agents' options are Java system properties and should be used as such (added with `-D` prefix). {% endhint %} | Option | Description | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `flopsar.id` | [Identifier](/4.0/administrator-guide/agent#agent-identifier) of a JVM instance. It should consist of only word characters and have no more than 62 characters. If you provide a name consisting of more characters, the agent will truncate the name. | | `flopsar.manager` | Socket address of the Flopsar TCP server. Agents initiate connections to this server and the connections are maintained until the agents are shutdown. It should be specified in the form: **host:port**. | | `flopsar.home` | Path to the Flopsar working directory. Log files are stored in this directory. | {% hint style="info" %} The Flopsar working directory must contain the`ext` subdirectory. This is the place you put all your plugin classes. {% endhint %} The above options must be set otherwise, the agent will not work. For example, your configuration could look like this one: ``` -javaagent:/path/to/flopsar-agent-*.jar \ -Dflopsar.id=myApplication.instance23 \ -Dflopsar.manager=172.16.0.1:9000 \ -Dflopsar.home=/path/to/some/dir ``` ### Optional Settings There are some additional options, which are not mandatory. | Option | Description | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `flopsar.ext.packages` | Comma-separated list of packages, which all formatter classes come from. If not specified or empty, it contains only `org.flopsar.ext` package by default. Classes from these packages will be loaded by the agent from the `ext` subdirectory. No class coming from any of the specified packages will be instrumented since the agent assumes these are plugin classes. | | `flopsar.cl` | Comma-separated list of class loader classes, which should be instrumented. You can use this option if you want to instrument some class loader, which is not automatically instrumented by the agent. This can be useful if the agent cannot load some classes by already instrumented class loaders. This option is empty by default. | | `flopsar.errline` | If you want the agent to skip line numbers (in stack traces) in the instrumented methods, set this option value to **false**. The default value is **true**. | | `flopsar.logsize` | Total maximum size in MB of log files. By default, this value is set to 100. By setting this option you specify how much space log files can use. | | `flopsar.qsize` | Total maximum size of the agent outbound queue size in MB. By setting this option you specify how much memory the agent can use for the queue. | | `flopsar.qratio` | Ratio of agent outbound queue size to the maximum JVM memory size in percentages. By default, this value is set to 20. By setting this option you specify how much memory the agent can use. If you specify both this option and `flopsar.qsize`, the latter takes precedence. | | `flopsar.debug` | This option enables/disables additional logging for class loading. If you have problems during instrumentation regarding class loading, you can enable this feature to track, which class loader loads classes. The default value is **false**. | | `flopsar.loglevel` | This option sets the initial logger level. By default, this option is set to **fine**. | {% hint style="warning" %} Do not put your plugin classes into packages, which have the same root package as your instrumented application, e.g. if all your application classes belong to `com.foo.bar.*` package, make sure your plugin classes do not. {% endhint %} For example, your extended configuration could look like this one: ``` -javaagent:/path/to/flopsar-agent-VER.jar \ -Dflopsar.id=MyAgent.instance23 \ -Dflopsar.manager=172.16.0.1:9000 \ -Dflopsar.home=/path/to/some/dir \ -Dflopsar.logsize=200 \ -Dflopsar.qsize=400 \ -Dflopsar.errline=true ``` There are various places where you can add your custom Java properties, it depends on the Java software you use. Please, refer to your Java software documentation for more details about adding extra Java properties. {% hint style="info" %} If you plan to install the agent in OSGi environments, make sure to add `com.flopsar` classes to the OSGi container classpath. Please, refer to your OSGi environment documentation for details. {% endhint %} ### Disabling Agent If you want to disable the agent temporarily, without deleting all the settings, you can do it by setting a `FLOSPAR_ENABLE=NO` environment variable and restart the application. ## Logging By default, agents always start with **fine** logger level and after 2 minutes it is switched to **info** automatically. You can change this behavior using `flopsar.loglevel` option. Log files are stored in a directory specified in `flopsar.home` option. The total log files' size cannot exceed the maximum size, which can be specified in `flopsar.logsize` option. There can be maximum five log files, which are rolled over when their size reaches the limit. Older files are overwritten by newer ones, so the total size of logs is kept under control. ## Data Collecting Considerations Although the agent is designed to collect as much data as possible, we are not able to guarantee that all the instrumented method calls data will be collected. It is because the agent priority is not to crash the application it runs on. That has some serious implications on the agent design and operation. We just favor stability and reliability over data completeness. Each call of an instrumented method, symbol, or metric value is represented as a serialized entity called *packet*. As soon as the packet is created it is ready to be sent to a database. In order for the packet to be sent, it must be first put in the outbound queue. This is the point where the agent decides whether put it or not. The packet is put in the queue if and only if the connection to the database is established and there is enough space in the outbound queue (the size of the outbound queue can be customized). Otherwise, the packet is dropped. This scenario holds for each packet. # Server ## Admin Mode The server is one of the key Flopsar components, which is responsible for managing the entire Flopsar environment. It is managed from the workstation application. You must be logged in admin mode, to manage the server. You need to select the **Admin mode** option in the login window. ![Admin Login Window](/files/-MTWRyjbySiaiJaO3Vqz) {% hint style="info" %} Only users with admin privileges can log in using **Admin mode**. {% endhint %} When you log in, you will see the server summary view. ![Server Summary](/files/-MToXJ4K6vNaLGyzv8nP) This view contains basic information about the server, you are connected to. The information contains: * **VERSION** server version * **BUILD** server build * **DATA SIZE** total size of data stored in the database * **VOLUME USAGE** current usage of the volume the database files are stored in * **STARTED AT** what time the server started at {% hint style="warning" %} Please note, if the volume usage is too high, the server will be shutdown automatically. There must be at least 1GB of free space for the server to run. {% endhint %} The license information covers: * **EXPIRES** license expiration date * **REGISTERED TO** licensee the license is issued for * **MAX AGENTS** maximum number of agents this license allows * **ID** license identifier ## Server Settings In this view you can edit the server settings. Any changes will not be effective until the server is restarted. ![Server Settings](/files/-Mb_Te6Ggn_KkWOUamrh) The following settings are available: * address at which the server listens for connections from agents, * address at which the server listens for connections from workstations, * both the server private key and the certificate (this pair is auto-generated when the server is installed), * archive settings ### Data Archiving In order to enable the data archiving you need to set the archive policy first. There are three possible choices: * **none**: the archiving is disabled, * **drop**: the old data will be dropped, * **archive**: the old data will be archived. If the archive policy is not none, than you need to set the time at which the archiving procedure will be checked if it should start or not. There are two conditions that decide if the archive procedure should start. The conditions are: size of the data or the data retention time. If one of the condition is satisfied, then the procedure starts. # Registering License {% hint style="info" %} You can manage the license only in the workstation application in the [admin mode](/4.0/administrator-guide/server#admin-mode). {% endhint %} To register a new license, use the options menu and click the **Register New License** menu item. You can either paste the contents of your license file or drag and drop the license file. ![Registering New License](/files/-MTolkRMMnrjLuBGRInW) If your license is valid, you can register it by clicking the **Register** button. {% hint style="warning" %} Changing a license affects your current license mapping. You should revise your [license mapping](/4.0/managing-licenses) and adjust it accordingly. {% endhint %} # Managing Users {% hint style="info" %} You can manage users only in the workstation application in the [admin mode](/4.0/administrator-guide/server#admin-mode). {% endhint %} ## Users List If a user has admin privileges, then its name has a crown icon. ![Users List](/files/-MTWwCk7XnJAPihoeUtc) To see user details you need to double click the selected user. To delete a user you need to click the options menu on the corresponding user record and click the **Delete** menu item. {% hint style="info" %} You cannot delete your own account. There must exist at least one user with admin privileges, so you need to use another account with admin privileges to delete your account. {% endhint %} To add a new user, you need to click the main options menu and click the **New User** menu item. ![](/files/-MTX2F269INaW0cm3aD3) Next, you must set a username and a password for your new user. You can click the **GENERATE PASSWORD** link to generate a random password for the user. When you click the **Save** button, the user will be created. ## User Details ![](/files/-MTX3LfoRHQ13kQt4dDx) Each user has the following attributes: * username * password * [permissions](/4.0/administrator-guide/managing-users#permissions) * list of [banned applications](/4.0/administrator-guide/managing-users#banning-applications) ### Changing User's Password To change user's password you need to click the **Change Password** menu item from the option menu. ![Change Password Form](/files/-MTvTk3ik6lZ5B-aYOby) Next, set a new password (or generate random by clicking the **GENERATE PASSWORD** link) and click the **Save** button. ## Permissions There are two types of users: administrator and regular users with selected permissions. The following permissions are available: | Permission | Description | | ------------------------- | ------------------------------------------------------------ | | Administrator | Users can edit agents' configurations. | | Manage profiles | Users can create, modify, and delete applications' profiles. | | View parameters | Users can see agents' online data. | | View online data | Users can query agents directly. | | Query agents | Users can send queries directly to agents. | | Manage licenses | Users can attach and detach licenses to/from agents. | | Edit online alerts | Users can edit online alerts' settings. | | Delete alert events | Users can delete online alerts' events. | | Manage exceptions mapping | Users can create, modify, and delete exceptions' aliases. | To edit user's permission, click the **Edit Permissions** menu item in the option menu. ![Edit Permissions](/files/-MTvg2YUdknQKOwyTetL) ## Banning Applications If you do not want a user to see data from particular applications, you can do this by banning the applications for that specific user. # Logging Each Flopsar module (application) uses logging levels. The levels are defined as follows: **`finest`** : most detailed logging. It should not be used during normal, production operations since it can degrade the application performance significantly. Use only in cases when you suspect some problems with the application, want to trace how it processes requests, requested by the Flopsar Technology Support to help investigate some problems or any other problems. It prints lots of diagnostic information, which might not be covered by this manual. **`finer`** : less detailed logging than `finest`. Used to log some additional information about the application operation and data processing. **`fine/debug`** : more detailed logging than `info`. Used to log some additional information about the application operation and data processing. **`info`** : default level used in applications. Used only to log some runtime information to verify a configuration and the like. You should use this level on normal, production operation. **`warn`** : used in cases when some error occurs but with no impact on the application stability and data integrity. **`error`** : normally, there should be no error messages in the log. The level is used in cases when some error occurs. These errors impact the application operation stability and data integrity but not all of them. The messages are informative, so you should know whether you can ignore them or not. Any system calls and external libraries errors are also logged in this level. **`fatal/severe/critical`** : there should be no such messages in the log. Messages of this level are only printed in cases when some fatal error occurred and the application cannot run any longer. For example, some misconfigured options can result in printing fatal messages in the log. Please, read them and fix your configuration. The messages are usually very informative. # Workstation ## Accessing Flopsar Environment When you start the workstation, a login window should appear. ![Login Window](/files/-MTV8CZJXq99zAwPb96V) You must fill out the login form to log in. The required fields are: * **CONNECTION ID** User-defined connection identifier. * **SERVER ADDRESS** Server socket address in the form `host:port`. If you use a default value `9443` for the port, then you can specify only an IP address in this field. The port will be added automatically. * **USERNAME** A username. * **PASSWORD** User's password. If you run the workstation for the first time, the **CONNECTION ID** combo box is empty. {% hint style="success" %} Default credentials are: the username is **admin** and the password is **flopsar**. {% endhint %} ## Settings The workstation stores its settings locally, on the machine it runs on. It makes use of [Java Preferences API](http://docs.oracle.com/javase/7/docs/technotes/guides/preferences/) to store its configuration. The physical location of the settings depends on the platform the workstation runs on. When you exit the application, it stores its current settings. ### General ![General Settings](/files/-MTWTsjTwUtizJ-PcwyA) By default, all the workstation logs are stored in a user home directory. This can be changed in the preferences window above, where you can set both the logger level and the location of the logs. You can also check if there is a new Flopsar version available. In order to check it, just press the **Check now** button. If you select **Check automatically**, then whenever you start the workstation, it will check for updates automatically. ### Connections Your server connection information is also stored in the settings. You can either edit or delete these connections in the preferences window below. ![Connections List](/files/-MTWU_-1gEHvlmJqXSdl) ### Permissions This view presents a list of all granted [permissions](/4.0/administrator-guide/workstation#permissions). ![Granted Permissions](/files/-MToop77ZF2n0xx5WU5X) # Data Inspector This is a special view, which makes an analysis of your application performance much easier. In order to run it, you need to fill out [the form](/4.0/data-inspector#search-form) below. Besides the methods execution analysis, you can also add some key-value metrics, which will be presented in the same resulting view. This is the fastest way to analyze your application since all you have to do is select the application and specify the time range. ## Search Form ![Search Form](/files/-MU8zVhH5lX9WcuR8DdH) ### Key-Value Metrics If you want to add to your search some key-value metrics, just click the **AVAILABLE KV METRICS** link and select the metrics from the tree. ![Sample Key-Value Metrics](/files/-MU9FdWLGL9xF3P9AP53) All the selected metrics are stored locally, in your workstation. You can access these metrics by clicking the **KV METRICS** link. ![Sample Selected Key-Value Metrics](/files/-MU9Hrej2-LvcA3pKS9V) You can manage your metrics in the view above. ## Results As a result of the data inspector, you will get a result similar to the one below. ### Overview ![Sample Results Overview : Duration](/files/-MU8v2dWBwmhmFdG5iZE) ![Sample Results Overview : Excaptions](/files/-MU8vANlhhqKntY8cR0h) ### Grid This view presents a grid of cells, which aggregates some specific values. In order to choose which values should be displayed, you need to select one of the options in the **View** combo box. By default, it presents the entry points view. Each cell represents a collection of methods' calls. Its width represents a time range, which is calculated as a quotient of the query total time range and the number of cells in a single row (which is 20). The cell height represents a duration range, which is specified in the **CELL HEIGHT** field. You can change this value (in milliseconds) by updating the field and clicking the reload button. There are two special rows in the grid: the bottom and top ones. The bottom row aggregates all methods' calls, which duration is less than the **CELL HEIGHT** value. For any next row, the duration is added to the previous row until the last one. The top row aggregates all the remaining methods' calls. #### Entry Points Each cell contains a number in its center. This number value represents the number of methods' calls aggregated in the cell. The cell background opacity is a qualitative representation of a ratio of the number of methods calls in the cell and the total number of methods calls. ![Sample Grid : Entry Points](/files/-MU8xDwwy2HPAOqz-sh-) #### On CPU ![Sample Grid : On CPU](/files/-MU8yNBjbfQ4-1xOuER3) #### Off CPU ![Sample Grid : Off CPU](/files/-MU8yUX1CD8_PXJaPUG8) ### Exceptions This view presents all the exceptions thrown in the methods, which satisfy your search filter. The exceptions are presented in a form of a list. Each record has two values, which both denote the number of exceptions thrown. The number with a red, flash icon represents how many exceptions have been thrown at the entry point level. The number with a gray, flash icon represents the total number of the thrown exceptions. ![Sample Exceptions](/files/-MU8ywniYD3G0alVkOrD) Each record has an action menu, which can be used to further investigate the exceptions. There are three actions you can use to analyze the time occurrences of the selected exception, entry points that have thrown the exception, and unique stack traces of the exception. #### Occurrences Time Histogram If you click the **Occurrences Time Histogram** menu item, you will get a histogram of the selected exception. The histogram will be displayed on the right side of the exception list. #### Exception Stack Traces If you click the **Exception Stack Traces** menu item, you will get a list of unique stack traces of the selected exception. ![Sample Exception Stack Trace](/files/-MU9LuHnO42Ldkzd0SYQ) # Database Browser The workstation allows you to browse all the collected data. In order to browse the data, you need to fill out the form below. ![](/files/-MTuMeC87ylMUyYkql-R) For quick browsing, you only need to select the application and the time range. Optionally, you can select the **Advanced search** option and fill out the additional form. The Parameters section allows narrowing your search by specifying two additional fields. In the **KEY** field, you must specify an exact key name for the parameters. In the **VALUE** field, you must specify a phrase to search for. How the phrase will be matched depends on the option selected below the field. If you select the **contains** option, only those methods will be returned, which parameters' value contains the phrase. If you select the **exact** option, only those methods will be returned, which parameters value equal to the phrase. If you select the **starts with** option, only those methods will be returned, which parameters value starts with the phrase. Finally, if you select the **regex** option, the phrase will be regarded as a regular expression and only those methods will be returned, which parameters value match the regular expression. By default, the search is performed only through entry points. If you want to search through every invocation, you need to select the **Deep search** option. {% hint style="info" %} Although the search can be performed in each invocation, the result will contain only [entry points](/4.0/method-execution-tree#terminology). {% endhint %} ## Search Results When you click the **Search** button, you will get a similar view to the one below. ![Sample Browsing Result](/files/-MTuVNb9toScZ_yvzypV) The search results are displayed in a form of a list. When you click an entry point from the list, and the entry point has parameters, they will be displayed in detail in the right list view. If you want to see the [execution tree](/4.0/database-browser#execution-tree) of the selected entry point, you need to double-click the corresponding record. ### Result Record Each record in the result list represents an [entry point](/4.0/method-execution-tree#terminology). ![](/files/-MTv639tQd2paScxxCvW) Each record contains the following information: * method name along with its signature * fully qualified class name the method comes from * thread name the method was executed in * method execution time (duration) * CPU time usage of the method execution * date the method was executed * time the method was executed at * invocations count of the corresponding execution tree * exceptions count in the corresponding execution tree #### Execution Tree When you double click some selected record, you will get a view with the record details. The view shows a tree representing an execution stack of the selected method call. This tree view is self-explanatory but there can be situations where this view can be a bit confusing. This can happen when the displayed execution stack does not exactly correspond to the execution flow you see in your application source code. In order to understand why this situation takes place, you must understand how the [execution stack tree is built](/4.0/method-execution-tree). ![Execution Tree](/files/-Mb_estShdD79cWQbsp3) Each element in the tree denotes some method invocation. The elements contain a method signature and its owner class, and parameters if there are any. If the selected method has paramaters, they are displayed in the list on the right. Each element has its own options menu with the following items: * **Get Class Bytes** : this action allows for downloading a class file of the selected method so that you can decompile it in any decompiler of your choice. * **Add Rule with Formatter to Profile** : creates a new rule with a formatter. The create rule form is prefilled with the method and class data. * **Exclude Method from Profile** : creates a new exclusion rule, which makes the method excluded from instrumentation. * **Exclude Class from Profile** : creates a new exclusion rule, which makes the class excluded from instrumentation. There are three available views of the execution tree. The view is changed by selecting one of the **VIEW** options: * **complete** : no method is filtered out (default). A complete tree is presented, * **critical** : filters out all methods but the one with the biggest duration value at any branch, * **exceptions** : filters out methods, which do not have exceptions thrown within their branches. There is a global options menu for the execution tree, which contains the following options: * **Find a Duration Gap** : finds a method in the tree, which has the largest value gap between its duration and all of its methods it invokes directly. * **Expand All** : expands the entire tree. * **Switch Positions Visibility** : makes the method elements position data visible. * **Export to File** : exports the entire tree to a file. #### Spectrum When you select the **Spectrum** tab, you will get a spectrum view. This view presents a *spectrum* of the execution tree, in other words, it shows a collection of all the components which constitute the tree along with their calls and exceptions statistics. ![](/files/-Mb_kzkdTZ-aJtTpU5fZ) Each record in the spectrum represents a unique method, i.e. the record identifies a method by its origin class and signature. If there are multiple calls of the same method in the tree, the number of times it is executed is presented on the right of the record entry. Each record has a colored background, which represents a ratio of a number of the corresponding method executions to the total number of all calls. Each record has also a banner with a number inside. This banner presents how many exceptions is thrown by the corresponding method. If you click on the record with a number in the banner, a popup window will appear with a list of all the exceptions thrown in the represented method. ![Exceptions Stack Trace](/files/-MbaBlPTV3GhDhrdkGX-) #### Flame Graph ![](/files/-Mb_lZaZnTUvO2xoBQOQ) # Metrics This view presents a collection of available key-value metrics, classes, and methods collected from the entire environment. ![Sample Metrics](/files/-MUd_TvvNtMtpU-wQW6z) ## Methods To make a graph of some method calls, click the **Graph** menu item for the selected method. Next, specify all the required fields and click the **Ok** button. ![Making a Method Call Graph](/files/-Mb__O8Y1ydj3yhXEJw8) You can add some conditions on the parameters by selecting the **Advanced search** check box and specifying both the parameter key and the pattern for its value. The parameter key entry must be an exact name of the key and the parameter value entry can be one of the four types. If you specify these additional conditions, the resulting graph will contain only those method calls, whose parameters satisfy the conditions. In either case, you should get the view similar to the one below. ![Method Call Graph](/files/-Mbb9lzhpCxBasLnE_qy) # Agents The **Agents** view contains a table with agents that are connected currently. ![Connected Agents](/files/-MU54Ju_THILaUn6K64x) The following information is presented in the table: | Column | Description | | ----------------- | ----------------------------------------------------------------------------------------------------------------- | | **APPLICATION** | Application the agent belongs to. | | **NAME** | Name of the agent. If the agent has no license assigned, the triangle icon is displayed otherwise, the green one. | | **PROFILE** | Profile the agent is assigned to. | | **UPTIME** | Agent's uptime. | | **SATURATION** | Ratio of the number of received packets to the total number of packets. | | **INST. METHODS** | Number of instrumented methods. | | **DTR** | Data Transfer Rate from the agent. | | **ERRORS** | Number of errors in the agent logs. | To see the [agent details](/4.0/agents#agent-details), click the link in the **NAME** column. ## Agent Actions Each agent record has the option menu, which allows executing queries on the agent directly. {% hint style="info" %} You must have the [Query Agents](/4.0/administrator-guide/managing-users#permissions) permission to see this option menu. {% endhint %} To execute a query, click the corresponding menu item in the option menu. ### JVM Properties This query retrieves all Java properties from JVM the agent runs on. ![Sample JVM Properties](/files/-MU5AZR6WRs1DGl0b-4O) ### Class Loaders This query retrieves a list of all detected class loaders. ![Sample Class Loaders ](/files/-MU5B8bbxPgqPsFncP4a) The result is presented in the form of a table. Each reported class loader can be in one of the four states: * **PENDING** : to be instrumented, * **NO NEED** : has been checked and no instrumentation is needed, * **SKIPPED** : will not be instrumented, * **INSTRUM** : has been instrumented. ### Packages This query allows you to see what packages are available in your application. It is useful if you want to know what packages are available in your Java environment so that you can add some class rules to your profile. The packages are reported up to the third level. ![Sample Packages](/files/-MU5D09eDluleEqTsyr6) ### Threads Dump This query retrieves a threads dump from the agent. The result header contains the following metrics: * `STARTED` : total number of threads created and also started since the JVM started, * `PEAK` : peak live thread count since the JVM started, * `TOTAL` : total number of threads, * `DAEMON` : current number of live daemon threads. ![Sample Threads Dump](/files/-MU5DozyyB5UkQOOHXV5) ### Agent's Log This query retrieves recent agent's logs. It will download up to 1 MB of recent logs. ![Sample Agent's Logs](/files/-MU5ELF_g-IF-embZdSS) ## Agent Details This view presents detailed information on the selected agent. ![Agent Details View](/files/-MUns907E4wKiNRapW2f) | Information | Description | | --------------- | ------------------------------------------------------------------------------------------------------------------ | | **application** | Application the agent belongs to. | | **name** | Name of the agent. | | **version** | Version of the agent. | | **platform** | Platform the agent is running on. | | **started at** | Time the agent started at. | | **uptime** | Agent uptime time. It shows how long the agent is up. | | **last update** | Agent last report time. It shows how much time has passed since the last agent report received by the workstation. | # Profiles A profile is a set of rules, which govern agents. It tells agents what data should be collected and how. Each profile needs to be activated to be eligible for deployment to agents. Which profile goes to which agent is determined by assigning applications to the profiles. In other words, the agents are not assigned to profiles directly. ## Profiles List In this view, you can see all your profiles. ![Profiles](/files/-MUIwMvlEpbA2o0emGJ5) Each profile can be either *active* or *inactive*. When the profile is active, there is a green icon next to the profile name indicating this fact. The **APPLICATION** column indicates which application is assigned to the profile. The number, next to the application name, denotes how many agents have the profile currently loaded. Each profile record has an actions menu, which is used to perform profile-related actions. ### Profile Actions To perform an action for the selected profile, click the actions menu and select the corresponding menu item. #### Changing Profile's Name To change the profile name, click the actions menu of the profile record and select the **Change Name** menu item. #### Assigning Applications If a profile is about to be used, it requires an application to be assigned to it. {% hint style="info" %} You can only assign applications to active profiles. The same application cannot be assigned to two different active profiles. {% endhint %} ## Adding Profiles There are two ways of adding profiles to the list: creating a new profile or importing the profile from a file. To create a new profile, click the main options menu and select the **New Profile** menu item. Next, specify a name for the profile, optionally select the application you want to assign to the new profile, and click the **Save** button. This will create an empty profile and add it to the list. You can also add profiles by importing them from files. To import the profile, you can either drag the profile file and drop it on the list or click the main options menu and select the **Import Profile from File** menu item. {% hint style="warning" %} Please note, a newly created profiles are not persisted in the server by default. You will need to save it manually in the profile editor otherwise when you restart the workstation, the profile will be gone. {% endhint %} ## Profile Editor A profile editor is a place where you can edit your profiles. You can access the editor by clicking the profile actions menu and selecting the **Edit** menu item. When you access the editor of an empty profile, you will get the view below. ![Profile Editor](/files/-MUsUE7_piheT8brbZJp) Each profile is composed of rules and options. A profile consists of two parts: instrumentation and JMX. In the instrumentation part, you define your instrumentation rules, and in the JMX part, you define [MBeans](https://docs.oracle.com/javase/tutorial/jmx/mbeans/) you want the data to be collected from. ### Instrumentation Rules To instrument methods, you need to decide which methods should be instrumented. This is governed by a set of rules. There are two types of rules: inclusions and exclusions. You use inclusion rules to specify what should be instrumented and exclusion rules to specify what mustn't be instrumented. There are two levels of rules: class and method rules. If you want to define a rule, that defines which classes should be instrumented, then you must use a class level rule. If you want to define a rule, that defines which methods should be instrumented, then you must use a method level rule. #### Class Rule Inclusion To create a new class rule, you must click the **Include** button and select the **Classes** menu item. Next, the corresponding form will pop up. ![Class Inclusion Form](/files/-MUJd7ZpZSMGjHh9e1zu) The form contains a text field, in which you must specify a pattern. The pattern value depends on the rule type you select in the form. There are four types at your disposal. * If you want to define a rule, which selects classes by their name, select the **Class name** button and specify a fully qualified name of the class. You can also specify a package along with the asterisk at the package name, instead. In this case, all classes from this packaged will be selected for instrumentation. * If you want to define a rule, which selects classes by the superclass they extend, select the **Super class name** button and specify a fully qualified name of the superclass, you want the classes to extend from. * If you want to define a rule, which selects classes by the interface they implement, select the **Interface name** button and specify a fully qualified name of the interface. * If you want to define a rule, which selects classes by their annotations, select the **Annotation name** button and specify a fully qualified name of the annotation the classes must be annotated with. #### Methods Access Modifiers Inclusion To create a new method rule, which selects methods by their access modifiers, click the **Include** button and select the **Method by Access Modifiers** menu item. ![Method Access Modifiers Form](/files/-MUJlyno5R5-Vg1stCxX) If you want to select methods that have package-level access, you need to select the **default** option. #### Methods with Exceptions Inclusion To create a new method level rule, which selects methods by exceptions they declare to throw, click the **Include** button and select the **Methods with Exceptions** menu item. ![Method with Exception Form](/files/-MUKAQOeCz2L09PcYCDC) If you want to select methods, that throw a specific exception, you need to enter a fully qualified class name of the exception. #### Methods with Formatters Inclusion If you want to instrument a specific method and inject some code to format the method data, click the **Include** button and select the **Methods with Formatters** menu item. ![Methods with Formatters Form](/files/-MUKBwokfdn0WDF0F-At) First of all, you need to specify, in the **CLASS** field, a fully qualified name of the class the method, you want to instrument, belongs to. Next, you must specify the name and signature of the method in the **METHOD NAME AND SIGNATURE** field. Each class name used in the method signature must be entered as a fully qualified name. The return type of the method must be specified at the beginning of the method name. The method arguments must be specified without labels. Next, if you want every invocation of the method to ignore the threshold, select the **Ignore threshold** option. This option will make your method always report, regardless of its duration. By default, methods are instrumented at the beginning of their body. If you want to instrument the method at the end of its body, select the **Instrument on exit** option. There are two types of formatters you can use. If you select the **Standard** formatter, then you do not need to specify any formatter, since the so-called [standard formatter](/4.0/developer-guide/agent-plugins#standard) will be used. If you want to use your own formatter, click the **Custom** option. If you have a jar file with your formatters, you can click the **Browse** button and select the jar file. This operation will scan the jar file, extract any available formatters, and populate the **IMPORTED FORMATTERS** combo box with the formatters signatures. Next, you can simply click the combo box, select the formatter, and both the **CLASS** and **METHOD NAME** fields will be filled out automatically. You can also specify the formatter class and method names manually. #### Class Rule Exclusion To create a class exclusion rule, you must click the **Exclude** button and select the **Classes** menu item. Next, the corresponding form will pop up. ![Class Exclusion Form](/files/-MUKWnaHE4xlQoFT-2wV) In order to exclude some classes, you must specify the pattern for fully qualified class names. If you specify an asterisk at the end, the rule will match every class, whose fully qualified name starts with the specified pattern. Otherwise, the exact match will be checked. #### Method Rule Exclusion If you want to exclude methods, you must specify a pattern for the method's name. The pattern should be a regular expression. ![Method Exclusion Form](/files/-MUKXv-1H79nroGzHapz) #### Exception Report Exclusion Normally, if an exception is thrown in an instrumented method, it is reported along with its stack trace. If you do not want to report it, you must define a rule for it by specifying a fully qualified class name of the exception. ![Exception Ignore Form](/files/-MUKYT86FHRWawwbcU3M) #### Hot Methods Exclusion This feature allows you to record instrumented method executions. The Hot Methods feature is very useful, when you tune your profile for minimal performance impact. In this way, you can easily detect the most frequently executed methods and exclude them from instrumentation, unless you really need to instrument them. You can also exclude classes using this feature. You can use the feature only when you have some agents attached to your profile. In other words, agents are able to report their hot methods if and only if they instrumented these methods earlier. That means, you cannot use this feature when you create a new profile and it has not been deployed yet. You must first deploy the profile and then edit the profile again. Only then, this feature will be available to you. To exclude some methods, using this feature, click the **Exclude** button and select the **Hot Methods** menu item. ![Hot Methods](/files/-MUcT5q58mHMVUJ0pXD2) {% hint style="info" %} If the Hot Methods table is empty, click the **Collect** button in the top-left corner. This will trigger an operation of retrieving hot methods from all the attached agents. {% endhint %} The collected hot methods are presented in a table. The table contents depend on the selected **Group By** item. When the **class** item is selected, the table presents a list of classes, when the method is selected, the table presents a list of methods, and when the **class & method** is selected, the table presents a list of methods along with their classes. The **CALLS COUNT** column contains the number of invocations of the corresponding record. The **DROPPED** column denotes how many percent of invocations have been dropped due to the duration threshold. If this value is high, then it is advised to exclude the corresponding method or class from your profile to reduce the performance impact of the instrumentation. The hot methods feature is always reset every time a new profile is loaded. The statistics are collected for about 5 minutes and then the feature is disabled. If you want to reenable it, just click the **Reset** button. ### MBeans This view enables specifying which MBeans values you want to collect. To add MBeans, you need to retrieve them from agents first. You can do this by clicking the **Collect MBeans** button. This will open a popup window with a list of available MBeans. If the list is empty, click the **Collect** button. This will query all the agents, that have the profile deployed. {% hint style="info" %} Please note, MBeans can be queried not earlier than two minutes after the agent start. {% endhint %} ![Sample MBeans](/files/-MUdXNRG9O5Mb7zLWWuv) ## Profile Activation When the profile is saved, it can be activated. Profile activation is a procedure, which enables deploying the profile to agents. An application must be assigned to the profile prior to the activation. When a profile is activated, it is deployed to agents that belong to the applications assigned to the profile and have the license. When the agent receives the profile, it checks whether there are any changes in the profile which require reloading. When the profile is reloaded, the hot methods feature is reset also. ## Active Profile An active profile's view can be accessed by clicking their names in the [profiles list](/4.0/profiles#profiles-list). In this view, you can see the profile details. ![Sample Active Profile](/files/-MUJX1kvGuHf6j-c1QY5) ## Instrumentation Rules Mechanics The fundamental feature of the agent is instrumentation. This feature makes use of [java.lang.instrument](https://docs.oracle.com/javase/8/docs/api/java/lang/instrument/package-summary.html) framework. A user profile describes what and where to instrument. However, there are some classes that are always excluded from instrumentation: ```java com.flopsar.* org.objectweb.* javassist.* org.apache.bcel.* jdk.internal.org.objectweb.asm.* $* ``` When the agent receives a profile to load, it first parses it and then sorts its instrumentation rules. If there are any changes in the profile that requires configuration reloading, the agent reloads those classes that have been instrumented earlier and those which are required by this new profile. When the profile is being loaded the stream of classes passes through the profile filters. The filters operate on two levels: *class* and *method* ones. The first filter excludes all the classes from packages, which match the internal exclusion set. Next, it goes through the second, user-defined, exclusion class filter. At this point, we have all the classes we want to instrument. In the next step, all the methods from each, selected class are passed through the method filters. At this point, a class can also be excluded from the profile if any of its methods do not satisfy the filters' conditions. In the end, we obtain a set of methods that are selected for instrumentation. ![Filtering Mechanism](/files/-MUIkpNDYRDzEjsR4z2u) The methods instrumentation is performed in a specific order. First, rules with custom formatters are applied, then rules with standard formatters, and finally rules without formatters at the end. For example, if there are two method rules, one instrumenting with a custom formatter and the other one instrumenting only `public` methods and our method is both `public` and its signature matches the one specified in the first rule, then the rule with the formatter will be applied. ![Instrumentation Order](/files/-MUInBygcvARa6QBNqrW) # Symptom Driven Diagnostics Besides historical data analysis, Flopsar provides symptom driven diagnostics feature. To enable this feature, you need to subscribe to the online data service first by clicking the main options menu and selecting the **Settings** menu item. ![](/files/-MYiFq_lB_pI1HDHN3Wd) When you subscribe to the service, you will be able to see online data. {% hint style="warning" %} You will need the [View online data](/4.0/administrator-guide/managing-users#permissions) permission to be able to see the online view. {% endhint %} ## Application Tiles ![](/files/-MZC6kDxa8MVd-Ur7dgh) Each tile in the view represents some application. The application name is displayed at the top of the tile. The tile consists of four vertical bars and a central double graph. Tiles can be added to the view from the list of applications on the right side. If you want to add an application, you need to select the application in the list, drag and drop it on the tiles view. Tiles can be removed from the view by clicking the tile options menu and selecting the **Remove** menu item. The central graph is composed of two graphs: *threads* and *load*. The **Thread** graph presents the total number of threads currently used in the application. The **Load** graph presents the total number of currently collected entry points. The **EXCEPTIONS** bar presents information about the exceptions' density in entry points. In other words, it shows what percent of the collected entry points have thrown exceptions. The value in the center of the bar denotes the current number of entry points with exceptions. The three-level bars present how many [incomplete entry points](/4.0/method-execution-tree#terminology) are currently processed in the application. The levels are duration-based, which means every incomplete entry point goes to one of the levels based on its current duration. The levels duration thresholds are user-defined. To edit the levels, click the options menu in the selected application tile and select the **Settings** menu item. ![](/files/-MZC71u9-C_8QKyV9MKc) In the levels edit form above, you can set the baseline thresholds for two levels. You need to specify the maximum time current [incomplete entry points](/4.0/method-execution-tree#terminology) can take to stay in the corresponding level. In other words, if some entry point is still being executed, and its execution time so far is less than the first level threshold, then it will go to the first level. Each refresh time, this entry point's execution time is checked and if its duration exceeds the level's threshold it is moved to the next level. Finally, the entry point either completes or goes to the last level and stays there until it completes. Every entry point, whose execution time exceeds the second level threshold will go to the third level automatically, so there is no need to set the third level threshold. The values in the center of bars denote the number of incomplete entry points for each bar. {% hint style="info" %} The online data is refreshed every 15 seconds. {% endhint %} ## Symptoms Besides the visual representation of currently executing methods, there is the fault detection feature, which provides visual and audible notifications when a possible fault is detected. The are three sensitivity levels of the symptom trigger: **low**, **medium** and **high**. If you want to disable the feature, you must select the **none** option in the **Symptom Trigger Sensitivity** combo box. Every fault is composed of events. The number of events depends on the selected fault sensitivity. There are two types of generated events: the duration based and the exception based. To view the fault list, you need to click the **Symptom Log** menu item in the option menu. ![](/files/-MZC7pfLbGnZMOKxcIAg) The Symptom Log contains a collection of all symptoms. There is a blue time axis with blue circles at the top. Each circle represents a symptom. To see the symptom events click one of the circles. # Managing Licenses Agents require a license to send their data to the server. Without the license, they cannot be profiled and send any data except their status. They must have the license assigned in the first place and this is performed manually. Despite the fact that agents require the license, you do not assign it to them directly. The license is assigned to applications instead. Every agent belongs to some application, so when you assign the license to an application, its agents will get it. {% hint style="info" %} You must have the [Manage licenses](/4.0/administrator-guide/workstation#permissions) permission to see this view. {% endhint %} ![Managing License](/files/-MTtd7u49n3oHMOTyQW1) The view above shows the maximum number of available licenses (MAX AGENTS) and the list of all applications. To assign licenses to the applications, you must edit the value in the corresponding rows. When you are done, just click the Save button to deploy your mapping. {% hint style="warning" %} If there are more agents, belonging to the same application, than licenses assigned to the application, not all of them will get the license. The agents will be selected randomly. {% endhint %} It will take a few seconds to deploy your license mapping across the applications. All agents whose license has been revoked will unload their profiles and stop sending their data. # Method Execution Tree ## Terminology Throughout the entire documentation, we use some terminology regarding the execution of methods. The picture below presents a single method execution tree. The first element of the tree is called ***Entry Point***. All the methods' calls, down the execution chain, are called ***Invocations***. The entire structure is called ***Execution Tree***. ![Execution Tree](/files/-MTuSePEN8ePQFkw5zIT) An entire execution tree is completed and built on the server. When some invocation comes to the server, and the corresponding execution tree, it is part of, cannot be found, a new tree is created. The entry point of this tree is then called ***Incomplete Entry Point***. When the entry point is completed, then its execution tree is stored in the database. ## Execution Tree Structure The picture below depicts the procedure of how the execution tree is built. In the picture below you can see four, sample execution trees of some method *A* call. ![](/files/-Ly9HbvOvi6WCxIFp61N) The first `a` view shows how the real execution stack looks. This is a real execution flow that takes place in your application. The second `b` view presents the same stack instrumented by your agent. Depending on your configuration, the instrumentation may not cover the entire stack. The empty, grey boxes represent missed, not instrumented methods. As you can see, at this very point your instrumented execution flow is not complete. Moreover, in your application runtime, there can be situations when some of the instrumented method calls will not be reported. The empty, red boxes in the third `c` view represent this situation. Finally, what you actually observe in the workstation is the last `d` view. This view presents an assembled stack with missing calls ignored. Now you know how to interpret the execution stack in the workstation and why the resulting stack can differ from the original one. # Agent Plugins An agent can be used standalone or it can be extended by means of Flopsar Extensions. Flopsar Agent is a powerful engine, which is extensible by plugins. The extensions are some extra jar files, which consist of user-defined POJO classes. The jars can be then included in the agent deployment. The plugins enable to extend the agent capabilities. We provide a mechanism that you can use to extend your application diagnosis and trace your data flow. It makes Flopsar an open tool, that can be extended. The plugin allows for processing data inside instrumented methods. There are two types of formatters, which can be used: **standard** and **custom**. ## Standard The **standard** formatter is embedded in the agent. This formatter will execute the following method on each argument it gets: ```java public static java.lang.String valueOf(java.lang.Object); ``` and return a result in a form of concatenated pairs of parameter identifiers and the corresponding `valueOf` function result. The following parameters are returned (in order of appearance): * `THIS` : If the instrumented method is static, this parameter is `null`, otherwise a class instance the method is executed on. * `ARGS` : List of all arguments of the instrumented method. * `RET` : Only if the instrumented method does not return void and the instrumentation is performed at the method exit. ## Custom If you are not satisfied with standard formatters and you want to extract some additional information (or make some preprocessing) you must implement your own, `custom` formatter. In order to take advantage of this feature, all you have to do is set up a configuration, create a simple POJO class and implement a single method (called *formatter*): ```java public static java.lang.String myFormatter(java.lang.Object[] args) ``` {% hint style="warning" %} You can give any name to the above method but the signature of the method must be preserved. Otherwise, it will not work. {% endhint %} The `args` argument of the formatter is an array of the instrumented method arguments delivered to your formatter. These are your original application objects, not clones. When you set up the configuration correctly, the agent will inject your formatter code into the instrumented methods during the instrumentation. It can be injected in one of the two places, i.e. at the beginning or the end of the method. Note, your formatter code is executed inside your instrumented method, so make sure your formatter code generates minimal overhead. The argument `args` is an array of the following objects (in order of appearance): * **`args[0]`** If the instrumented method is static this element is `null`, otherwise it is a reference to a class instance the instrumented method is executed on. * **`args[1..N]`** List of all arguments of the instrumented method in the same order as they appear in the method signature. These are the references to your original application objects, not some *clones*. * **`args[N+1]`** If the instrumented method returns void this element is `null`, otherwise it is a reference to the object instance this method returns. ***Please note, this argument is passed to the formatter if and only if the instrumentation is performed at the end of the method***. where **N** is the number of the instrumented method arguments. When you set up the configuration correctly, the agent will inject your formatter code into the instrumented methods during the instrumentation. It can be injected in one of the two places, i.e. at the beginning or the end of the method. Note, your formatter code is executed inside your instrumented method, so make sure your formatter code generates minimal overhead. {% hint style="warning" %} If you crash your application because of badly written formatters, do not blame us. Formatters code can be written by anyone, that is why they are not supported by Flopsar Technology. {% endhint %} In fact, it is up to you what extra features the agent will have. You can even extend your agent installation to cover a distributed transaction tracing. {% hint style="success" %} Use `U+001E` as a separator in your formatters. Otherwise, the result will not be formatted properly in the workstation. {% endhint %} Your result should always consist of pairs of keys and values. `U+001E` separates each entry in the result, which means you should always have an odd number of separators. The resulting string size cannot be greater than 8192 B, otherwise it will be truncated. There are some tips, you should take into account when writing formatters: * Design and implement formatters with performance in mind. If your formatters code performs poorly, obviously the instrumented methods will perform poorly as well. * Use `try-catch` block to protect your application from any errors you can make in your formatter implementation. Otherwise, any errors inside the formatter can interfere with your application processing. * Do not modify arguments delivered to your formatter since they are your genuine application objects. Make sure that whatever you do with them will not cause any problem to your application flow. * Try not to use reflections, unless there is no other way. * Try not to create too many new objects, otherwise you risk increased GC overhead. * Deploy your formatter classes into separate jar files. This can save you a lot of trouble when dealing with class loading issues. ### Example Suppose, you have some method: ```java public FooBar foo(foo.bar.Object1, foo.bar.Object2, ...); ``` and you are interested in some additional information extracted from the first and second arguments. Your formatter implementation can look like the one below: ```java public static java.lang.String myFormatter(java.lang.Object[] args){ final char SEPARATOR = 0x1E; try { foo.bar.Object0 _this = (foo.bar.Object0)args[0]; foo.bar.Object1 obj1 = (foo.bar.Object1)args[1]; foo.bar.Object2 obj2 = (foo.bar.Object2)args[2]; /* implement your logic here to produce string output. */ String output1 = some_logic1(obj1); String output2 = some_logic2(obj2); return "Object1"+SEPARATOR+output1+SEPARATOR+"Object2"+SEPARATOR+output2; } catch(Throwable ex){ return "Error"+SEPARATOR+ex.getMessage(); } } ``` {% hint style="danger" %} No formatter implementation is supported by Flopsar Technology. {% endhint %} {% hint style="info" %} Please refer to our [GitHub repository](https://github.com/flopsar/formatters) for a sample implementation of plugins. {% endhint %} # FAQ ## Can I use Flopsar when my license expires? Yes, you can. If your license expires, the server will unload profiles from all agents but one. This one agent is selected randomly. Your entire Flopsar environment will continue to run but with [limitations](/4.0/overview/licensing). ## # Third-party Components A list of third-party components used by Flopsar software. | Component | License | | ----------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------: | | ​[ASM](https://asm.ow2.io/)​ | [3-Clause BSD License](https://asm.ow2.io/license.html)​ | | [LZ4](https://lz4.github.io/lz4/)​ | [License](https://github.com/lz4/lz4/blob/dev/lib/LICENSE)​ | | [JFoeniX](http://jfoenix.com/)​ | [Apache License Version 2.0](http://jfoenix.com/documentation.html#license)​ | | [Apache Commons](https://commons.apache.org/) | [Apache License Version 2.0](https://www.apache.org/licenses/LICENSE-2.0.html)​ | | [Netty](https://netty.io/) | [Apache License Version 2.0](https://github.com/netty/netty/blob/4.1/LICENSE.txt) | | [JGraphT](https://jgrapht.org/) | [Eclipse Public License 2.0](https://github.com/jgrapht/jgrapht/blob/master/license-EPL.txt) | | [Swift Crypto](https://github.com/apple/swift-crypto.git) | [Apache License Version 2.0](https://github.com/apple/swift-crypto/blob/main/LICENSE.txt) | | [Swift NIO](https://github.com/apple/swift-nio.git) | [Apache License Version 2.0](https://github.com/apple/swift-nio/blob/main/LICENSE.txt) | | [Swift NIO SSL](https://github.com/apple/swift-nio-ssl.git) | [Apache License Version 2.0](https://github.com/apple/swift-nio-ssl/blob/main/LICENSE.txt) | | [SwiftLog](https://github.com/apple/swift-log.git) | [Apache License Version 2.0](https://github.com/apple/swift-log/blob/main/LICENSE.txt) | | [Swift Argument Parser](https://github.com/apple/swift-argument-parser) | [Apache License Version 2.0](https://github.com/apple/swift-argument-parser/blob/main/LICENSE.txt) | | [PromiseKit](https://github.com/mxcl/PromiseKit) | [MIT](https://github.com/mxcl/PromiseKit/blob/master/LICENSE) | | [RxSwift](https://github.com/ReactiveX/RxSwift.git) | [MIT](https://github.com/ReactiveX/RxSwift/blob/main/LICENSE.md) | # What is Flopsar? {% hint style="info" %} End of life for Flopsar 5.0. This product is no longer supported. {% endhint %} Flopsar, developed by Flopsar Technology, is the cutting-edge Fault Detection and Diagnosis software for JVM systems. Our mission is simple: to detect errors and problems in your Java systems swiftly and assist in identifying the root cause effectively. In the world of systems maintenance, availability and responsiveness are paramount. At Flopsar, we understand that minimizing system outages and swiftly resolving issues are critical factors. When a system fails, it's not enough to know that it failed—we must know why. Gone are the days of spending hours or even days searching for the cause. We tackle these challenges head-on. Unlike traditional APM tools, Flopsar eliminates the need for users to predict potential system issues in advance. We don't settle for vague averages or aggregated data that leave you guessing about your application's processing. Flopsar provides you with a highly customizable and extensible product that adapts to your unique needs. It seamlessly integrates with any framework and application server, making it truly agnostic. But Flopsar offers much more than that. Our powerful features not only enhance systems maintenance but also drive successful application development processes. With Flopsar, you can extend its capabilities using plugins, allowing you to perform additional processing at runtime without making any changes to your application source code. # Supported Platforms and Requirements What platforms and architectures are supported. ## Agent Agent is a native library, supporting the following platforms:
Platformx86_64ARM
GNU/Linuxtruefalse
{% hint style="warning" %} Agent library requires `libc` in version at least 2.18. {% endhint %} ### Supported JVM Agent supports JVM version from 1.7 through 17. ## Server The server application is distributed for multiple platforms:
Platformx86_64ARM
GNU/Linuxtruefalse
## Workstation This is a web application, that can be deployed on any web server that supports web sockets and is able to run as a reverse proxy. # Architecture Fundamentals Flopsar consists of three components: * Agent * Server * Workstation Workstation, as a web application, can be served from either the server directly or any web server, which is able to operate as a reverse proxy.

Workstation bundled with the server

Workstation served from a reverse proxy

## Networking Agents initiate bidirectional TCP connections to the server, which listens to these connections on a socket address specified in the server configuration. # Agent Agent is a native library, which is attached to JVMs. It runs on a JVM and its main goal is to collect data from applications running on the JVM and the JVM itself. Each agent is attached to a JVM. It initiates a single bidirectional TCP connection to the Flopsar server. Agent is started by adding the following option to a `java` command: 
-agentpath:<path-to-agent>=<agent-options>
For example, if the option `-agentpath:/opt/flopsar/libflopsar.so=opt1,opt2` is specified, the agent will be loaded from the `/opt/flopsar/libflopsar.so` file and the configuration options `opt1,opt2` will be passed to it. {% hint style="info" %} There are various places where you can add your custom JVM options, it depends on the Java software you use. Please, refer to your Java software documentation for more details about adding extra JVM options. {% endhint %} ## Configuration Agent configuration is specified by a set of options. There are two types of settings: mandatory and optional. ### Mandatory Options Each option must be specified in a form: `key=value`, where `key` is an option name. They must be specified along with the agent library itself. * **`app`** Application name, the agent is assigned to, * **`server`** Socket address of the Flopsar server. Agents initiate connections to this server and the connections are maintained until the agents are shutdown. It should be specified in the form `IPv4:port`, * **`home`** Path to the agent working directory. This directory contains the agent extensions and logs. {% tabs %} {% tab title="Example" %} The following option 
-agentpath:/opt/agent/libflopsar.so=app=MyApplication,server=192.168.10.11:9000,home=/opt/flopsar
will be evaluated as follows: * the agent will be loaded from the `/opt/agent/libflopsar.so` file * the agent will be attached to *MyApplication* * the agent will attempt to connect to the Flopsar server at `192.168.10.11:9000` * the agent will store its logs to `/opt/flopsar` and load its extension from `/opt/flopsar/ext` directory. {% endtab %} {% endtabs %} ### Additional Options Each option must be specified in a form: `-Dflopsar.key=value`, where `key` is one of the following values:
Option keyDefault valueDescription
nameAuto-generated and reset every time the agent startsAgent instance name.
extorg.flopsar.extA package name, which all extensions classes come from. Classes from these packages will be loaded by the agent from the ext sub-directory in the agent working directory. Note, that no class coming from the specified package will be instrumented since the agent assumes these are extension classes.
clex-A list of class loaders, that should be excluded from instrumentation. The list entries should be fully qualified class names separated by the + sign.
bcpfalseA feature, which changes the way how the com.flopsar.Flopsar class is loaded into the Bootstrap ClassLoader. If set to true, the agent will extract a flopsar.jar file into the agent home directory and add it to the bootstrap classloader path. If either absent or set to false, the agent will load the com.flopsar.Flopsar class into the Bootstrap ClassLoader internally. This option is usually used if you get exceptions like java.lang.NoClassDefFoundError: com/flopsar/Flopsar
qsize10 MiBMaximum size of the agent outbound queue size. By setting this option you specify how much memory the agent can use for the queue. The value should be specified along with the unit, e.g. 30mib, 20m, 50M, 30MiB, 4gib etc.
log.size50 MiBMaximal size a single logger file can reach. When the value is reached, the logger file is rolled.
log.cnt5Maximal number of rolled logger files.
log.debugfalseSets the logger level to DEBUG.
[^1]: The absolute path from which to load the agent library. [^2]: Comma-separated agent configuration options. # Server Server is one of the key Flopsar components, which is responsible for managing the entire Flopsar environment. It is a single, binary file `flopsar-server`. ## CLI The Flopsar server has a few command line options. All the available options of the server can be listed by executing the command: ```shell-session $ flopsar-server --help ``` ## Configuration All the server settings are configured in a file `settings.yml`. This file contains the following default settings: 
eula: true
data: /path/to/data
agent:
  server_ip: 0.0.0.0
  server_port: 9000
ui:
  server_ip: 0.0.0.0
  server_port: 9443
ssl:
  enable: true
  cert: server.crt
  key: server.pem
logger:
  max_size: 50MiB
  max_files: 5
archive:
  enable: false
  policy: purge
  start_at: 03:00:00.0
  retention: 3days
pass_policy:
  enable: true 
  min_len: 8
  digits: true 
  special_chars: false
  capital_letters: false
  forbidden_chars: ''
#### `eula` This option must be set to `true`, otherwise the server will not run. This is the EULA accept flag. #### `data` This option sets a path to the Flopsar environment directory. #### `agent` This option sets a server socket address, which is used by agents to connect to the server. * `server_ip` - specifies the IP address (CIDR format), on which the server is to listen for TCP/IP connections from agents. Only agents should connect to this address. If you want the server to listen on all interfaces, put `0.0.0.0` value. * `server_port` - specified the IP port, on which the server is to listen for TCP/IP connections from agents. #### `ui` This option sets a server socket address, which is used by workstation clients to connect to the server. * `server_ip` - specifies the IP address (CIDR format), on which the server is to listen for TCP/IP connections from workstation clients. Only workstation clients should connect to this address. * `server_port` - specified the IP port, on which the server is to listen for TCP/IP connections from workstation clients. #### `ssl` This option switches on/off encryption on the workstation clients connections. Please note, agent-server connections are not encrypted. * `enable` - if set to `true`, enables the SSL encryption. If set to `false`, disables it. * `cert` - the server certificate file name. * `key` - the server private key file name. The SSL files must be stored in the `ssl` directory. #### `logger` This option manages the server logger. * `max_size` - maximal size a logger file can reach. When the value is reached, the logger file is rolled. * `max_files` - maximal number of logger files. The oldest file is overwritten, so that the maximal number files is maintained according to this option value. #### `archive` This option manages [data archiving](#data-archiving). * `enable` - if set to `true`, enables the data archiving feature. If set to `false`, disables it. * `policy` - if set to `purge`, removes all the old data. If set to `archive`, backup the data. * `start_at` - sets a time of day the archive procedure should start. The time of day is specified in the format `HH:MM:SS.m` * `retention` - sets the data retention. **`pass_policy`** This option allows you to configure the password requirements for your system. It provides several parameters to define the desired policy. * `enable` - if set to `true`, enables the passwords policy feature. If set to `false`, disables it. * `min_len` - sets a requirement for the minimal password length. * `digits` - if set to `true`, passwords must have at least one digit. * `special_chars` - if set to `true`, passwords must have at least one special character. * `capital_letters` - If set to true, passwords must have at least one capital letter. * `forbidden_chars` - this parameter allows you to specify a list of characters that are not allowed in passwords. If any of the specified characters are found in a password, it will be considered invalid. If the parameter is empty, all characters are allowed in passwords. ## File System The server stores all of its data in a `` directory, which has the following structure: {% code title="\" %} ``` /arch /logs /storage /web /var ``` {% endcode %} ## Storage Flopsar makes use of a hybrid approach, which incorporates both SQL database and a custom engine storage. The engine storage is used to store the data while the SQL datatabse to store configuration data. ### Data Archiving The Data Archiving feature serves as a valuable tool to effectively manage the volume of your database. By enabling this feature, a daily execution is scheduled at the specified time. This process, governed by the chosen policy, facilitates either data purging or data archiving. To exclusively retain recent data while disregarding older records, simply configure the system to adopt the `purge` policy as specified in the configuration file. Alternatively, opt for the `archive` policy, which entails compressing the data and relocating it to the designated `/arch` directory. The archived data is stored within a zip file, encompassing invocations, key-value data, and their corresponding symbols. # Installation Before you install the software, make sure you meet the following requirements: * you have at least a few GB of storage for the data. * your networking environment does not block TCP connections between components. The installation procedure covers the server and the agents. ## Server The server binary is distributed as both `deb` and `rpm` packages. First of all, you need to [download](https://cdn.flopsar.com/) the server installation package for [your platform and architecture](/5.0/overview/supported-platforms-and-requirements#server). If you want to serve the workstation application from the Flopsar server, download `flopsar-workstation.zip` file too. For Debian based distributions you can install it by executing the following command: ```shell-session # dpkg -i flopsar-server-..deb ``` For Red Hat based distributions you can install it by executing one of the following commands: ```shell-session # rpm -ih flopsar-server-..rpm # zypper install flopsar-server-..rpm # yum install flopsar-server-..rpm ``` In both cases, the installation procedure performs the following operations: * installs `flopsar-server` binary * creates a `flopsar` user, which is used to run the server * adds a `flopsar` `systemd` service * creates a global configuration at `/etc/flopsar` ### Environment Installation When the server is installed, next you need to create a new Flopsar environment where the data will be stored. You have two options at your disposal, either bundled environment or separated one. {% tabs %} {% tab title="Workstation Included" %} Copy `flopsar-workstation.zip` to your destination machine. Now, you need to create a new Flopsar environment by executing the following command: ```shell-session $ flopsar-server create --app flopsar-workstation.zip ``` where `` is a path to the location where the server home directory will be created. If the command executes successfully, it will create a new directory at the specified path. {% endtab %} {% tab title="Workstation Excluded" %} You need to create a new Flopsar environment by executing the following command: ```shell-session $ flopsar-server create ``` where `` is a path to the location where the server home directory will be created. If the command executes successfully, it will create a new directory at the specified path. Note, since the workstation application is not served from the Flopsar server in this case, you need to download the `flopsar-workstation.zip` file and install it on your own web server. For example, in nginx web server you should add the following `location` sections to your `server` configuration: {% code title="nginx.conf" lineNumbers="true" %} ```nginx server { ..... location / { try_files $uri $uri/ =404; } location /api/ { proxy_pass http://localhost:9443; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "Upgrade"; proxy_set_header Host $host; } .... } ``` {% endcode %} The example above assumes that the Flopsar server listens to requests on `localhost:9443` socket address and the Flopsar server has its `ssl.enable` configuration option value set to `false`. Obviously, you should adjust the `proxy_pass` option to your own needs according to the [server configuration](/5.0/administrator-guide/server#ui). {% endtab %} {% endtabs %} {% hint style="warning" %} If you start the server using `systemd`, note that it will run it as `flopsar` user by default. Make sure this user has read/write permissions to the environment directory. {% endhint %} ### Server Configuration The server stores its [configuration](/5.0/administrator-guide/server#configuration) globally in `/etc/flopsar/settings.yml` file. You must edit this file to make at least two changes. The first one is required and you must accept the [EULA](https://cdn.flopsar.com/#/eula) by setting option `eula: true` otherwise, you will not be able to run the server. The second one is to set the `data` option, which points to you environment directory. {% hint style="info" %} This new environment will be created with some default settings. Adjust these settings to your needs accordingly. Please, refer to the [server configuration](/5.0/administrator-guide/server#configuration) for more details. {% endhint %} ### Running the Server Finally, you can start the server by executing the following command: ```shell-session # systemctl start flopsar ``` If you installed the workstation application bundled with the Flopsar Server, then you can access the application at the address specified at the [server configuration](/5.0/administrator-guide/server#ui). If you installed the workstation application outside the Flopsar server, you can access it at the address specified in your web server configuration. In either case, you should be able to access the workstation application login page at the address: `https://:9443`

Workstation Login Page

{% hint style="info" %} Default credentials are: `admin/flopsar` {% endhint %} ### Quick Sample Installation This section presents a sample, quick installation of the Flopsar server. First download the installation package along with the workstation application. Next, execute the following command: ```shell-session $ sudo dpkg -i flopsar-server-5.0.0.amd64.deb ``` or if you use `rpm` package: ```shell-session $ sudo rpm -i flopsar-server-5.0.0.x86_64.rpm ``` Next, let's create a new Flopsar environment at `/opt` directory: ```shell-session $ sudo flopsar-server create --app flopsar-workstation.zip /opt/flopsar ``` If the environment has been created successfully, the following output should be displayed: 
 Creating a new Flopsar Server environment...
 Unpacking Flopsar Workstation application...
 Flopsar Server environment created successfully.
Next, change the ownership of the environment directory to `flopsar` user: ```shell-session $ sudo chown -R flopsar /opt/flopsar ``` Next, open the `/etc/flopsar/settings.yml` file: {% code lineNumbers="true" %} ```yaml eula: false data: /change/me agent: server_ip: 0.0.0.0 server_port: 9000 ui: server_ip: 0.0.0.0 server_port: 9443 ssl: enable: true cert: ssl/server.crt key: ssl/server.pem logger: max_size: 50.00 MiB max_files: 5 archive: enable: false policy: purge start_at: 03:00:00.0 retention: 3days ``` {% endcode %} and make changes in both `eula` and `data` options (line nr 1 and 2): {% code lineNumbers="true" %} ```yaml eula: true data: /opt/flopsar agent: server_ip: 0.0.0.0 server_port: 9000 ui: server_ip: 0.0.0.0 server_port: 9443 ssl: enable: true cert: ssl/server.crt key: ssl/server.pem logger: max_size: 50.00 MiB max_files: 5 archive: enable: false policy: purge start_at: 03:00:00.0 retention: 3days ``` {% endcode %} And finally, start the server: ```shell-session $ sudo systemctl start flopsar ``` Now, you should be able to access the workstation application at [https://localhost:9443](https://localhost:9443/) ## Agent To install the agent, follow carefully the instructions below: 1. select the agent library file for your platform and architecture. 2. copy the agent library to your server machine. 3. create a directory (we call this directory a Flopsar home directory) and `ext` subdirectory for the agent data. 4. configure the agent. 5. optionally put all your agent plugins into the `ext` directory. 6. restart your application. # Upgrade ## Agent Upgrading the agent is simple, you just override the agent library with the new one. ## Server If you use an `rpm` package, then you upgrade the server installation by executing the following command: ```shell-session # rpm -U --noscripts flopsar-server-..rpm ``` If you use deb package, then you upgrade the server installation by executing the following command: ```shell-session # dpkg -i flopsar-server-..deb ``` ## Workstation If the workstation is served from the server itself, to upgrade it you need to execute the following command: ```shell-session # flopsar-server update --app flopsar-workstation.zip ``` where `` is a location of the server home directory. {% hint style="warning" %} Make sure, the user the Flopsar server is running on has appropriate permissions to the Workstation files after upgrade. {% endhint %} # Managing Users Users are managed in the workstation application. They can be added by another user appropriate permissions or they can signup themselves. ## User's Permissions A user authorization model is based on permissions. The permissions are divided into logical groups. ### Administration | Permisson | Description | | ------------------ | ---------------------------------------- | | Update license | Allows to change a license. | | Edit users | Allows to create, edit and delete users. | | Delete application | Allows to delete applications data. | ### Agents | Permission | Description | | ----------------------- | ------------------------------------------- | | View agents | Allows to view currently connected agents. | | Query agents | Allows to send queries to agents directly. | | Switch agent's features | Allows to enable/disable agents's features. | ### Configuration | Permission | Description | | --------------------- | --------------------------------------------- | | View profiles | Allows to view profiles and profiles' details | | Assign applications | Allows to assign applications to profiles | | Edit profiles | Allows to create, edit and delete profiles | | Edit profiles state | Allows to deploy/undeploy profiles | | Edit license mappings | Allows to edit the license mappings | ### Data Access | Permission | Description | | --------------- | ---------------------------------- | | View parameters | Allows to view methods' parameters | # Managing a License When you do not have a license, you can still run the Flopsar environment with some limitations, i.e. you can collect data only from a single agent. If you want to collect data from multiple agents, you will need a valid license. This license must be uploaded to the server. You can do this from the License page.

License page

You need to press the `Update` button and upload the license by pasting its contents to the `License Content` field.

Registering a new license

The maximal number of agents, you can collect data from, is determined by the license you have. The license has also an expire date, unless you have a permanent one. When the license expires, all the connected agents get their license revoked. Only a single agent is left with its license. This implies the agents stop collecting nearly all data. ## License Mapping If you have less agents connected than your license allows, all of your agents will get the license granted and you do need to do anything with it. However, if this is not the case, only a part of your agents will get the license. Which agents get the license is done automatically by default. If you want to have a control over this process, you must then assign the license manually. {% hint style="warning" %} You need to have Edit license mapping permission to be able to edit the license mapping. {% endhint %} You will need to create a mapping, which specifies how many agents from a selected application should get the license. {% hint style="info" %} If you have more agents from the same application than the number you assigned for them in the mapping, only part of them will get the license. The agents, which get the license will be selected randomly. {% endhint %} # Logging ## Logging Levels Each Flopsar module (application) uses logging levels. The levels are defined as follows: **`debug`** : more detailed logging than `info`. Used to log some additional information about the application operation and data processing. Use only in cases when you suspect some problems with the application, want to trace how it processes requests or you are requested by the Flopsar Technology Support to help investigate some problems, or any other problems. It prints lots of diagnostic information, which might not be covered by this manual. **`info`** : default level used in applications. Used only to log some runtime information to verify a configuration and the like. You should use this level on normal, production operation. **`warn`** : used in cases when some error occurs but with no impact on the application stability and data integrity. **`error`** : normally, there should be no error messages in the log. The level is used in cases when some error occurs. These errors impact the application operation stability and data integrity but not all of them. The messages are informative, so you should know whether you can ignore them or not. Any system calls and external libraries errors are also logged in this level. ## Messages All logger messages have the following format: ``` DATETIME LEVEL [XXXXXXXX] LOG_ENTRY ``` where: `DATETIME` - date time of the log entry, `LEVEL` - log level, `XXXXXXXX` - first eight characters of the agent instance name, `LOG_ENTRY` - log entry ## Agent The agent stores its logs in its `` directory. ## Server The server stores its logs in the `/logs` directory. Please, refer to the [server configuration](/5.0/administrator-guide/server#logger) for more details on its logger settings. # Profiles A profile is a collection of settings and rules, which govern agents. It tells agents what data and metrics should be collected. Each profile needs to be activated to be eligible for deployment to agents. Which profile goes to which agent is determined by assigning applications to the profiles. There is a one-to-one correspondence between profiles and applications. In other words, the agents are not assigned to profiles directly. ## Profile State Machine Every profile can be in one of three states: `draft`, `ready` or `deployed`.

Profile State Machine

State-to-state connections refer to operations that are allowed on the states they originate from. ## Managing Profiles Profiles are managed on the `Profiles` page.

Profiles View

To add a new profile, you must select the `New` item from the `Options` menu, specify a name for the profile and press the `Save` button. Next, you can click the profile name in the list to view its [details](#profile-details). To refresh the profiles list, just select the `Refresh` item from the `Options` menu. {% hint style="info" %} A profile can be deleted only in the profile details page. {% endhint %} ## Profile The profile details page contains a selected profile detailed information, such as its settings, instrumentation rules and JMX list of MBeans.

Profile details view

### Profile Settings Each profile has some settings, which are either instrumentation or JMX related. The instrumentation settings are: * **`CPU Time`** this option enables/disables CPU time metric, * **`Automatic tuning`** this option sets one of the three available automatic tuning modes: `light`, `moderate` or `aggressive`. Automatic tuning is a feature which drops irrelevant method invocation reports automatically. If you want the automatic tuning algorithm to pass more methods invocation data from agents, just select the `light` mode. If you want to have less data, select higher modes like `moderate` or `aggressive`. * **`JMX Period`** this option sets the time interval at which JMX data is collected. ### Instrumentation Rules In order to make a profile deployable, you must define some instrumentation rules. You can do this in the `Instrumentation rules` tab.

Empty Instrumentation Rules View

Here, you can define your rules which can be either inclusive or exclusive (see [Rules](/5.0/user-guide/profiles/instrumentation#rules) for details). #### Inclusive Rules Inclusive rules are the ones that precisely define what classes and methods should be instrumented. To add such a rule, just press the `Inclusive Rule` button. A dialog window, with the rules wizard, should open then.

Inclusion Rule Wizard

The wizard is composed of three steps. In the first step, you need to specify what classes should be taken. The following options are available for the `Class level pattern`: * **`Package`** the pattern is a name of a package that classes should belong to, e.g. `com.foo.bar` * **`Classname`** the pattern is a fully qualified class name. Only those classes will be taken into account whose name matches the specified pattern, e.g. `com.foo.bar.MyClass` * **`Superclass`** the pattern is a fully qualified name of a class that classes should inherit from. Only those classes will be taken into account which inherit from the specified class directly, e.g. `com.foo.bar.MyClass` * **`Interface`** the pattern is a fully qualified name of an interface that classes should implement directly. Only those classes will be taken into account which implement the specified interface, e.g. `com.foo.bar.MyInterface` In the second step, you need to specify which methods should be instrumented.

Inclusion Rule Wizard

There are three ways you can choose from to specify the filter for methods for the `Pattern type`: * **`Access Modifiers`** use this pattern to select only those methods whose access modifier is at least one of the chosen modifiers * **`Name`** use this pattern to select only those methods whose name matches the specified value in the `Pattern` field * **`Name & Signature`** use this pattern to select only those methods whose name and signature matches the specified value exactly in the `Name and Signature` field. {% hint style="info" %} For **`Name & Signature`** option you must specify the value in the same way you would do it in your source code, except you must specify fully qualified names for classes here. {% endhint %} In the last step, you need to specify what kind of action should be performed when the instrumented method is to be executed.

Inclusion Rule WIzard

#### Actions Suppose you have a class defined below and the method `someMethod` is instrumented. You must select one action handler for your rule. {% code title="SampleClass.java" overflow="wrap" lineNumbers="true" %} ```java public class SampleClass { public int someField; private long someField2; private Foo someField3; public void someMethod(String arg1, int arg2, Bar arg3) { /* method block */ } } ``` {% endcode %} {% hint style="info" %} Not all actions are available since their availability depends on the selected `Pattern type` in the previous step. {% endhint %} What you get depends on the selected action handler. You have five choices at your disposal for the `Action handler`: **Basic** This action will report only the fact that a method has been called along with some basic information like duration, timestamp etc. **Standard** This action will report the same data as the above action and method arguments additionally. The method arguments will be presented as a result of executing `String.valueOf` function on each one of them. In the case of the sample class you will get: ```java // SomeClass instance String.valueOf(this) // someMethod arguments arg1 arg2 String.valueOf(arg3) ``` **Finer Standard** This action is an extended version of the action above. Additionally, if the method is an instance method it will report values of the instance object fields. Static fields will not be reported. In the case of the sample class you will get: ```java // SomeClass instance fields someField someField2 String.valueOf(someField3) // someMethod arguments arg1 arg2 String.valueOf(arg3) ``` **Finest Standard** This action is similar to the action above but in this case, if an argument is an object, its instance fields will be reported instead of its value. Static fields will not be reported. In the case of the sample class you will get: ```java // SomeClass instance fields someField someField2 String.valueOf(someField3) // methodSample arguments fields if any arg1 arg2 // Bar instance fields barField1 barField2 ... ``` **Custom** This action will execute a user defined custom code that is provided by the agent extensions and report its output along with the basic information. In this case, you decide what you get by defining your own [agent extension](/5.0/developer-guide/agent-extensions). {% hint style="warning" %} Please note, Finer Standard and Finest Standard can degrade your code execution performance so use it carefully. Custom handler can do this also if your implementation is poor. {% endhint %} #### Additional Features Depending on the selected `Action handler`, there are some additional features you can select in the `Features` section: * **`Always report`** this feature makes all the instrumented methods ignore the automatic tuning in the profile [settings](#profile-settings) * **`Instrument on exit`** this feature injects the instrumentation code at the end of the method body Additionally, if you selected the **`Execute custom code`** option, you will need to specify both a class name and a method name of your custom extension code, the agent is about to inject, in the `User-defined extension` section. A fully qualified name for the `Class name` field is required and a method name for the `Method name` field. If you are done with the rule, press the `Save` button to add the rule to the instrumentation rules list. #### Exclusive Rules By default, those methods are instrumented which fulfill the conditions defined by the inclusive rules. However, sometimes these rules are too wide, i.e. they let too many methods or classes in. In order to exclude some classes and methods from instrumentation, you need to define exclusive rules. You can do it by pressing the `Exclusive Rule` button. A new popup window will appear then.

Exclusion Rule Form

To add a new rule, you need to specify both class and method patterns. The following options are available for the `Class level pattern`: * **`Package`** the pattern is a name of a package that classes should belong to. Only those classes will be excluded from instrumentation which belong to the specified package. * **`Classname`** the pattern is a fully qualified class name. Only those classes will be excluded from instrumentation whose name matches the specified pattern, e.g. `com.foo.bar.MyClass` * **`Superclass`** the pattern is a fully qualified name of a class that classes should inherit from. Only those classes will be excluded from instrumentation which inherit from the specified class directly, e.g. `com.foo.bar.MyClass` * **`Interface`** the pattern is a fully qualified name of an interface that classes should implement directly. Only those classes will be excluded from instrumentation which implement the specified interface, e.g. `com.foo.bar.MyInterface` For the `Method level pattern` you need to specify one of the following options: * **`Method name`** those methods will be excluded from instrumentation whose name matches the specified value in the `Pattern` field * **`Method signature`** those methods will be excluded from instrumentation whose signature matches exactly the one specified in the `Pattern` field {% hint style="info" %} If you want to exclude all methods from a class, select the **`Method name`** pattern, put `.*` into the `Pattern` and select the `regex` option. {% endhint %} If you are done with the rule, press the `Save` button to add the rule to the instrumentation rules list. #### List of Rules When you add new rules, they are presented in a list. The rules are evaluated in the order of their appearance in the list (see [Rules Evaluation Order](/5.0/user-guide/profiles/instrumentation#rules-evaluation-order) for more details).

Instrumentation Rules

You can click each rule to see its details. Exclusive rules are denoted by red squares, while inclusive ones are denoted by green squares. If you want to edit a rule, just click the pencil icon in the right down corner of the selected rule. If you want to delete a rule, click the trash icon in the right down corner of the selected rule. To clone a rule, click the copy icon in the right down corner of the selected rule. A rule can be in one of the two states: enabled or disabled. You can switch the rule state by clicking the `Enable rule` switch. When a rule is disabled, it will not be deployed to agents. Each rule has also a description, which is self-explanatory. ### JMX This view enables specifying which MBeans values you want to collect. To add MBeans, you need to retrieve them from agents first. You can do this by pressing the `Add ObjectNames` button.

JMX View

This operation will open a new popup window.
Next, press the `Refresh` button to retrieve the MBeans. {% hint style="info" %} You must first assign an application to your profile to be able to retrieve MBeans from the connected agents. {% endhint %}
Now, you can press the `Add` button for any MBean you want to select. When you are done with adding the MBeans either click outside the popup window or press the **`Escape`** button to close the popup window. ### Deploying a Profile To deploy a profile, a few conditions must be fulfilled. The profile must have instrumentation rules defined, must be assigned to an application and be in the `ready` state. # Instrumentation Instrumentation is a JVM feature, which enables modification of the byte-codes of methods. ## Limitations There are some limitations imposed on the instrumentation, i.e. not every entity can be instrumented. These limitations are hard-coded in agents and you cannot change them. There are both class and method limitations. ### Classes Classes from the following packages are always excluded from instrumentation: ```java com.flopsar.* com.sun.* javassist.* java.* javax.* jdk.* org.apache.bcel.* org.objectweb.* sun.* ``` Additionally, the following class entities are also excluded from instrumentation: * interfaces * enum classes * annotation classes * modules * synthetic classes * primitives * arrays ### Methods The following methods are always excluded from instrumentation: * abstract methods * synthetic methods * native methods * constructors * class initializers ## Rules To instrument methods, you need to decide which methods should be instrumented. This is governed by a set of rules. There are two types of rules: *inclusions* and *exclusions*. You use inclusion rules to specify what should be instrumented and exclusion rules to specify what mustn't be instrumented. Each rule has to define both a class and a method filter. The class part specifies which classes should be taken into account and the method filter specifies which methods, from these classes, should be instrumented and what data they should collect. This implies that multiple rules can instrument the same class.

Multiple rules instrumenting the same class

### Rules Evaluation Order All the rules are evaluated in a very specific order. When a class is passed to the rules filter the exclusion rules are evaluated first. If the class part of the exclusion rule matches, the method part is checked next. If both class and method match the rule, the method will not be instrumented and other rules will not be evaluated for this method. #### Example Suppose we have a rule set consisting of four rules: two exclusive and two inclusive. Let's analyze how a class is evaluated against the rules. The pictures below will help us to visualize the flow. Each row in the picture represents a single rule evaluation flow. Red arrows represent an exclusive rules flow, while the green ones represent the inclusive rules flow. First column represents the class part of a rule while the second column the method part of a rule.

Sample Rules Evaluation

Suppose we have a class `A`, which is passed to the rules filter. The class has four methods. First, the exclusion rules are evaluated. The first exclusion rule matches the class, so the method part of the rule is evaluated next. As we can see, the `method_4` matches the method part of the rule. That means, this method is excluded from instrumentation and it becomes invisible to the rest of the rules. Next, the second exclusion rule is evaluated. Again, we have a match for the class part but this time we do not have a match in the method part of the rule. That means, the class with the remaining methods is passed further. Since, there are no more exclusive rules, we get to the first inclusive rule (the third row). Here we have a match in the class part of the rule, so we move to the method part where we have a match for the `method_2`. This means, that method will be instrumented by the first inclusive rule and it becomes invisible to the last rule. Finally, we get to the last inclusive rule. Again, we have a match in both the class and method part. This time `method_3` is matched. This method will be instrumented by the last inclusive rule. There left only one method `method_1`, which did not match any of the rules. This method will not be instrumented. The final result of the evaluation is that two methods will be instrumented by different rules and the rest of the methods will be left unchanged. ## Retransformation Retransformation is a feature which enables to reload a collection of classes. This operation is preformed when you deploy a profile. Unfortunately, this operation is expensive and even though the instrumentation itself is blazingly fast, the internal JVM mechanics involved into the retransformation process takes much more time. {% hint style="warning" %} During the retransformation process you application performance will be degraded and your application can become unresponsive. {% endhint %} That is something you should always take into account when deploying profiles, e.g. try not to deploy profiles when your application is under heavy load. # Data Browser If you want to explore method invocations and their collected data, you should use the **`Data Browser`** page. ## Browser Form

Data Browser Form

There is only one mandatory field you must fill to make a query and retrieve the result. You must specify the date range in the fields `Date` and `Time Interval`. This will retrieve all available data from the specified time range. ### Additional Options If you want to narrow your search results to an application, you should specify the application name in the `Application` field. This will retrieve methods invocations only from the specified application. To narrow your search results even more, you can fill out the `Pattern` field. The specified value is a pattern to search for in methods invocations data. You need to select one of the four types: * **`contains`** the pattern will match when the methods data contains the pattern value, * **`exact`** the pattern will match only if the methods data are exactly the same as the pattern value, * **`starts`** the pattern will match when the methods data starts with the pattern value, * **`regex`** the pattern will match when the methods data satisfies the regular expression in the pattern value. {% hint style="info" %} If the `Pattern` field is filled, the search query will retrieve any method invocation that satisfies the query conditions, otherwise only root invocations of execution stacks will be retrieved. {% endhint %} If you want to retrieve only those invocations that were executed within a particular thread, just specify the value in the `Thread name` field. ## Search Results The search results are presented in a form of a list. Each list record presents a single method invocation. If you want to see more details about the selected method invocation, just click the method signature and you will get details of the execution stack this method is a member of.

Browsing Sample Results

### Execution Stack Details This view presents an complete execution stack of the selected method.
#### Execution Stack The method execution stack is depicted as an inverted stack, where a root call is located at the top and all consequent invocations are placed on each other. Each bar in the stack denotes a single method invocation. The width of the bar denotes the method duration and it is scaled relative to the root call bar width. The time flows from left to right, hence the order of bars on the same level matters. Each bar has one of four background colors. If an invocation has no additional data, its background is grey. If the invocation have only parameters its background is green. If it has only an exception its background is red then. It if has both an exception and parameters its background is orange.

Sample Execution Stack

The stack is interactive, if you click one of the bars you will get its method invocation details, below the execution stack. The method is identified by its signature and a class it belongs to. There are two *copy* icons, next to both `Class name` and `Method` labels. If you click the icon, the corresponding value will be copied to the profiles clipboard, which can be used in [defining instrumentation rules](/5.0/user-guide/profiles#inclusive-rules). #### Spectrum This view presents a list of all methods called within the execution stack. The methods are aggregated and sorted. You can sort them with respect to one of the three criteria: self time, calls count and exceptions count.
# Data Analysis This view is useful when you want to analyze your collected method invocations. There are only two fields that need to be fill out. You must specify both the date range and the application. Then, the analysis will be performed for those method invocations which ended within the specified time range and came from the specified application.

Data Analysis Form

## Duration Distribution The first part of the analysis presents the methods duration distribution with respect to percentiles. The chart below presents a sample analysis result. ### Plot The `Plot` tab presents a methods overall duration distribution. The analysis is performed for all methods invocations within the time range specified in the data analysis form.

Sample Duration Distribution

The chart is divided into percentile sections. The percentile values are located on the `x`-axis. The left `y`-axis presents the method duration value, while the right one the number of method invocations. Each percentile section has a bar which represents the number of method invocations within the percentile interval. The blue curve represents the methods duration. To properly read the chart, you need to focus on the percentile axis (`x`-axis). Note, the `x`-axis is not scaled linearly. Suppose now, we have the analysis results as the one above in the picture. We interpret the chart at `96.881th` percentile as follows: * `96.881` percent of all invocations execute below `233` ms. * there are about `8300` invocations whose duration is between `170` and `233` ms. Note, `170` ms is taken from the previous percentile, which is `93.757` in this case. It is easy to see, in the sample case, that the number of methods invocations decreases rapidly as percentile grows. There are only a few outliers, whose duration is above a minute. It is up to a user whether a further analysis is required or not. If a more detailed investigation is needed a user can proceed with the analysis by pressing the `Temporal analysis` button. ### Summary The Summary tab allows a user to compare the analysis results to the data from previous periods. This time a few specific percentiles are selected to compare them with the ones from previous periods.

Sample Analysis Summary

There are three special columns, in both `Duration` and `Count` columns. These columns are denoted as `1d`, `3d` and `1w` which mean 1 day, 3 days and 1 week respectively. They contain the corresponding changes of percentile values one day ago, three days ago and one week ago respectively with respect to the current values. In order to retrieve the values you need to click the refresh button in the selected column header. The values are then compared with the current ones within the corresponding time range but from previous days. For example, if you want to compare the current results with the ones from 3 days ago, just click the refresh button in the `3d` column header. If the current duration value is less than three days ago, the value change is displayed in green, otherwise it is displayed in red. In the picture above, one can easily see that the current duration value for the 75th percentile is 63 ms. This result is 17 ms less than three days ago. Moreover, currently we have 200,639 invocations, while three days ago there were 40,038 less invocations. The percentile value in the fourth row is customizable. It can be changed by selecting one of the available values in the `Custom percentile` buttons group. ## Temporal Analysis This part of the analysis presents how the percentile values change in time for the specified period. There are two tabs with the results. The first one `Values` presents how the percentile values change in time. Just like in the duration distribution there are bars and curves. The bars represent the amounts of invocations in time periods. The curves present the percentile values of duration. There are four percentiles curves that can be displayed simultaneously, i.e. 25th, 50th, 75th and 9\*th. The last one is customizable and can be changed in the `Highest percentile` option.

Sample Temporal Analysis

The second tab `Sections` presents how many invocations are in each percentile. ## 2D Analysis The 2D analysis is represented in a form of a heatmap chart. The chart presents a distribution of one of the selected `Data type` with respect to both duration (vertical axis) and time (horizontal axis). The chart is divided into cells, which aggregate data for particular ranges of duration and time. For `Count` data type, each cell represents a number of invocations within the cell range. For `Off-Cpu` data type, each cell represents a percentage of method invocations, whose off-cpu time is the largest. For `Exceptions` data type, each cell represents a number of methods that throw exceptions.

Sample 2D Analysis

You can change the number of cells the chart is divided into, by selecting one of the `Resolution` value. You can also change the range of methods that are analyzed by selecting one of the `Precentile` values. For example, if the `Percentile` value is 50, it means that the chart shows only those method invocations whose duration value is below the value at 50th percentile. ## Exceptions This view presents top ten [root exceptions](/5.0/user-guide/glossary#execution-stack) thrown by methods.
# Metrics This page allows to plot graphs of methods invocations and other key-value metrics. For methods graphs, you need to select the `Method` option in the `Metric Type` field. Next in the `Class` field, you need to specify a pattern for the class name you search for. You must specify at least three characters to retrieve a list of classes whose names contain the specified characters. The pattern you specify will be sought in the class names only, without the package part. If you select one of the classes, a list of its methods will be retrieved.

Metrics Form

For key-value graphs, you need to select the `Key-Value` option in the `Metric Type` field. Next in the `Key` field you need to specify a pattern to search for in the available metrics. You need to specify at least three characters to retrieve a list of the metrics keys. # Glossary Throughout the entire documentation, we use some terminology regarding executions of methods. ## Execution Stack The picture below presents a single method execution stack. The first element of the tree is called ***Root Call***. All the methods' invocations, down the execution stack, are called simply ***Calls*** and ***Invocations*** interchangeably***.*** The entire structure is called ***Execution Stack***. When an exception is thrown by some of the methods and propagated through the stack, the origin exception is called ***Root Exception***.

Execution Stack

### How the Stack is Assembled The picture below depicts the procedure of how the execution tree is built. In the picture below you can see four, sample execution trees of some method *`A`* call.
The first `a` view shows how the real execution stack looks. This is a real execution flow that takes place in your application. The second `b` view presents the same stack instrumented by your agent. Depending on your configuration, the instrumentation may not cover the entire stack. The empty, grey boxes represent missed, not instrumented methods. As you can see, at this very point your instrumented execution flow is not complete. Moreover, in your application runtime, there can be situations when some of the instrumented method calls will not be reported. This empty, violet box in the third `c` view represents this situation. Finally, what you actually observe in the workstation is the last `d` view. This view presents an assembled stack with missing calls ignored. Now you know how to interpret the execution stack in the workstation and why the resulting stack can differ from the original one. ### Self-time Self-time is a part of a method duration, in which the method executes its own code without calling any other methods. Suppose we have a situation depicted below. A method `A` executes within `duration_a`. However, this method calls another method `B` and waits while the method `B` is executing its own code. Then, when the method `B` returns, the method `A` can continue executing its code.

Self-time

It's easy to see that the method `B` execution time contributes to the method `A` execution time and the difference between method `A` duration and method `B` duration is the **self-time** of the method `A`. # FAQ
How do I restore access to the server if I forgot my password? If there is another user who is able to access the server via Workstation application, your password can be reset by this user, provided that they have appropriate permissions to do so. If there is no such user, you must login to the machine where the server runs on and reset your access using the command below: ```shell-session $ flopsar-server users --reset ``` This operation will set the password to `flopsar` and also grant the specified user all the permissions.
# Agent Extensions An agent can be used standalone or it can be extended by user custom code. Flopsar Agent is a powerful engine, which is extensible by plugins. The extensions are some extra jar files, which consist of user-defined POJO classes. The jars can be then included in the agent deployment. The plugins enable to extend the agent capabilities. We provide a mechanism that you can use to extend your application diagnosis and trace your data flow. It makes Flopsar an open tool, that can be extended. The plugin allows for processing data inside instrumented methods. There are two types of extensions, which can be used: **standard** and **custom**. ## Standard The **standard** extension is embedded in the agent. This extension will execute the following method on each argument it gets: ```java public static java.lang.String valueOf(java.lang.Object); ``` and return a result in a form of concatenated pairs of parameter identifiers and the corresponding `valueOf` function result. The following parameters are returned (in order of appearance): * `THIS` : If the instrumented method is static, this parameter is `null`, otherwise a class instance the method is executed on. * `ARGS` : List of all arguments of the instrumented method. ## Custom If you are not satisfied with standard extension and you want to extract some additional information (or make some preprocessing) you must implement your own, `custom` extension. In order to take advantage of this feature, all you have to do is set up a configuration, create a simple POJO class and implement a single method: ```java public static java.lang.String myFormatter(java.lang.Object[] args) ``` {% hint style="warning" %} You can give any name to the above method but the signature of the method must be preserved. Otherwise, it will not work. {% endhint %} The `args` argument of the extension method is an array of the instrumented method arguments delivered to your method. These are your original application objects, not clones. When you set up the configuration correctly, the agent will inject your code into the instrumented methods during the instrumentation. It can be injected in one of the two places, i.e. at the beginning or the end of the method. Note, your formatter code is executed inside your instrumented method, so make sure your formatter code generates minimal overhead. The argument `args` is an array of the following objects (in order of appearance): * **`args[0]`** If the instrumented method is static this element is `null`, otherwise it is a reference to a class instance the instrumented method is executed on. * **`args[1..N]`** List of all arguments of the instrumented method in the same order as they appear in the method signature. These are the references to your original application objects, not some *clones*. where **N** is the number of the instrumented method arguments. When you set up the configuration correctly, the agent will inject your formatter code into the instrumented methods during the instrumentation. It can be injected in one of the two places, i.e. at the beginning or the end of the method. Note, your extension code is executed inside your instrumented method, so make sure your code generates minimal overhead. In fact, it is up to you what extra features the agent will have. You can even extend your agent installation to cover a distributed transaction tracing. {% hint style="info" %} Use `U+001E` as a separator in your code. Otherwise, the result will not be formatted properly in the workstation. {% endhint %} Your result should always consist of pairs of keys and values. `U+001E` separates each entry in the result, which means you should always have an odd number of separators. The resulting string size cannot be greater than 8192 B, otherwise it will be truncated. There are some tips, you should take into account when writing extensions: * Design and implement extensions with performance in mind. If your code performs poorly, obviously the instrumented methods will perform poorly as well. * Use `try-catch` block to protect your application from any errors you can make in your extension implementation. Otherwise, any errors inside the extension code can interfere with your application processing. * Do not modify arguments delivered to your extension since they are your genuine application objects. Make sure that whatever you do with them will not cause any problem to your application flow. * Try not to use reflections, unless there is no other way. * Try not to create too many new objects, otherwise you risk increased GC overhead. * Deploy your extension classes into separate jar files. This can save you a lot of trouble when dealing with class loading issues. * Your extension code should be stateless. ## Example Suppose, you have some method: ```java public FooBar foo(foo.bar.Object1, foo.bar.Object2, ...); ``` and you are interested in some additional information extracted from the first and second arguments. Your extension implementation can look like the one below: ```java public static java.lang.String myFormatter(java.lang.Object[] args){ final char SEPARATOR = 0x1E; try { foo.bar.Object0 _this = (foo.bar.Object0)args[0]; foo.bar.Object1 obj1 = (foo.bar.Object1)args[1]; foo.bar.Object2 obj2 = (foo.bar.Object2)args[2]; /* implement your logic here to produce string output. */ String output1 = some_logic1(obj1); String output2 = some_logic2(obj2); return "Object1"+SEPARATOR+output1+SEPARATOR+"Object2"+SEPARATOR+output2; } catch(Throwable ex){ return "Error"+SEPARATOR+ex.getMessage(); } } ``` {% hint style="warning" %} Flopsar Technology cannot be held responsible for any damage done by badly written user's extensions. {% endhint %} # Server API {% openapi src="/files/aSPJMzPGymNd79jvoQAY" path="/api/auth/login" method="post" expanded="false" %} [api.yml](https://2268302518-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FRRwSLM5GTvsX7vYPRXpo%2Fuploads%2FXOfLHEIIfW5TZAWYYusX%2Fapi.yml?alt=media\&token=12d66233-ccae-4d7e-8d88-32f9cccb6df6) {% endopenapi %} {% openapi src="/files/aSPJMzPGymNd79jvoQAY" path="/api/apps" method="get" %} [api.yml](https://2268302518-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FRRwSLM5GTvsX7vYPRXpo%2Fuploads%2FXOfLHEIIfW5TZAWYYusX%2Fapi.yml?alt=media\&token=12d66233-ccae-4d7e-8d88-32f9cccb6df6) {% endopenapi %} {% openapi src="/files/aSPJMzPGymNd79jvoQAY" path="/api/data/symbols/{kind}" method="post" %} [api.yml](https://2268302518-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FRRwSLM5GTvsX7vYPRXpo%2Fuploads%2FXOfLHEIIfW5TZAWYYusX%2Fapi.yml?alt=media\&token=12d66233-ccae-4d7e-8d88-32f9cccb6df6) {% endopenapi %} {% openapi src="/files/aSPJMzPGymNd79jvoQAY" path="/api/data/calls" method="post" %} [api.yml](https://2268302518-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FRRwSLM5GTvsX7vYPRXpo%2Fuploads%2FXOfLHEIIfW5TZAWYYusX%2Fapi.yml?alt=media\&token=12d66233-ccae-4d7e-8d88-32f9cccb6df6) {% endopenapi %} # What is Flopsar? {% hint style="info" %} End of life for Flopsar 5.1. This product is no longer supported. {% endhint %} Flopsar, developed by Flopsar Technology, is the cutting-edge Fault Detection and Diagnosis software for JVM systems. Our mission is simple: to detect errors and problems in your Java systems swiftly and assist in identifying the root cause effectively. In the world of systems maintenance, availability and responsiveness are paramount. At Flopsar, we understand that minimizing system outages and swiftly resolving issues are critical factors. When a system fails, it's not enough to know that it failed—we must know why. Gone are the days of spending hours or even days searching for the cause. We tackle these challenges head-on. Unlike traditional APM tools, Flopsar eliminates the need for users to predict potential system issues in advance. We don't settle for vague averages or aggregated data that leave you guessing about your application's processing. Flopsar provides you with a highly customizable and extensible product that adapts to your unique needs. It seamlessly integrates with any framework and application server, making it truly agnostic. But Flopsar offers much more than that. Our powerful features not only enhance systems maintenance but also drive successful application development processes. With Flopsar, you can extend its capabilities using plugins, allowing you to perform additional processing at runtime without making any changes to your application source code. # Supported Platforms and Requirements What platforms and architectures are supported. ## Agent Agent is a native library, supporting the following platforms:
Platformamd64 (x86)ARM
GNU/Linuxtruefalse
Windowstruefalse
{% hint style="warning" %} The agent library requires `libc` in version at least 2.14 {% endhint %} ### Supported JVM Agent supports JVM version from **1.7** through **17**. ## Server The server application is distributed for the following platforms:
Platformamd64 (x86)ARM
GNU/Linuxtruefalse
{% hint style="warning" %} The server requires `libc` in version at least 2.18 {% endhint %} There are no fixed system requirements to run the server, since it strongly depends on your controlled environment. The server runs multiple jobs like storing, collecting, analyzing and serving the collected data. The server is designed to take advantage of multiple CPU cores, so the more cores the better. A reasonable minimum should be at least 2 CPU cores and 4GB of RAM. However, if you notice that the server experiences some performance issues like delays in serving and pushing data, consider to bump the system resources for the server. ## Workstation This is a web application, that can be deployed on any web server that supports web sockets and is able to run as a reverse proxy. {% hint style="warning" %} The Workstation application requires browsers with native **ECMAScript 2015 (ES6)** support. {% endhint %} If you experience some issues with the application try to install a newer browser. # Versioning Every version label is composed of three numbers `X.Y.Z`: `X` Major version number: major changes, e.g. architecture change. `Y` Minor version number: minor changes, e.g. new features. `Z` Small version number: bugs fixes. {% hint style="info" %} Versions `X.Y` are equivalent to `X.Y.0` {% endhint %} Do not mix binaries from different releases, since they are not compatible. You can mix only those components, which have the same major and minor release numbers. {% hint style="danger" %} Incompatible components are unable to work together. Any connection between two incompatible components is dropped. {% endhint %} # What's New This release is focused mainly on making the Flopsar usage simpler. The following features have been added. * The server gets a new logger backend *syslog*. * Agents report how many extensions they have loaded. * Agents detect and report `java.lang.Error` errors. * The workstation sidebar is now collapsible. * The server settings are displayed in the Server page. * The server statistics are collected and presented in the Server page. * Data masking has been introduced. * The Flopsar environment name can now be set. * The workstation sidebar colors are customizable. * Automatic recreation of the Flopsar environment from the server archive. * The number of assigned licenses is reported in the License page. * Central agent extensions management has been introduced. * No more agent logs. The agent diagnostics is now available instead. * Agents can be disabled with a new startup option. * All the instrumented classes are checked before they are passed to the JVM. * A class byte code can be downloaded from JVMs. * New API method for retrieving the current Flopsar environment. * Various fixes and improvements. # Architecture Fundamentals Flopsar consists of three components: * Agent * Server * Workstation Workstation, as a web application, can be served from either the server directly or any web server, which is able to operate as a reverse proxy.

Workstation bundled with the server

Workstation served from a reverse proxy

## Networking Agents initiate bidirectional TCP connections to the server, which listens to these connections on a socket address specified in the server configuration. # Agent Agent is a native library, which is attached to JVMs. It runs on a JVM and its main goal is to collect data from applications running on the JVM and the JVM itself. Each agent is attached to a JVM. It initiates a single bidirectional TCP connection to the Flopsar server. ## Attaching Agents Agents can be attached to the JVM in two ways. The first one is by adding the following option to a `java` command: 
-agentpath:<path-to-agent>=<agent-options>
For example, if the option `-agentpath:/opt/flopsar/libflopsar.so=opt1,opt2` is specified, the agent will be loaded from the `/opt/flopsar/libflopsar.so` file and the configuration options `opt1,opt2` will be passed to it. {% hint style="info" %} There are various places where you can add your custom JVM options, it depends on the Java software you use. Please, refer to your Java software documentation for more details about adding extra JVM options. {% endhint %} The second way of attaching an agent is by setting the `JAVA_TOOL_OPTIONS` environment variable. 
$ export JAVA_TOOL_OPTIONS="-agentpath:<path-to-agent>=<agent-options>"
{% hint style="warning" %} Please note, the `JAVA_TOOL_OPTIONS` environment variable will be picked by any JVM instance running in the environment where this variable is set. {% endhint %} ## Configuration Agent configuration is specified by a set of options. There are two types of settings: mandatory and optional. ### Mandatory Options Each option must be specified in a form: `key=value`, where `key` is an option name. They must be specified along with the agent library itself. * **`app`** Application name, the agent is assigned to, * **`server`** Socket address of the Flopsar server. Agents initiate connections to this server and the connections are maintained until the agents are shutdown. It should be specified in the form `IPv4:port`, {% tabs %} {% tab title="Example" %} The following option 
-agentpath:/opt/agent/libflopsar.so=app=MyApplication,server=192.168.10.11:9000
will be evaluated as follows: * the agent will be loaded from the `/opt/agent/libflopsar.so` file * the agent will be attached to *MyApplication* * the agent will attempt to connect to the Flopsar server at `192.168.10.11:9000` {% endtab %} {% endtabs %} ### Additional Options Each option must be specified in a form: `-Dflopsar.key=value`, where `key` is one of the following values:
Option keyDefault valueDescription
nameAuto-generated and reset every time the agent startsAgent instance name.
clex-A list of class loaders, that should be excluded from instrumentation. The list entries should be fully qualified class names separated by the + sign.
bcp-A feature, which changes the way how the com.flopsar.Flopsar class is loaded into the Bootstrap ClassLoader. Its value should be a path to a directory, that is writable to JVM process. If set, the agent will extract a flopsar.jar file into the specified directory and add it to the bootstrap classloader path. If absent, the agent will load the com.flopsar.Flopsar class into the Bootstrap ClassLoader internally. This option is usually used if you get exceptions like java.lang.NoClassDefFoundError: com/flopsar/Flopsar
qsize10 MiBMaximum size of the agent outbound queue size. By setting this option you specify how much memory the agent can use for the queue. The value should be specified along with the unit, e.g. 30mib, 20m, 50M, 30MiB, 4gib etc.
disableDisables the Flopsar agent.
[^1]: The absolute path from which to load the agent library. [^2]: Comma-separated agent configuration options. # Server Server is one of the key Flopsar components, which is responsible for managing the entire Flopsar environment. It is a single, binary file `flopsar-server`. ## CLI The Flopsar server has a few command line options. All the available options of the server can be listed by executing the command: ```shell-session $ flopsar-server --help ``` ## Configuration All the server settings are configured in a file `settings.yml`. This file contains the following default settings: {% code title="settings.yml" fullWidth="false" %} ```yaml eula: false name: my environment data: /path/to/data max_ext_size: 1MiB agent: server_ip: 0.0.0.0 server_port: 9000 ui_srv: server_ip: 0.0.0.0 server_port: 9443 ui: bg_dark: '#141A1F' bg_light: '#312e81' fg_dark: '#cbd5e1' fg_light: '#f1f5f9' ssl: enable: true cert: server.crt key: server.pem logger: backend: Local max_size: 50MiB max_files: 5 archive: enable: false policy: purge start_at: 03:00:00.0 retention: 3days pass_policy: enable: false min_len: 8 digits: false special_chars: false capital_letters: false forbidden_chars: '' ``` {% endcode %} #### `eula` This option must be set to `true`, otherwise the server will not run. This is the EULA accept flag. #### `name` Name of the environment. #### `data` This option sets a path to the Flopsar environment directory. #### `max_ext_size` This option sets a maximal size of an extension jar file. #### `agent` This option sets a server socket address, which is used by agents to connect to the server. * `server_ip` - specifies the IP address (CIDR format), on which the server is to listen for TCP/IP connections from agents. Only agents should connect to this address. If you want the server to listen on all interfaces, put `0.0.0.0` value. * `server_port` - specified the IP port, on which the server is to listen for TCP/IP connections from agents. #### `ui_srv` This option sets a server socket address, which is used by workstation clients to connect to the server. * `server_ip` - specifies the IP address (CIDR format), on which the server is to listen for TCP/IP connections from workstation clients. Only workstation clients should connect to this address. * `server_port` - specified the IP port, on which the server is to listen for TCP/IP connections from workstation clients. #### `ui` This option sets colors for the navigation bar. * `bg_dark` - specifies the color of the navigation bar background in a dark mode. * `bg_light` - specifies the color of the navigation bar background in a light mode. * `fg_dark` - specifies the color of the navigation bar text in a dark mode. * `fg_light` - specifies the color of the navigation bar text in a light mode. #### `ssl` This option switches on/off encryption on the workstation clients connections. Please note, agent-server connections are not encrypted. * `enable` - if set to `true`, enables the SSL encryption. If set to `false`, disables it. * `cert` - the server certificate file name. * `key` - the server private key file name. The SSL files must be stored in the `ssl` directory. #### `logger` This option manages the server logger. * `backend` - one of the two backend for loggers: `Local` or `Syslog`. * When `Local` is specified, the server log files will be stored in the `log` directory. * When `Syslog` is specified, the logs will be managed by the syslog. In this case, both options `max_size` and `max_files` are ignored. * `max_size` - maximal size a logger file can reach. When the value is reached, the logger file is rolled. * `max_files` - maximal number of logger files. The oldest file is overwritten, so that the maximal number files is maintained according to this option value. #### `archive` This option manages [data archiving](#data-archiving). * `enable` - if set to `true`, enables the data archiving feature. If set to `false`, disables it. * `policy` - if set to `purge`, removes all the old data. If set to `archive`, backup the data. * `start_at` - sets a time of day the archive procedure should start. The time of day is specified in the format `HH:MM:SS.m` * `retention` - sets the data retention. **`pass_policy`** This option allows you to configure the password requirements for your system. It provides several parameters to define the desired policy. * `enable` - if set to `true`, enables the passwords policy feature. If set to `false`, disables it. * `min_len` - sets a requirement for the minimal password length. * `digits` - if set to `true`, passwords must have at least one digit. * `special_chars` - if set to `true`, passwords must have at least one special character. * `capital_letters` - If set to true, passwords must have at least one capital letter. * `forbidden_chars` - this parameter allows you to specify a list of characters that are not allowed in passwords. If any of the specified characters are found in a password, it will be considered invalid. If the parameter is empty, all characters are allowed in passwords. ## File System The server stores all of its data in a `` directory, which has the following structure: {% code title="\" %} ``` /arch /logs /storage /web /var ``` {% endcode %} ## Storage Flopsar makes use of a hybrid approach, which incorporates both SQL database and a custom engine storage. The engine storage is used to store the data while the SQL datatabse to store configuration data. ### Data Archiving The Data Archiving feature serves as a valuable tool to effectively manage the volume of your database. By enabling this feature, a daily execution is scheduled at the specified time. This process, governed by the chosen policy, facilitates either data purging or data archiving. To exclusively retain recent data while disregarding older records, simply configure the system to adopt the `purge` policy as specified in the configuration file. Alternatively, opt for the `archive` policy, which entails compressing the data and relocating it to the designated `/arch` directory. The archived data is stored within a zip file, encompassing invocations, key-value data, and their corresponding symbols. ### Data Restoring All the archive files can be used to restore the Flopsar environment. Suppose you have the `ARCHIVE.zip` file created during the archive procedure. You will need the workstation application also. In order to create a new local environment, just to explore the archive data, you must execute the following command: ```shell-session $ flopsar-server create --archive ARCHIVE.zip --app flopsar-workstation.zip ``` This operation will create a new environment in the `` directory. To access this environment, you must execute the following command: ```shell-session $ flopsar-server local ``` This command will start the Flopsar server in a `local` mode, which is used only for data access purposes. Then you should be able to access the environment at the address # GUI The Flopsar server can be accessed in the Workstation application. There are some activities that can be performed in this web application. You can access the server page by pressing the `Server` button in the side bar.

Server page

The `Server` tab contains some basic information about the server installation and the machine it runs on. Most of the data displayed in the `Info` section is self-explanatory. The `Available space` shows how much free space is available for the server data. The `Settings` section displays the contents of the server [settings configuration file](/5.1/administrator-guide/server#configuration). The `CPU usage` shows the current server process CPU usage. The `Memory usage` shows the current server memory usage. ## Server Statistics The server collects and persists some data on its operation. These data can be useful to monitor the server itself. The collected data can be explored in the `Stats` tab. To retrieve the data you need to specify the time range you want the data from and press the `Refresh` button.

Server sample statistics

There are six metrics available. The `Online agents` graph presents how many agents were connected to the server simultaneously. The `Incomplete stacks` graph presents how many [method executions stacks](/5.1/user-guide/glossary#execution-stack) were being collected in the server internal collector. The `Agents inbound traffic` graph shows the data transfer rate received from all the connected agents. The `Database size` graph shows how much disk space is used for all the collected data. ## Database Contents The server collects all the data sent from the agents in a database. The database contents can be viewed in the `Database` tab.

Sample database contents view

This view is a more fine-grained view of the data. It shows which application's data takes disk space most. The data are classified in logical groups. The `Method Data` group contains methods' parameters and can be safely deleted if you are not interested in them. The `Exceptions` group contains only exceptions stack traces and can also by safely deleted. You can also delete all data of the selected application by clicking the trash icon in the `Total` column. # Installation Before you install the software, make sure you meet the following requirements: * you have at least a few GB of storage for the data. * your networking environment does not block TCP connections between components. The installation procedure covers the server and the agents. ## Server The server binary is distributed as both `deb` and `rpm` packages. First of all, you need to [download](https://cdn.flopsar.com/) the server installation package for [your platform and architecture](/5.1/overview/supported-platforms-and-requirements#server). If you want to serve the workstation application from the Flopsar server, download `flopsar-workstation.zip` file too. For Debian based distributions you can install it by executing the following command: ```shell-session # dpkg -i flopsar-server-..deb ``` For Red Hat based distributions you can install it by executing one of the following commands: ```shell-session # rpm -ih flopsar-server-..rpm # zypper install flopsar-server-..rpm # yum install flopsar-server-..rpm ``` In both cases, the installation procedure performs the following operations: * installs `flopsar-server` binary * creates a `flopsar` user, which is used to run the server * adds a `flopsar` `systemd` service * creates a global configuration at `/etc/flopsar` ### Environment Installation When the server is installed, next you need to create a new Flopsar environment where the data will be stored. You have two options at your disposal, either bundled environment or separated one. {% tabs %} {% tab title="Workstation Included" %} Copy `flopsar-workstation.zip` to your destination machine. Now, you need to create a new Flopsar environment by executing the following command: ```shell-session $ flopsar-server create --app flopsar-workstation.zip ``` where `` is a path to the location where the server home directory will be created. If the command executes successfully, it will create a new directory at the specified path. {% endtab %} {% tab title="Workstation Excluded" %} You need to create a new Flopsar environment by executing the following command: ```shell-session $ flopsar-server create ``` where `` is a path to the location where the server home directory will be created. If the command executes successfully, it will create a new directory at the specified path. Note, since the workstation application is not served from the Flopsar server in this case, you need to download the `flopsar-workstation.zip` file and install it on your own web server. For example, in nginx web server you should add the following `location` sections to your `server` configuration: {% code title="nginx.conf" lineNumbers="true" %} ```nginx server { ..... location / { try_files $uri $uri/ =404; } location /api/ { proxy_pass http://localhost:9443; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "Upgrade"; proxy_set_header Host $host; } .... } ``` {% endcode %} The example above assumes that the Flopsar server listens to requests on `localhost:9443` socket address and the Flopsar server has its `ssl.enable` configuration option value set to `false`. Obviously, you should adjust the `proxy_pass` option to your own needs according to the [server configuration](/5.1/administrator-guide/server#ui). {% endtab %} {% endtabs %} {% hint style="warning" %} If you start the server using `systemd`, note that it will run it as `flopsar` user by default. Make sure this user has read/write permissions to the environment directory. {% endhint %} ### Server Configuration The server stores its [configuration](/5.1/administrator-guide/server#configuration) globally in `/etc/flopsar/settings.yml` file. You must edit this file to make at least two changes. The first one is required and you must accept the [EULA](https://cdn.flopsar.com/#/eula) by setting option `eula: true` otherwise, you will not be able to run the server. The second one is to set the `data` option, which points to you environment directory. {% hint style="info" %} This new environment will be created with some default settings. Adjust these settings to your needs accordingly. Please, refer to the [server configuration](/5.1/administrator-guide/server#configuration) for more details. {% endhint %} ### Running the Server Finally, you can start the server by executing the following command: ```shell-session # systemctl start flopsar ``` If you installed the workstation application bundled with the Flopsar Server, then you can access the application at the address specified at the [server configuration](/5.1/administrator-guide/server#ui). If you installed the workstation application outside the Flopsar server, you can access it at the address specified in your web server configuration. In either case, you should be able to access the workstation application login page at the address: `https://:9443`

Workstation Login Page

{% hint style="info" %} Default credentials are: `admin/flopsar` {% endhint %} ### Docker Image Alternatively, you can use a Docker image which has preinstalled the server along with the workstation. All you need to do is pull the image: ```shell-session $ docker pull flopsar/server:5.1.0 ``` and run the server by executing the following command: ```shell-session $ docker run -d -p 9443:9443 -p 9000:9000 flopsar/server:5.1.0 ``` or if you use `podman`: ```shell-session $ podman run -d -p 9443:9443 -p 9000:9000 docker.io/flopsar/server:5.1.0 ``` {% hint style="info" %} Please note, if the server is run in this way the EULA is accepted automatically. {% endhint %} ### Quick Sample Installation This section presents a sample, quick installation of the Flopsar server. First download the installation package along with the workstation application. Next, execute the following command: ```shell-session $ sudo dpkg -i flopsar-server-5.0.0.amd64.deb ``` or if you use `rpm` package: ```shell-session $ sudo rpm -i flopsar-server-5.0.0.x86_64.rpm ``` Next, let's create a new Flopsar environment at `/opt` directory: ```shell-session $ sudo flopsar-server create --app flopsar-workstation.zip /opt/flopsar ``` If the environment has been created successfully, the following output should be displayed: 
 Creating a new Flopsar Server environment...
 Unpacking Flopsar Workstation application...
 Flopsar Server environment created successfully.
Next, change the ownership of the environment directory to `flopsar` user: ```shell-session $ sudo chown -R flopsar /opt/flopsar ``` Next, open the `/etc/flopsar/settings.yml` file: {% code lineNumbers="true" %} ```yaml eula: false name: my environment data: /path/to/data max_ext_size: 1MiB agent: server_ip: 0.0.0.0 server_port: 9000 ui_srv: server_ip: 0.0.0.0 server_port: 9443 ui: bg_dark: '#141A1F' bg_light: '#312e81' fg_dark: '#cbd5e1' fg_light: '#f1f5f9' ssl: enable: true cert: server.crt key: server.pem logger: backend: Local max_size: 50MiB max_files: 5 archive: enable: false policy: purge start_at: 03:00:00.0 retention: 3days pass_policy: enable: true min_len: 8 digits: true special_chars: false capital_letters: false forbidden_chars: '' ``` {% endcode %} and make changes in both `eula` and `data` options (line 1 and 3): {% code lineNumbers="true" %} ```yaml eula: true name: my environment data: /opt/flopsar max_ext_size: 1MiB agent: server_ip: 0.0.0.0 server_port: 9000 ui_srv: server_ip: 0.0.0.0 server_port: 9443 ui: bg_dark: '#141A1F' bg_light: '#312e81' fg_dark: '#cbd5e1' fg_light: '#f1f5f9' ssl: enable: true cert: server.crt key: server.pem logger: backend: Local max_size: 50MiB max_files: 5 archive: enable: false policy: purge start_at: 03:00:00.0 retention: 3days pass_policy: enable: true min_len: 8 digits: true special_chars: false capital_letters: false forbidden_chars: '' ``` {% endcode %} And finally, start the server: ```shell-session $ sudo systemctl start flopsar ``` Now, you should be able to access the workstation application at [https://localhost:9443](https://localhost:9443/) ## Agent To install the agent, follow carefully the instructions below: 1. select the agent library file for your platform and architecture. 2. copy the agent library to your server machine. 3. configure the agent. 4. restart your application. # Upgrade ## Upgrade from 5.0 To upgrade from 5.0 version you must perform a few operations. First of all, you must [upgrade all the binaries](#binaries-upgrade). Next, request for a new 5.1 license file. First, you must update your `setting.yml` server file. You can do this by creating a new settings file by executing the following command: ```sh $ flopsar-server config . ``` This will save a default server configuration to a `settings.yml` file in the current directory. {% hint style="danger" %} Make sure your current `settings.yml` file is not in the current directory, since it will be overwritten by the above command. {% endhint %} Next, open this file and move your current settings to this new file. Finally, overwrite your old setting file with this new one. Alternatively, you can compare your current settings file and the new file and add all new options to your current settings file. Either way, you should have a [valid settings](/5.1/administrator-guide/server#configuration) file. Second, you must migrate your data to a new version. To do this, execute the command: ```sh $ flopsar-server migrate ``` This ends the migration and you should have a working Flopsar 5.1 environment. ## Binaries Upgrade ### Agent Upgrading the agent is simple, you just override the agent library with the new one. ### Server If you use an `rpm` package, then you upgrade the server installation by executing the following command: ```shell-session # rpm -U --noscripts flopsar-server-..rpm ``` If you use deb package, then you upgrade the server installation by executing the following command: ```shell-session # dpkg -i flopsar-server-..deb ``` ### Workstation If the workstation is served from the server itself, to upgrade it you need to execute the following command: ```shell-session # flopsar-server update --app flopsar-workstation.zip ``` where `` is a location of the server home directory. {% hint style="warning" %} Make sure, the user the Flopsar server is running on has appropriate permissions to the Workstation files after upgrade. {% endhint %} # Managing Users Users are managed in the workstation application. They can be added by another user appropriate permissions or they can signup themselves.

Sample users view

## User's Permissions A user authorization model is based on permissions. The permissions are divided into logical groups. ### Administration | Permisson | Description | | ------------------ | ---------------------------------------- | | Update license | Allows to change a license. | | Edit users | Allows to create, edit and delete users. | | Delete application | Allows to delete applications data. | | View server info | Allows to view the server info page | | View users | Allows to view other users | | View license | Allows to view the server license | ### Agents | Permission | Description | | ----------------------- | ------------------------------------------- | | View agents | Allows to view currently connected agents. | | Query agents | Allows to send queries to agents directly. | | Switch agent's features | Allows to enable/disable agents's features. | ### Configuration | Permission | Description | | --------------------- | --------------------------------------------- | | View profiles | Allows to view profiles and profiles' details | | Assign applications | Allows to assign applications to profiles | | Edit profiles | Allows to create, edit and delete profiles | | Edit profiles state | Allows to deploy/undeploy profiles | | Edit license mappings | Allows to edit the license mappings | | Manage extensions | Allows to add and delete agents extensions. | ### Data Access | Permission | Description | | --------------- | ---------------------------------- | | View parameters | Allows to view methods' parameters | ## Hiding Applications If you do not want a user to see data from particular applications, you can do this by banning the applications for that specific user. In order to edit the banned applications list click the `Banned applications` tab.

Banned applications view

Next, press the Edit button to make changes to the list.

Edit banned applications form

Then select the application and press the `Add` button. Finally, press the `Save` button to save the list. ## Password Recovery If there is another user who is able to access the server via the workstation application, your password can be reset by this user, provided that they have appropriate permissions to do so. If there is no such user, you must login to the machine the server runs on and reset your access using the command below: ```shell-session $ flopsar-server users --reset ``` This operation will set the password to `flopsar` and also grant the specified user all the permissions. # Managing a License When you do not have a license, you can still run the Flopsar environment with some limitations, i.e. you can collect data only from a single agent. If you want to collect data from multiple agents, you will need a valid license. This license must be uploaded to the server. You can do this from the License page.

License page

You need to press the `Update` button and upload the license by pasting its contents to the `License Content` field.

Registering a new license

The maximal number of agents, you can collect data from, is determined by the license you have. The license has also an expire date, unless you have a permanent one. When the license expires, all the connected agents get their license revoked. Only a single agent is left with its license. This implies the agents stop collecting nearly all data. ## License Mapping If you have less agents connected than your license allows, all of your agents will get the license granted and you do need to do anything with it. However, if this is not the case, only a part of your agents will get the license. Which agents get the license is done automatically by default. If you want to have a control over this process, you must then assign the license manually. {% hint style="warning" %} You need to have Edit license mapping permission to be able to edit the license mapping. {% endhint %} You will need to create a mapping, which specifies how many agents from a selected application should get the license. {% hint style="info" %} If you have more agents from the same application than the number you assigned for them in the mapping, only part of them will get the license. The agents, which get the license will be selected randomly. {% endhint %} # Logging ## Logging Levels Each Flopsar module (application) uses logging levels. The levels are defined as follows: **`debug`** : more detailed logging than `info`. Used to log some additional information about the application operation and data processing. Use only in cases when you suspect some problems with the application, want to trace how it processes requests or you are requested by the Flopsar Technology Support to help investigate some problems, or any other problems. It prints lots of diagnostic information, which might not be covered by this manual. **`info`** : default level used in applications. Used only to log some runtime information to verify a configuration and the like. You should use this level on normal, production operation. **`warn`** : used in cases when some error occurs but with no impact on the application stability and data integrity. **`error`** : normally, there should be no error messages in the log. The level is used in cases when some error occurs. These errors impact the application operation stability and data integrity but not all of them. The messages are informative, so you should know whether you can ignore them or not. Any system calls and external libraries errors are also logged in this level. ## Messages All logger messages have the following format: ``` DATETIME LEVEL [XXXXXXXX] LOG_ENTRY ``` where: `DATETIME` - date time of the log entry, `LEVEL` - log level, `XXXXXXXX` - first eight characters of the agent instance name, `LOG_ENTRY` - log entry ## Server Depending on the selected logger backend, the logs can be access in different ways. If the logger backend is `Local`, the server logs are stored in the `/logs` directory. If the backed is `Syslog`, you can access the logs by executing the command: ```shell-session $ journalctl -t flopsar -r ``` Please, refer to the [server configuration](/5.1/administrator-guide/server#logger) for more details on its logger settings. # Profiles A profile is a collection of settings and rules, which govern agents. It tells agents what data and metrics should be collected. Each profile needs to be activated to be eligible for deployment to agents. Which profile goes to which agent is determined by assigning applications to the profiles. There is a one-to-one correspondence between profiles and applications. In other words, the agents are not assigned to profiles directly. ## Profile State Machine Every profile can be in one of three states: `draft`, `ready` or `deployed`.

Profile State Machine

State-to-state connections refer to operations that are allowed on the states they originate from. ## Managing Profiles Profiles are managed on the `Profiles` page.

Profiles View

To add a new profile, you must select the `New` item from the `Options` menu, specify a name for the profile and press the `Save` button. Next, you can click the profile name in the list to view its [details](#profile-details). To refresh the profiles list, just select the `Refresh` item from the `Options` menu. {% hint style="info" %} A profile can be deleted only in the profile details page. {% endhint %} ## Profile The profile details page contains a selected profile detailed information, such as its settings, instrumentation rules and JMX list of MBeans.

Profile details view

### Profile Settings Each profile has some settings, which are either instrumentation or JMX related. The instrumentation settings are: * **`CPU Time`** this option enables/disables CPU time metric, * **`Automatic tuning`** this option sets one of the three available automatic tuning modes: `light`, `moderate` or `aggressive`. Automatic tuning is a feature which drops irrelevant method invocation reports automatically. If you want the automatic tuning algorithm to pass more methods invocation data from agents, just select the `light` mode. If you want to have less data, select higher modes like `moderate` or `aggressive`. * **`JMX Period`** this option sets the time interval at which JMX data is collected. ### Instrumentation Rules In order to make a profile deployable, you must define some instrumentation rules. You can do this in the `Instrumentation rules` tab.

Empty Instrumentation Rules View

Here, you can define your rules which can be either inclusive or exclusive (see [Rules](/5.1/user-guide/profiles/instrumentation#rules) for details). #### Inclusive Rules Inclusive rules are the ones that precisely define what classes and methods should be instrumented. To add such a rule, just press the `Inclusive Rule` button. A dialog window, with the rules wizard, should open then.

Inclusion Rule Wizard

The wizard is composed of three steps. In the first step, you need to specify what classes should be taken. The following options are available for the `Class level pattern`: * **`Package`** the pattern is a name of a package that classes should belong to, e.g. `com.foo.bar` * **`Classname`** the pattern is a fully qualified class name. Only those classes will be taken into account whose name matches the specified pattern, e.g. `com.foo.bar.MyClass` * **`Superclass`** the pattern is a fully qualified name of a class that classes should inherit from. Only those classes will be taken into account which inherit from the specified class directly, e.g. `com.foo.bar.MyClass` * **`Interface`** the pattern is a fully qualified name of an interface that classes should implement directly. Only those classes will be taken into account which implement the specified interface, e.g. `com.foo.bar.MyInterface` In the second step, you need to specify which methods should be instrumented.

Inclusion Rule Wizard

There are three ways you can choose from to specify the filter for methods for the `Pattern type`: * **`Access Modifiers`** use this pattern to select only those methods whose access modifier is at least one of the chosen modifiers * **`Name`** use this pattern to select only those methods whose name matches the specified value in the `Pattern` field * **`Name & Signature`** use this pattern to select only those methods whose name and signature matches the specified value exactly in the `Name and Signature` field. {% hint style="info" %} For **`Name & Signature`** option you must specify the value in the same way you would do it in your source code, except you must specify fully qualified names for classes here. {% endhint %} In the last step, you need to specify what kind of action should be performed when the instrumented method is to be executed.

Inclusion Rule WIzard

#### Actions Suppose you have a class defined below and the method `someMethod` is instrumented. You must select one action handler for your rule. {% code title="SampleClass.java" overflow="wrap" lineNumbers="true" %} ```java public class SampleClass { public int someField; private long someField2; private Foo someField3; public void someMethod(String arg1, int arg2, Bar arg3) { /* method block */ } } ``` {% endcode %} {% hint style="info" %} Not all actions are available since their availability depends on the selected `Pattern type` in the previous step. {% endhint %} What you get depends on the selected action handler. You have five choices at your disposal for the `Action handler`: **Basic** This action will report only the fact that a method has been called along with some basic information like duration, timestamp etc. **Standard** This action will report the same data as the above action and method arguments additionally. The method arguments will be presented as a result of executing `String.valueOf` function on each one of them. In the case of the sample class you will get: ```java // SomeClass instance String.valueOf(this) // someMethod arguments arg1 arg2 String.valueOf(arg3) ``` **Finer Standard** This action is an extended version of the action above. Additionally, if the method is an instance method it will report values of the instance object fields. Static fields will not be reported. In the case of the sample class you will get: ```java // SomeClass instance fields someField someField2 String.valueOf(someField3) // someMethod arguments arg1 arg2 String.valueOf(arg3) ``` **Finest Standard** This action is similar to the action above but in this case, if an argument is an object, its instance fields will be reported instead of its value. Static fields will not be reported. In the case of the sample class you will get: ```java // SomeClass instance fields someField someField2 String.valueOf(someField3) // methodSample arguments fields if any arg1 arg2 // Bar instance fields barField1 barField2 ... ``` **Custom** This action will execute a user defined custom code that is provided by the agent extensions and report its output along with the basic information. In this case, you decide what you get by defining your own [agent extension](/5.1/developer-guide/agent-extensions). {% hint style="warning" %} Please note, Finer Standard and Finest Standard can degrade your code execution performance so use it carefully. Custom handler can do this also if your implementation is poor. {% endhint %} #### Additional Features Depending on the selected `Action handler`, there are some additional features you can select in the `Features` section: * **`Always report`** this feature makes all the instrumented methods ignore the automatic tuning in the profile [settings](#profile-settings) * **`Instrument on exit`** this feature injects the instrumentation code at the end of the method body Additionally, if you selected the **`Execute custom code`** option, you will need to specify both a class name and a method name of your custom extension code, the agent is about to inject, in the `User-defined extension` section. A fully qualified name for the `Class name` field is required and a method name for the `Method name` field. If you are done with the rule, press the `Save` button to add the rule to the instrumentation rules list. #### Exclusive Rules By default, those methods are instrumented which fulfill the conditions defined by the inclusive rules. However, sometimes these rules are too wide, i.e. they let too many methods or classes in. In order to exclude some classes and methods from instrumentation, you need to define exclusive rules. You can do it by pressing the `Exclusive Rule` button. A new popup window will appear then.

Exclusion Rule Form

To add a new rule, you need to specify both class and method patterns. The following options are available for the `Class level pattern`: * **`Package`** the pattern is a name of a package that classes should belong to. Only those classes will be excluded from instrumentation which belong to the specified package. * **`Classname`** the pattern is a fully qualified class name. Only those classes will be excluded from instrumentation whose name matches the specified pattern, e.g. `com.foo.bar.MyClass` * **`Superclass`** the pattern is a fully qualified name of a class that classes should inherit from. Only those classes will be excluded from instrumentation which inherit from the specified class directly, e.g. `com.foo.bar.MyClass` * **`Interface`** the pattern is a fully qualified name of an interface that classes should implement directly. Only those classes will be excluded from instrumentation which implement the specified interface, e.g. `com.foo.bar.MyInterface` For the `Method level pattern` you need to specify one of the following options: * **`Method name`** those methods will be excluded from instrumentation whose name matches the specified value in the `Pattern` field * **`Method signature`** those methods will be excluded from instrumentation whose signature matches exactly the one specified in the `Pattern` field {% hint style="info" %} If you want to exclude all methods from a class, select the **`Method name`** pattern, put `.*` into the `Pattern` and select the `regex` option. {% endhint %} If you are done with the rule, press the `Save` button to add the rule to the instrumentation rules list. #### List of Rules When you add new rules, they are presented in a list. The rules are evaluated in the order of their appearance in the list (see [Rules Evaluation Order](/5.1/user-guide/profiles/instrumentation#rules-evaluation-order) for more details).

Instrumentation Rules

You can click each rule to see its details. Exclusive rules are denoted by red squares, while inclusive ones are denoted by green squares. If you want to edit a rule, just click the pencil icon in the right down corner of the selected rule. If you want to delete a rule, click the trash icon in the right down corner of the selected rule. To clone a rule, click the copy icon in the right down corner of the selected rule. A rule can be in one of the two states: enabled or disabled. You can switch the rule state by clicking the `Enable rule` switch. When a rule is disabled, it will not be deployed to agents. Each rule has also a description, which is self-explanatory. ### JMX This view enables specifying which MBeans values you want to collect. To add MBeans, you need to retrieve them from agents first. You can do this by pressing the `Add ObjectNames` button.

JMX View

This operation will open a new popup window.
Next, press the `Refresh` button to retrieve the MBeans. {% hint style="info" %} You must first assign an application to your profile to be able to retrieve MBeans from the connected agents. {% endhint %}
Now, you can press the `Add` button for any MBean you want to select. When you are done with adding the MBeans either click outside the popup window or press the **`Escape`** button to close the popup window. ### Deploying a Profile To deploy a profile, a few conditions must be fulfilled. The profile must have instrumentation rules defined, must be assigned to an application and be in the `ready` state. # Instrumentation Instrumentation is a JVM feature, which enables modification of the byte-codes of methods. ## Limitations There are some limitations imposed on the instrumentation, i.e. not every entity can be instrumented. These limitations are hard-coded in agents and you cannot change them. There are both class and method limitations. ### Classes Classes from the following packages are always excluded from instrumentation: ```java com.flopsar.* com.sun.* javassist.* java.* javax.* jdk.* org.apache.bcel.* org.objectweb.* sun.* ``` Additionally, the following class entities are also excluded from instrumentation: * interfaces * enum classes * annotation classes * modules * synthetic classes * primitives * arrays ### Methods The following methods are always excluded from instrumentation: * abstract methods * synthetic methods * native methods * constructors * class initializers ## Rules To instrument methods, you need to decide which methods should be instrumented. This is governed by a set of rules. There are two types of rules: *inclusions* and *exclusions*. You use inclusion rules to specify what should be instrumented and exclusion rules to specify what mustn't be instrumented. Each rule has to define both a class and a method filter. The class part specifies which classes should be taken into account and the method filter specifies which methods, from these classes, should be instrumented and what data they should collect. This implies that multiple rules can instrument the same class.

Multiple rules instrumenting the same class

### Rules Evaluation Order All the rules are evaluated in a very specific order. When a class is passed to the rules filter the exclusion rules are evaluated first. If the class part of the exclusion rule matches, the method part is checked next. If both class and method match the rule, the method will not be instrumented and other rules will not be evaluated for this method. #### Example Suppose we have a rule set consisting of four rules: two exclusive and two inclusive. Let's analyze how a class is evaluated against the rules. The pictures below will help us to visualize the flow. Each row in the picture represents a single rule evaluation flow. Red arrows represent an exclusive rules flow, while the green ones represent the inclusive rules flow. First column represents the class part of a rule while the second column the method part of a rule.

Sample Rules Evaluation

Suppose we have a class `A`, which is passed to the rules filter. The class has four methods. First, the exclusion rules are evaluated. The first exclusion rule matches the class, so the method part of the rule is evaluated next. As we can see, the `method_4` matches the method part of the rule. That means, this method is excluded from instrumentation and it becomes invisible to the rest of the rules. Next, the second exclusion rule is evaluated. Again, we have a match for the class part but this time we do not have a match in the method part of the rule. That means, the class with the remaining methods is passed further. Since, there are no more exclusive rules, we get to the first inclusive rule (the third row). Here we have a match in the class part of the rule, so we move to the method part where we have a match for the `method_2`. This means, that method will be instrumented by the first inclusive rule and it becomes invisible to the last rule. Finally, we get to the last inclusive rule. Again, we have a match in both the class and method part. This time `method_3` is matched. This method will be instrumented by the last inclusive rule. There left only one method `method_1`, which did not match any of the rules. This method will not be instrumented. The final result of the evaluation is that two methods will be instrumented by different rules and the rest of the methods will be left unchanged. ## Retransformation Retransformation is a feature which enables to reload a collection of classes. This operation is preformed when you deploy a profile. Unfortunately, this operation is expensive and even though the instrumentation itself is blazingly fast, the internal JVM mechanics involved into the retransformation process takes much more time. {% hint style="warning" %} During the retransformation process you application performance will be degraded and your application can become unresponsive. {% endhint %} That is something you should always take into account when deploying profiles, e.g. try not to deploy profiles when your application is under heavy load. # Data Browser If you want to explore method invocations and their collected data, you should use the **`Data Browser`** page. ## Browser Form

Data Browser Form

There is only one mandatory field you must fill to make a query and retrieve the result. You must specify the date range in the fields `Date` and `Time Interval`. This will retrieve all available data from the specified time range. ### Additional Options If you want to narrow your search results to an application, you should specify the application name in the `Application` field. This will retrieve methods invocations only from the specified application. To narrow your search results even more, you can fill out the `Pattern` field. The specified value is a pattern to search for in methods invocations data. You need to select one of the four types: * **`contains`** the pattern will match when the methods data contains the pattern value, * **`exact`** the pattern will match only if the methods data are exactly the same as the pattern value, * **`starts`** the pattern will match when the methods data starts with the pattern value, * **`regex`** the pattern will match when the methods data satisfies the regular expression in the pattern value. {% hint style="info" %} If the `Pattern` field is filled, the search query will retrieve any method invocation that satisfies the query conditions, otherwise only root invocations of execution stacks will be retrieved. {% endhint %} If you want to retrieve only those invocations that were executed within a particular thread, just specify the value in the `Thread name` field. ## Search Results The search results are presented in a form of a list. Each list record presents a single method invocation. If you want to see more details about the selected method invocation, just click the method signature and you will get details of the execution stack this method is a member of.

Browsing Sample Results

### Execution Stack Details This view presents an complete execution stack of the selected method.
#### Execution Stack The method execution stack is depicted as an inverted stack, where a root call is located at the top and all consequent invocations are placed on each other. Each bar in the stack denotes a single method invocation. The width of the bar denotes the method duration and it is scaled relative to the root call bar width. The time flows from left to right, hence the order of bars on the same level matters. Each bar has one of four background colors. If an invocation has no additional data, its background is grey. If the invocation have only parameters its background is green. If it has only an exception its background is red then. It if has both an exception and parameters its background is orange.

Sample Execution Stack

The stack is interactive, if you click one of the bars you will get its method invocation details, below the execution stack. The method is identified by its signature and a class it belongs to. There are two *copy* icons, next to both `Class name` and `Method` labels. If you click the icon, the corresponding value will be copied to the profiles clipboard, which can be used in [defining instrumentation rules](/5.1/user-guide/profiles#inclusive-rules). #### Spectrum This view presents a list of all methods called within the execution stack. The methods are aggregated and sorted. You can sort them with respect to one of the three criteria: self time, calls count and exceptions count.
# Data Analysis This view is useful when you want to analyze your collected method invocations. There are only two fields that need to be fill out. You must specify both the date range and the application. Then, the analysis will be performed for those method invocations which ended within the specified time range and came from the specified application.

Data Analysis Form

## Duration Distribution The first part of the analysis presents the methods duration distribution with respect to percentiles. The chart below presents a sample analysis result. ### Plot The `Plot` tab presents a methods overall duration distribution. The analysis is performed for all methods invocations within the time range specified in the data analysis form.

Sample Duration Distribution

The chart is divided into percentile sections. The percentile values are located on the `x`-axis. The left `y`-axis presents the method duration value, while the right one the number of method invocations. Each percentile section has a bar which represents the number of method invocations within the percentile interval. The blue curve represents the methods duration. To properly read the chart, you need to focus on the percentile axis (`x`-axis). Note, the `x`-axis is not scaled linearly. Suppose now, we have the analysis results as the one above in the picture. We interpret the chart at `96.881th` percentile as follows: * `96.881` percent of all invocations execute below `233` ms. * there are about `8300` invocations whose duration is between `170` and `233` ms. Note, `170` ms is taken from the previous percentile, which is `93.757` in this case. It is easy to see, in the sample case, that the number of methods invocations decreases rapidly as percentile grows. There are only a few outliers, whose duration is above a minute. It is up to a user whether a further analysis is required or not. If a more detailed investigation is needed a user can proceed with the analysis by pressing the `Temporal analysis` button. ### Summary The Summary tab allows a user to compare the analysis results to the data from previous periods. This time a few specific percentiles are selected to compare them with the ones from previous periods.

Sample Analysis Summary

There are three special columns, in both `Duration` and `Count` columns. These columns are denoted as `1d`, `3d` and `1w` which mean 1 day, 3 days and 1 week respectively. They contain the corresponding changes of percentile values one day ago, three days ago and one week ago respectively with respect to the current values. In order to retrieve the values you need to click the refresh button in the selected column header. The values are then compared with the current ones within the corresponding time range but from previous days. For example, if you want to compare the current results with the ones from 3 days ago, just click the refresh button in the `3d` column header. If the current duration value is less than three days ago, the value change is displayed in green, otherwise it is displayed in red. In the picture above, one can easily see that the current duration value for the 75th percentile is 63 ms. This result is 17 ms less than three days ago. Moreover, currently we have 200,639 invocations, while three days ago there were 40,038 less invocations. The percentile value in the fourth row is customizable. It can be changed by selecting one of the available values in the `Custom percentile` buttons group. ## Temporal Analysis This part of the analysis presents how the percentile values change in time for the specified period. There are two tabs with the results. The first one `Values` presents how the percentile values change in time. Just like in the duration distribution there are bars and curves. The bars represent the amounts of invocations in time periods. The curves present the percentile values of duration. There are four percentiles curves that can be displayed simultaneously, i.e. 25th, 50th, 75th and 9\*th. The last one is customizable and can be changed in the `Highest percentile` option.

Sample Temporal Analysis

The second tab `Sections` presents how many invocations are in each percentile. If you want to know what methods are reported in a selected interval, just move your mouse over one of the percentile curves and click the curve.

Sample top contributors

## 2D Analysis The 2D analysis is represented in a form of a heatmap chart. The chart presents a distribution of one of the selected `Data type` with respect to both duration (vertical axis) and time (horizontal axis). The chart is divided into cells, which aggregate data for particular ranges of duration and time. For `Count` data type, each cell represents a number of invocations within the cell range. For `Off-Cpu` data type, each cell represents a percentage of method invocations, whose off-cpu time is the largest. For `Exceptions` data type, each cell represents a number of methods that throw exceptions.

Sample 2D Analysis

You can change the number of cells the chart is divided into, by selecting one of the `Resolution` value. You can also change the range of methods that are analyzed by selecting one of the `Precentile` values. For example, if the `Percentile` value is 50, it means that the chart shows only those method invocations whose duration value is below the value at 50th percentile. ## Exceptions This view presents top ten [root exceptions](/5.1/user-guide/glossary#execution-stack) thrown by methods.
# Metrics This page allows to plot graphs of methods invocations and other key-value metrics. For methods graphs, you need to select the `Method` option in the `Metric Type` field. Next in the `Class` field, you need to specify a pattern for the class name you search for. You must specify at least three characters to retrieve a list of classes whose names contain the specified characters. The pattern you specify will be sought in the class names only, without the package part. If you select one of the classes, a list of its methods will be retrieved.

Metrics Form

For key-value graphs, you need to select the `Key-Value` option in the `Metric Type` field. Next in the `Key` field you need to specify a pattern to search for in the available metrics. You need to specify at least three characters to retrieve a list of the metrics keys. # Data Masking In situations where a profile is set up in such a way that agents report methods and their parameters, it is possible that some of these parameters may contain sensitive data. To ensure that such data remains concealed and is not visible to unauthorized individuals, you can apply masking techniques. To mask the data just click the `Tools` link in the side bar.

Sample data masking

Then, follow the steps below to mask the data: 1. Press on the `Add` button located in the top right corner. This will open the data masking settings. 2. In the data masking settings, you can specify the parameter key for masking sensitive data. 3. Save the data masking settings once you have defined the desired rules. By following these steps, you can effectively mask sensitive data contained within agent report parameters, thus ensuring data privacy and security. # Agents Each agent instance is assigned to an application. You can view all the connected agents in the `Applications` page. You just need to select the application and you will get a list of all agent instances of this application.

Sample application agents

All the agents instances are presented in a table. All the table columns are self-explanatory. The first `Name` column contains an instance name and some additional information. The green icon in front of the instance name denotes that this agent has an assigned license. If the name of the instance is in green, it means that this instance has a profile. If you click the instance name you will get a detailed view of the selected agent instance. ## Agent Details Each agent provides some information about its operations and the JVM it runs within. The data about the agent and the JVM are updated every 5 seconds. The data are not persisted anywhere so they are delivered as long as the agent is able to maintain its connection to the Flopsar server.

Sample agent details

Nearly all the presented data are self-explanatory. The `Errors` graph shows the total number of the agent errors, which can be investigated in the [Diagnostics](#diagnostics) tab. ### Errors Detection If an agent detects a JVM error of `java.lang.Error` it is displayed at the top of the agent details view.
Agents do not detect all errors of `java.lang.Error`. Only a subset of the most important errors are reported, such as: * `VerifyError` - thrown when the class [verifier](https://docs.oracle.com/javase/specs/jvms/se17/html/jvms-4.html#jvms-4.10) detects that a class file, though well formed, contains some sort of internal inconsistency or security problem. If this error is caused by the agent instrumentation, you must exclude the offending class from your profile. * `NoClassDefFoundError` - thrown if the JVM or a ClassLoader instance tries to load in the definition of a class (as part of a normal method call or as part of creating a new instance using the new expression) and no definition of the class could be found. * `ClassFormatError` - thrown when the JVM attempts to read a class file and determines that the file is malformed or otherwise cannot be interpreted as a class file. If this error is caused by the agent instrumentation, you must exclude the offending class from your profile. * `StackOverflowError` - thrown when a stack overflow occurs because an application recurses too deeply. * `OutOfMemoryError` - thrown when the JVM cannot allocate an object because it is out of memory, and no more memory could be made available by the garbage collector. * `UnsupportedClassVersionError` - thrown when the JVM attempts to read a class file and determines that the major and minor version numbers in the file are not supported. * `VirtualMachineError` - thrown to indicate that the JVM is broken or has run out of resources necessary for it to continue operating. * `ClassCircularityError` - thrown when the Java Virtual Machine detects a circularity in the superclass hierarchy of a class being loaded. * `UnsatisfiedLinkError` - thrown if the JVM cannot find an appropriate native-language definition of a method declared native. * `NoSuchMethodError` - thrown if an application tries to call a specified method of a class (either static or instance), and that class no longer has a definition of that method. * `NoSuchFieldError` - thrown if an application tries to access or modify a specified field of an object, and that object no longer has that field. If you want to find out more about the detected error, just click the error link and you will get a popup window with more details.

Sample error messages

You can always clear the errors from the agent registry by pressing the `Clear` button on the errors warning banner. ### JVM Memory Each agent collects JVM heap memory usage. By default, agents collect only the overall memory usage. If you want to know which objects occupy the memory, just press the `Enable heap sampling` button and wait for the data.

Sample heap memory usage

{% hint style="warning" %} Please note, the heap sampling may have a negative impact on your application performance, so do not use it over a long period of time. {% endhint %} ### JVM Threads Each agent collects JVM threads count. If you need to know what threads are currently

Sample threads sampling

{% hint style="warning" %} Please note, the threads sampling may have a negative impact on your application performance, so do not use it over a long period of time. {% endhint %} ### Diagnostics The agent does not have log files, instead it maintains a list of errors that can occurred in the agent. These errors are divided into logical groups. To see which errors have occurred, just click one of the bars in the graph.

Sample diagnostics view

Each error code is presented in a form `NNLNN`. It is divided in two parts with `L` as a separator. When checking a code, you should only check the first part `NNL*` of the code. Since the same errors can occur multiple times, only the last error date and message are presented. The `Count` value shows how many times the error occurred. The error message is optional and can be empty in some cases. #### Server Communication Errors
Error codeDescription
103L*Data retrieved from the server is corrupt.
104L*Agent and server uses different versions of the data protocol.
#### Instrumentation Errors
Error codeDescription
402L*Class bytes parse failed.
403L*Classes retransformation failed.
406L*Some method will not be instrumented.
#### JVMTI Errors Agents call multiple JVMTI functions, which can return errors. These errors are reported by agents. In order to map the agent errors `2XXL*` on JVMTI ones, you need to take into account only `XX` part of the error, which is a hexadecimal value of the original JVMTI error.
Error codeDescription
20AL* - 274L*JVMTI errors
#### Internal Communication Errors
Error codeDescription
501L*Some data cannot be passed to the outbound queue.
#### Internal Errors
Error codeDescription
301L*Some resource is currently unavailable.
302L*JVM does not support a capability.
303L*Some class constructor cannot be found.
304L*Some class has not been found by the agent.
305L*Some method has not been found by the agent.
306L*Some class object cannot be created.
308L*Some thread is detached from JVM.
309L*Failed to attach to JVM thread.
30AL*Some exception has been thrown.
# Managing Extensions The [agent extensions](/5.1/developer-guide/agent-extensions) are managed and stored in the server. You can access the management view by clicking the `Tools` link in the side bar.

Sample extensions

The `Extensions` tab presents a list of all available extensions, stored in the server. If an extension has an icon next to its class name, it means there is a source code of this extension, which you can see by clicking this icon. In order to add new extensions click the `New extension` item in the `Options` menu. Then, select the jar file with the extension classes. If the jar file contains valid extensions, you should get a view with detected extensions.

Sample new extensions

You can verify that the extensions you upload are the ones you wanted before you send them to the server. When a new extension class is already present in the server registry, you will see a red icon in the `Overwrite` column. If the extension has its source code attached, you will see a green icon in the `Source attached` column. If everything is fine, just press the `Accept` button to upload the extensions. {% hint style="info" %} There are sample extensions ([http](https://github.com/flopsar/plugin-http) and [jdbc](https://github.com/flopsar/plugin-jdbc) plugins) available publicly, which can be found at our [GitHub](https://github.com/flopsar) repos. {% endhint %} # Glossary Throughout the entire documentation, we use some terminology regarding executions of methods. ## Execution Stack The picture below presents a single method execution stack. The first element of the tree is called ***Root Call***. All the methods' invocations, down the execution stack, are called simply ***Calls*** and ***Invocations*** interchangeably***.*** The entire structure is called ***Execution Stack***. When an exception is thrown by some of the methods and propagated through the stack, the origin exception is called ***Root Exception***.

Execution Stack

### How the Stack is Assembled The picture below depicts the procedure of how the execution tree is built. In the picture below you can see four, sample execution trees of some method *`A`* call.
The first `a` view shows how the real execution stack looks. This is a real execution flow that takes place in your application. The second `b` view presents the same stack instrumented by your agent. Depending on your configuration, the instrumentation may not cover the entire stack. The empty, grey boxes represent missed, not instrumented methods. As you can see, at this very point your instrumented execution flow is not complete. Moreover, in your application runtime, there can be situations when some of the instrumented method calls will not be reported. This empty, violet box in the third `c` view represents this situation. Finally, what you actually observe in the workstation is the last `d` view. This view presents an assembled stack with missing calls ignored. Now you know how to interpret the execution stack in the workstation and why the resulting stack can differ from the original one. ### Self-time Self-time is a part of a method duration, in which the method executes its own code without calling any other methods. Suppose we have a situation depicted below. A method `A` executes within `duration_a`. However, this method calls another method `B` and waits while the method `B` is executing its own code. Then, when the method `B` returns, the method `A` can continue executing its code.

Self-time

It's easy to see that the method `B` execution time contributes to the method `A` execution time and the difference between method `A` duration and method `B` duration is the **self-time** of the method `A`. # Agent Extensions An agent can be used standalone or it can be extended by user custom code. Flopsar Agent is a powerful engine, which is extensible by plugins. The extensions are some extra jar files, which consist of user-defined POJO classes. The jars can be then uploaded to the Flopsar server. The plugins enable to extend the agent capabilities. We provide a mechanism that you can use to extend your application diagnosis and trace your data flow. It makes Flopsar an open tool, that can be extended. The plugin allows for processing data inside instrumented methods. There are two types of extensions, which can be used: **standard** and **custom**. ## Standard The **standard** extension is embedded in the agent. This extension will execute the following method on each argument it gets: ```java public static java.lang.String valueOf(java.lang.Object); ``` and return a result in a form of concatenated pairs of parameter identifiers and the corresponding `valueOf` function result. The following parameters are returned (in order of appearance): * `THIS` : If the instrumented method is static, this parameter is `null`, otherwise a class instance the method is executed on. * `ARGS` : List of all arguments of the instrumented method. ## Custom If you are not satisfied with standard extension and you want to extract some additional information (or make some preprocessing) you must implement your own, `custom` extension. In order to take advantage of this feature, all you have to do is set up a configuration, create a simple POJO class and implement a single method: ```java public static java.lang.String myFormatter(java.lang.Object[] args) ``` {% hint style="warning" %} You can give any name to the above method but the signature of the method must be preserved. Otherwise, it will not work. {% endhint %} The `args` argument of the extension method is an array of the instrumented method arguments delivered to your method. These are your original application objects, not clones. When you set up the configuration correctly, the agent will inject your code into the instrumented methods during the instrumentation. It can be injected in one of the two places, i.e. at the beginning or the end of the method. Note, your formatter code is executed inside your instrumented method, so make sure your formatter code generates minimal overhead. The argument `args` is an array of the following objects (in order of appearance): * **`args[0]`** If the instrumented method is static this element is `null`, otherwise it is a reference to a class instance the instrumented method is executed on. * **`args[1..N]`** List of all arguments of the instrumented method in the same order as they appear in the method signature. These are the references to your original application objects, not some *clones*. where **N** is the number of the instrumented method arguments. When you set up the configuration correctly, the agent will inject your formatter code into the instrumented methods during the instrumentation. It can be injected in one of the two places, i.e. at the beginning or the end of the method. Note, your extension code is executed inside your instrumented method, so make sure your code generates minimal overhead. In fact, it is up to you what extra features the agent will have. You can even extend your agent installation to cover a distributed transaction tracing. {% hint style="info" %} Use `U+001E` as a separator in your code. Otherwise, the result will not be formatted properly in the workstation. {% endhint %} Your result should always consist of pairs of keys and values. `U+001E` separates each entry in the result, which means you should always have an odd number of separators. The resulting string size cannot be greater than 8192 B, otherwise it will be truncated. There are some tips, you should take into account when writing extensions: * Design and implement extensions with performance in mind. If your code performs poorly, obviously the instrumented methods will perform poorly as well. * Use `try-catch` block to protect your application from any errors you can make in your extension implementation. Otherwise, any errors inside the extension code can interfere with your application processing. * Do not modify arguments delivered to your extension since they are your genuine application objects. Make sure that whatever you do with them will not cause any problem to your application flow. * Try not to use reflections, unless there is no other way. * Try not to create too many new objects, otherwise you risk increased GC overhead. * Deploy your extension classes into separate jar files. This can save you a lot of trouble when dealing with class loading issues. * Your extension code should be stateless. ## Example Suppose, you have some method: ```java public FooBar foo(foo.bar.Object1, foo.bar.Object2, ...); ``` and you are interested in some additional information extracted from the first and second arguments. Your extension implementation can look like the one below: ```java public static java.lang.String myFormatter(java.lang.Object[] args){ final char SEPARATOR = 0x1E; try { foo.bar.Object0 _this = (foo.bar.Object0)args[0]; foo.bar.Object1 obj1 = (foo.bar.Object1)args[1]; foo.bar.Object2 obj2 = (foo.bar.Object2)args[2]; /* implement your logic here to produce string output. */ String output1 = some_logic1(obj1); String output2 = some_logic2(obj2); return "Object1"+SEPARATOR+output1+SEPARATOR+"Object2"+SEPARATOR+output2; } catch(Throwable ex){ return "Error"+SEPARATOR+ex.getMessage(); } } ``` {% hint style="warning" %} Flopsar Technology cannot be held responsible for any damage done by badly written user's extensions. {% endhint %} # Server API {% openapi src="/files/aSPJMzPGymNd79jvoQAY" path="/api/auth/login" method="post" expanded="false" %} [api.yml](https://4184112761-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1l0Vb5GSU1RRV2waQeIU%2Fuploads%2FFDHSwo1UvVpcx4faA04t%2Fapi.yml?alt=media\&token=12d66233-ccae-4d7e-8d88-32f9cccb6df6) {% endopenapi %} {% openapi src="/files/aSPJMzPGymNd79jvoQAY" path="/api/apps" method="get" %} [api.yml](https://4184112761-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1l0Vb5GSU1RRV2waQeIU%2Fuploads%2FFDHSwo1UvVpcx4faA04t%2Fapi.yml?alt=media\&token=12d66233-ccae-4d7e-8d88-32f9cccb6df6) {% endopenapi %} {% openapi src="/files/aSPJMzPGymNd79jvoQAY" path="/api/data/symbols/{kind}" method="post" %} [api.yml](https://4184112761-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1l0Vb5GSU1RRV2waQeIU%2Fuploads%2FFDHSwo1UvVpcx4faA04t%2Fapi.yml?alt=media\&token=12d66233-ccae-4d7e-8d88-32f9cccb6df6) {% endopenapi %} {% openapi src="/files/aSPJMzPGymNd79jvoQAY" path="/api/data/calls" method="post" expanded="false" %} [api.yml](https://4184112761-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F1l0Vb5GSU1RRV2waQeIU%2Fuploads%2FFDHSwo1UvVpcx4faA04t%2Fapi.yml?alt=media\&token=12d66233-ccae-4d7e-8d88-32f9cccb6df6) {% endopenapi %} # What is Flopsar? {% hint style="info" %} End of life for Flopsar 5.2. This product is no longer supported. {% endhint %} Flopsar, developed by Flopsar Technology, is the cutting-edge Fault Detection and Diagnosis software for JVM systems. Our mission is simple: to detect errors and problems in your Java systems swiftly and assist in identifying the root cause effectively. In the world of systems maintenance, availability and responsiveness are paramount. At Flopsar, we understand that minimizing system outages and swiftly resolving issues are critical factors. When a system fails, it's not enough to know that it failed—we must know why. Gone are the days of spending hours or even days searching for the cause. We tackle these challenges head-on. Unlike traditional APM tools, Flopsar eliminates the need for users to predict potential system issues in advance. We don't settle for vague averages or aggregated data that leave you guessing about your application's processing. Flopsar provides you with a highly customizable and extensible product that adapts to your unique needs. It seamlessly integrates with any framework and application server, making it truly agnostic. But Flopsar offers much more than that. Our powerful features not only enhance systems maintenance but also drive successful application development processes. With Flopsar, you can extend its capabilities using plugins, allowing you to perform additional processing at runtime without making any changes to your application source code. # Supported Platforms and Requirements What platforms and architectures are supported. ## Agent Agent is a native library, supporting the following platforms:
PlatformArchitectureJVM supportlibcARM
GNU/Linuxamd64 (x86)1.7 - 172.14+false
GNU/Linuxaarch64 (ARM)1.8 - 172.31+false
Windowsamd64 (x86)1.7 - 17-false
## Server The server application is distributed for the following platforms:
PlatformArchitecturelibcARM
GNU/Linuxamd64 (x86)2.18+false
GNU/Linuxaarch64 (ARM)2.31+false
There are no fixed system requirements to run the server, since it strongly depends on your controlled environment. The server runs multiple jobs like storing, collecting, analyzing and serving the collected data. The server is designed to take advantage of multiple CPU cores, so the more cores the better. A reasonable minimum should be at least 2 CPU cores and 4GB of RAM. However, if you notice that the server experiences some performance issues like delays in serving and pushing data, consider to bump the system resources for the server. ## Workstation This is a web application, that can be deployed on any web server that supports web sockets and is able to run as a reverse proxy. {% hint style="warning" %} The Workstation application requires browsers with native **ECMAScript 2015 (ES6)** support. {% endhint %} If you experience some issues with the application try to install a newer browser. # Versioning Every version label is composed of three numbers `X.Y.Z`: `X` Major version number: major changes, e.g. architecture change. `Y` Minor version number: minor changes, e.g. new features. `Z` Small version number: bugs fixes. {% hint style="info" %} Versions `X.Y` are equivalent to `X.Y.0` {% endhint %} --- [Next Page](/llms-full.txt/1)