> For the complete documentation index, see [llms.txt](https://docs.flopsar.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.flopsar.com/user-guide/forensic-investigations.md).

# Forensic Investigations

How to prepare Flopsar for, and carry out, investigations that start from a phrase reported by an end user.

{% hint style="info" %}
[Investigation](https://docs.flopsar.com/user-guide/observing-and-diagnosing/investigation.md), evidence kept with investigations, and the audit records of searches by parameter value described on this page are available since Flopsar **7.2**. The preparation steps apply to earlier versions as well.
{% endhint %}

Some teams use Flopsar to reconstruct what happened to a specific user or transaction after the fact. A typical request reads: *"I logged in at about 10:15, tried to submit the order, and it failed"*, together with an identifier — a login, an order number, a reference shown in an error message. The support team then looks for every recorded invocation in which that phrase appears and reads the execution stacks to establish what the application did.

Flopsar supports this workflow, but only if it was configured for it **before** the incident happened. The agent records what the deployed profile tells it to record, automatic tuning drops part of the fast invocations, masking makes some values unsearchable, and retention removes old data. This page explains what to set up in advance, how to run the search, and how to interpret the result — in particular, why *"not found"* does not prove *"did not happen"*.

{% hint style="warning" %}
Flopsar is a diagnostic tool. Its data is a selective, assembled record of instrumented method executions, not a complete or tamper-proof transaction log. Use it to explain what happened; do not use it as the sole evidence that something did **not** happen. See [Interpreting the result](#interpreting-the-result).
{% endhint %}

## Preparing the environment

Work through these points for every application you expect to investigate. Each of them is a deliberate trade-off between investigative value, overhead, and data protection, so agree on them with the application owner and, where personal data is involved, with your data protection officer.

### 1. Record the operations users report

Identify the entry points that correspond to actions users talk about — signing in, submitting a form, placing an order, starting a payment — and add an **Allow** rule for each in the application's [monitoring profile](https://docs.flopsar.com/user-guide/setting-up-monitoring/monitoring-profiles.md):

* Use the **Name & Signature** method pattern and the **Standard** or **Extended** match action, so that the method's **arguments** are captured. With the Access Modifiers or Name pattern only timing is recorded, and there is nothing to search for.
* Make sure that the identifier you will be given is actually among the captured values. If it is not passed as an argument of any instrumented method (for example, it lives in an HTTP header or in the session), record it with a [plugin](https://docs.flopsar.com/user-guide/plugins.md) on the same method.
* Prefer methods whose arguments carry business meaning (`placeOrder(Order)`, `login(String)`) over generic framework methods, whose arguments are often opaque objects.

### 2. Keep these invocations from being tuned away

To limit overhead, the agent does not report every invocation it observes:

* Invocations shorter than 1 µs are always discarded.
* With **Auto Tuning** set to **light**, **moderate**, or **aggressive**, an invocation is discarded when its duration is below 5%, 10%, or 25% respectively of the average duration of the root method of its stack. When the root invocation itself qualifies and nothing beneath it was reported, the whole stack is discarded.
* Invocations that end with an exception are never discarded by these mechanisms.

A successful, fast sign-in is exactly the kind of invocation that tuning removes, so it may simply be missing from the data. Enable **Always report** on every rule from the previous step; it makes the agent report the matching invocations regardless of their duration and of the tuning level. Disabling Auto Tuning for the whole profile has the same effect for all rules, at a higher overhead cost.

### 3. Capture a shared key that links the steps

A user's story usually spans several requests — sign-in, navigation, the failing action — possibly in more than one application. To follow it, at least one captured value must be common to all of them: a session id, a correlation or request id, or a user id. Capture it on every rule from step 1. **Show correlated calls** can then list all root calls that share it.

### 4. Decide what must remain searchable

[Data masking](https://docs.flopsar.com/user-guide/setting-up-monitoring/data-masking.md) is applied when data is stored and is irreversible. A masked value is replaced with asterisks, so it **cannot be found by search** either. The built-in patterns cover, among others, e-mail addresses, phone numbers, PESEL, NIP, IBAN, and payment card numbers — the very identifiers users tend to report.

* Prefer a non-personal identifier for investigations — an internal user id, session id, order number, or a reference code displayed to the user in error messages — and ask the reporter for that.
* Do not remove or weaken masking patterns only to make data searchable without the approval of whoever is responsible for data protection. Mask changes are recorded in the audit trail.
* Remember that masking applies to captured arguments and OpenTelemetry attributes, **not to exception payloads**. See [Known Risks & Limitations](https://docs.flopsar.com/security/known-risks-and-limitations.md).

### 5. Keep the data long enough

Reports often arrive days after the event. Data older than the retention period is removed by the daily retention procedure configured in the [`[archive]` section](https://docs.flopsar.com/administrator-guide/configuration/server-configuration-reference.md#archive) of `settings.toml`; the default retention is `72h`.

* Set `retention` to cover your typical reporting delay, and size the server's storage accordingly.
* Calls and other data you add to an investigation as evidence are copied and are not removed by retention; see [Keeping evidence](#keeping-evidence). *Available since Flopsar 7.2.*
* With `policy = "archive"`, expired data is written to a ZIP file instead of being deleted. Archived data is **not searchable from the Workstation**; to inspect it, open the archive in local mode as described in [Archives](https://docs.flopsar.com/administrator-guide/server/archives.md).
* Balance retention against data minimization; see [Privacy & Data Handling](https://docs.flopsar.com/trust-and-compliance/privacy-and-data-handling.md).

### 6. Control who can investigate

Searching by phrase means reading captured arguments, which may contain personal data.

* Grant **View data** and **View parameters** only to the people who handle such requests; see [Managing Roles](https://docs.flopsar.com/administrator-guide/managing-roles.md).
* Viewing captured arguments or exception data is recorded in the audit trail as `sensitive_data_view`. Searches by parameter value — in **Show correlated calls** and in Data Browser queries with a parameter condition — are recorded the same way, with the searched value and the date range. Searches by class, method, or exception only are not recorded.

## Running an investigation

### Gather the facts from the reporter

Before you search, collect:

* **When** — the time of the event as precisely as possible, and the time zone the reporter used.
* **What** — the identifier to search for, spelled exactly as the user saw it.
* **Where** — the application, or the business function, so you know which application to search.
* **Outcome** — what the user saw (an error message, a code, a blank page, a timeout).

### Find the invocations

Search for the identifier in [Data Browser](https://docs.flopsar.com/user-guide/observing-and-diagnosing/data-browser.md) with a custom query; you can combine it with class, method, thread, or exception conditions:

1. Set the **Date range** to a window around the reported time. Start narrow (for example ±15 minutes) and widen only if needed; every search scans the whole range.
2. Select the **Application**.
3. Add a **parameter condition** with the identifier as the value. Leave the key empty to match the value in any captured parameter.
4. Choose the match kind carefully. **Contains** is the default and is **case-sensitive**. To ignore case, switch to **Regex** and prefix the expression with `(?i)`, for example `(?i)jan\.kowalski`. Special regex characters such as `.`, `+`, or `(` must be escaped.
5. Optionally enable **Root calls only** to get one row per request instead of every nested method.

Results arrive in batches; use **Load more results** until the list is complete.

### Read what happened

Open each relevant invocation:

* The **execution tree** shows what the request did and where the time went. Remember that only instrumented methods appear in it; see [Assembled stacks](https://docs.flopsar.com/user-guide/key-concepts-and-glossary.md#assembled-stacks-why-what-you-see-can-differ-from-reality).
* The **parameter inspector** shows the captured values — often the most direct evidence of what the user actually submitted.
* The **exception** view shows the type and trace of a failure. Exceptions can be filtered only by type, not by the text of their message; if the reported phrase appears only in an exception message, search by the identifier in the arguments and then open the exceptions of the matching invocations.

### Follow the user's session

The identifier a user reports is often recorded only in some steps — the login when signing in, an order number when ordering — while a session or correlation id links all of them. Open a matching call, find that id in the parameters of the execution tree, and use **Show correlated calls** on it. The [Correlated calls](https://docs.flopsar.com/user-guide/observing-and-diagnosing/data-correlation.md) window lists the root calls that recorded the same id around that time, across threads and applications, so you can see the sign-in, the steps in between, and the failing action in one place.

## Interpreting the result

An empty or incomplete result has several possible explanations besides *"it did not happen"*:

* The operation is not covered by an Allow rule, or its rule captures timing only.
* The identifier is not among the captured values, or it was captured under a different form (case, formatting, leading zeros).
* The value was masked when stored.
* The invocation was discarded by automatic tuning because **Always report** was not enabled.
* The data is older than the retention period, or it has been moved to an archive.
* The agent was not connected, or a different profile was deployed at that time. The [audit trail](https://docs.flopsar.com/security/logging-monitoring-and-audit-trail.md) records agent connections (`agent_connect`, `agent_disconnect`) and profile deployments (`profile_deploy`).
* The time window or the time zone is wrong.

State the conclusion in terms of what the data shows — *"no recorded invocation of `placeOrder` contained the order number between 10:00 and 10:30"* — rather than as a statement about what the user did.

## Recording findings

Create an [investigation](https://docs.flopsar.com/user-guide/observing-and-diagnosing/investigation.md) under the ticket number. Add what proves your conclusion with **Add to case**: the calls, the analysis of the period, the exception, a thread dump, a Key-Value series, the correlated calls. Record what the user reported with **Add entry**, and add screenshots or log excerpts as files. Write your conclusions in **Findings**, referring to the evidence as `[E1]`, and attach the **PDF report** to the ticket.

To work on the case with colleagues, [share](https://docs.flopsar.com/user-guide/observing-and-diagnosing/investigation/sharing.md) it with them as editors or viewers. Making it **Public** instead reveals its evidence, with the captured parameters, and your findings to everyone with **View parameters**, so share with chosen people when that circle is too wide. The report is a file outside Flopsar: it may contain personal data from titles, notes, entries and files, and storing and deleting it is up to your ticketing process.

Before concluding that something was unusual, [replay](https://docs.flopsar.com/user-guide/observing-and-diagnosing/investigation/replay.md) the evidence in a normal period, for example the day or the week before.

### Keeping evidence

Recorded data passes out of retention. To keep what proves your conclusion, add it to an investigation with **Add to case**, as described in [Adding Evidence](https://docs.flopsar.com/user-guide/observing-and-diagnosing/investigation/adding-evidence.md). *Available since Flopsar 7.2.*

* **What is kept.** A copy made by the server when you add it. For a call, the whole call stack with the parameters and exception stack traces of every call in it. Masked values stay masked; exception payloads are not masked. Files added by hand are kept as they are, without masking. Add only what the case needs.
* **Where.** In `storage/evidence` of the server data directory (`FLOPSAR_HOME`), one file per item, compressed and encrypted with the server master key. The investigation database holds the list of items and the SHA-256 hash of each copy. Retention and archiving do not touch the directory. Back it up together with the database and the master key; a copy cannot be read without the key it was encrypted with. See [Storage and backup](https://docs.flopsar.com/user-guide/observing-and-diagnosing/investigation/reference.md#storage-and-backup).
* **Integrity.** The server checks the hash every time a copy is read and refuses one that was changed, moved to another investigation, or encrypted with another key. The PDF report lists the full hashes, so a copy can be matched against a report handed over earlier, and the audit trail keeps the hash of every report made.
* **Who can read it.** The owner and the members of the investigation, and, when it is public, everyone with **View data** and **View parameters**; each of them only for the applications their role allows. The owner and editors add evidence; editors change only the items they added. Opening a copy is recorded in the audit trail as `evidence_view`.
* **How long.** Until it is deleted: deleting an item removes its copy, and deleting the investigation removes all of them. Delete evidence when the case is closed and its retention under your data protection rules has ended.

For findings made outside Investigation, record in the ticket:

* the application, instance, thread, and start time of the invocation,
* the method and class,
* the relevant parameter values and the exception, if any,
* the search criteria you used, including the date range and match kind.

These details let anyone with access find the same invocation again while it is within retention. Attach screenshots where they help, keeping in mind that they may contain personal data and must be handled accordingly.

## Readiness checklist

* [ ] Entry points users report are covered by Allow rules with **Name & Signature** and **Standard** or **Extended**.
* [ ] The identifiers you expect to receive are among the captured values.
* [ ] **Always report** is enabled on those rules.
* [ ] A shared session or correlation id is captured on every step of the user journey.
* [ ] It is agreed which identifiers stay searchable and which are masked.
* [ ] `archive.retention` covers the typical delay between an incident and its report.
* [ ] Only the investigating team holds **View parameters**.
