Insight Server

Insight Server

The Insight server is a document-oriented database (Apache Lucene Solr) that is used for frevvo Reports and Task/Submission searches. The software is included in the frevvo tomcat bundle. The Insight Server works in conjunction with frevvo and a web app (insight.war embedded in frevvo.war) to retrieve the information for the frevvo reports. frevvo administrators, designers and users assigned form/flow Edit permissions can run the Insight Data Batch Job to refresh submissions when changes are made to Searchable Fields in a form/flow.

Here is how it works:


  • frevvo users initiate a form or a workflow process.
  • The frevvo repository and the Insight server are updated with the information real-time.
  • The Insight service (insight.war) is a batch job that runs automatically on upgrade but can also be performed manually. The batch job takes the existing frevvo submissions and creates Solr documents for them.
  • The Reports query the Insight Server.

Always use a fresh Insight instance for frevvo. Multiple databases should never point to the same Insight instance.

On This Page:

Insight Server Files

When you unzip the tomcat bundle, you will see a new <frevvo-home>\solr directory that contains the Insight Server software.

  • A solr core is created for each tenant.

  • A directory is created for each core under frevvo/data/solr. The documents in each core are stored under the 'data' directory under the core directory.

 

  • Report data kept in the subdirectories of <frevvo-home>\data\solr

  • When you delete a tenant, the core gets deleted.

Insight Server Location Configuration parameter 

The default server and port are localhost:8983. The Insight Server URL configuration parameter is located in the <frevvo-home>\tomcat\conf\Catalina\localhost\context.xml.default file. Configuring the location of the Insight Server in this file makes it available to the insight and frevvo web application warfiles. Modify the insight.server.url parameter in the <frevvo-home>\tomcat\conf\frevvo-config.properties file if the Insight Server is running on a different machine than frevvo.

# Insight settings 
insight.enabled=true
insight.server-url=http://localhost:8983/solr

Starting the Insight Server from the command line

frevvo requires Insight software for the Reporting and Submissions View feature. The Insight software is included in the tomcat bundle. The Insight server MUST be started BEFORE starting frevvo. Instructions for Windows and Linux operating systems are listed here:

 Windows:

  1. Open a command prompt. Navigate to <frevvo-home>\solr. Type bin\solr.cmd start to run the Insight server in the background, listening on the default port 8983.

    You will see this message: 

  2. Browse http:<your server:port>/solr to verify the Insight server is running. The <server:port> default values are localhost:8983. Change <your server>  to the server name and <your port> to the port the Insight server is running on if they are different than the defaults. Do not include the angle brackets <>. You will see the Insight server (Solr) dashboard with the current status:

If you close the Insight server (Solr) startup window, the server will stop running. Leave the window open or set up frevvo and the Insight Server to run as Windows services.

Linux:

  1. Ensure that the scripts are executable: chmod 755 <frevvo-home>/tomcat/bin/*.sh
  2. Navigate to <frevvo-home>/solr/bin. Run this command to make the solr startup file executable:

    chmod +x solr
  3. Navigate back to the solr directory.
  4. Start the Insight server by typing bin/solr start - this starts the Insight server in the background, listening on the default port 8983.
  5. Browse http:<your server:port>/solr to verify the Insight server is running. The <server:port> default values are localhost:8983. Change <your server>  to the server name and <your port> to the port the Insight server is running on if they are different than the defaults. Do not include the angle brackets <>. You will see the Insight server (Solr) dashboard with the current status.

You do not have to restart the Insight server every time you restart frevvo.

Stopping the Insight Server from the command line

Use the bin\solr.cmd -all command to stop all instances of the Insight Server.

Follow these steps:

  1. Navigate to <frevvo-home>\solr
  2. Type bin\solr.cmd stop -all



    To stop a particular instance of the Insight Server, use the -p option to specify the port number that the Insight Server process is running on. Ex: bin\solr.cmd stop -p 8983

     

Running frevvo and the Insight Server as Windows Services

The Insight Server must be started BEFORE frevvo from a command prompt, or you will see a Configuration Checklist warning and you will not be able to proceed. If you close the Insight Server startup window, on a Windows system, the server will stop running. You can leave the window open or set up frevvo and the Insight Server to run as Windows services.

If you are using the frevvo tomcat bundle, follow the steps below to run the Insight Server (Solr) as a Windows service. Otherwise, general instructions can be found on this website.

  1. Install the Insight Server as a Windows service
  2. Install frevvo as a Windows service
  3. Add a Dependency to the frevvo service

Install the Insight Server as a Windows Service

We will use a service helper such as NSSM (Non-sucking Service Manager). NSSM requires Java 7 or higher. You can download it from here.

  1. Extract the NSSM zipfile to a Program Files\nssm directory
  2. Open a command prompt. Run as an administrator
  3. Navigate to the directory under c:\Program Files\nssm\ that contains the nssm.exe file for Windows64 or Windows32

    For example, the path in Windows 7 would be C:\Program Files\nssm\nssm-2.24\win64
  4. Type: nssm install <service name> - Replace <service name> with the name you want to use for the Insight Server service.

    Ex: nssm install insightserver
  5. Browse to the location of the solr.cmd script by clicking on the three dots to the right of the Application Path field. This should be in <frevvo-home>\solr\bin.

  6. Click on the solr.cmd file, then click Open.  The Application Path field will populate.

  7. Notice the Startup directory field also populates. REMOVE bin from the Startup directory field. The Startup directory should be: c:\<frevvo-home>\solr\

  8. Enter these Arguments:

    start -f -p 8983

    • the -f option starts the Insight Server in the foreground so NSSM can kill it when it needs to be stopped or started
    • the -p option specifies the port that the Insight Server is running on. The default is 8983. Change 8983 to the port number for your installation if necessary.
  9. Optionally, click the Details tab to specify a Display Name and Description - Ex: Insight Server, Script to Stop/Start Solr5. Notice the Startup type is set to Automatic.

  10. Click the Install service button. A success message displays:

  11. Start the Insight Server service from the Computer Management Services pane.


  12. Browse http://<your server:port>/solr to verify the Insight server is running. The <server:port> default values are localhost:8983. Change <your server>  to the server name and <your port> to the port the Insight server is running on if they are different than the defaults. Do not include the angle brackets <>. You will see the Insight server dashboard with the current status:

Install frevvo as a Windows service

There are batch files in the <frevvo-home> directory to Install/Configure frevvo as a Windows service. Refer to Installing frevvo as a Windows Service for more details and troubleshooting tips.

  1. Navigate to <frevvo-home>. Run the Install-Service batch file as an administrator to install frevvo as a service.
  2. Configure the frevvo service if you are using Windows 64 bit by clicking on the Configure Service batch file. You must run the Configure-Service batch file as an administrator.

    1. Change the Startup type to Automatic. Save the changes.

  3. frevvo will appear under Windows services with the name frevvoforms with a Startup type of Manual. Change Startup Type to Automatic (Delayed Start), then start the service.

Add a Dependency to the frevvo service

Once the two services are installed and started, add a dependency to the Insight server for the frevvoforms service. The dependency ensures that the Insight service is running before the frevvoforms service starts.

  1. From the administrative command prompt, type the following command with the name you gave to the Insight Server service:

    sc config frevvoforms depend= <name of the Insight Server service>



  2. Check the Dependencies tab of your Insight Server service in the Windows Services list. It should list frevvoforms as a system component that depends on the Insight service.



  3. If you manually stop the Insight Server service, it will ask you if you want to stop frevvo.

Insight Server Security

Solr should only be accessible to the frevvo server i.e. http://localhost:8983/, and should not be remotely accessible. Most customers don’t allow external http access to a server (only https). Since solr is not exposed over https, remote code execution vulnerability can be mitigated (a remote attacker cannot exploit this vulnerability as it is not exposed).

Ensure solr is stopped before making these changes

  • On Windows: In solr.in.cmd (in the solr\bin folder) replace REM SOLR_JETTY_HOST=0.0.0.0 with set SOLR_JETTY_HOST=127.0.0.1

  • An alternative approach is to start solr with the -Djetty.host=127.0.0.1 parameter (this works for any platform)

Another option to enhance solr security is to block external access to port 8983, for example using firewall rules. It is also possible to whitelist and blacklist IP addresses using the SOLR_IP_WHITELIST and SOLR_IP_BLACKLIST parameters respectively (in the solr.in.cmd/sh file).

frevvo accesses solr via the client API and does not use velocity. It also encodes all solr query parameters, so it is not possible to attack via injection.

Troubleshooting Tips

Insight Server Log

Indexing related messages are captured in the <frevvo-home>\tomcat\logs\frevvo.YYYY-MM-DD file. For example, if you run the Insight Data Batch process manually or if it runs automatically during an upgrade, messages from the process are found in the frevvo.log.

Failing or Long Running Refresh Process

If your Refresh Searchable Fields/Items (re-indexing) process is failing or taking a long time, more than a couple of hours, the cause may be that the frevvo database index statistics have become outdated. Your database admin should perform a maintenance process to Update Statistics. Please refer to your SQL database documentation for instructions. After verifying the index statistics are up to date, re-run the Refresh Searchable Fields/Items process.