Red Hat/Rocky Linux Installation and Maintenance Guide

This document provides a comprehensive guide for setting up and managing an Unmeshed instance on a Linux server, specifically targeting Red Hat Enterprise Linux (RHEL) and Rocky Linux distributions. Whether you're deploying Unmeshed for development, testing, or production use, this guide will walk you through the complete installation process, configuration steps, and ongoing management tasks.

• System requirements and prerequisites
• Step-by-step installation process
• Initial configuration and setup
• Security hardening recommendations
• Monitoring and maintenance procedures
• Troubleshooting common issues

Before beginning the installation, ensure you have:

• A Red Hat Enterprise Linux (RHEL) 8/9 or Rocky Linux 8/9 server
• Root or sudo access to the system
• Basic familiarity with Linux command line operations
• Network connectivity for downloading packages and dependencies

We will refer to the following location as the primary installation directory throughout this guide. If required, you can change this folder to another location based on your system requirements or preferences.

Primary Installation Directory: /unmeshed

This directory will contain all the necessary files, configurations, and data for your Unmeshed instance.

Minimum Disk Size: We recommend a minimum size of 256 GB to ensure adequate space for the installation files, runtime data, logs, and future growth.

The /unmeshed directory should be mounted from a reliable, backed up disk to ensure data persistence and availability. This is crucial because this disk store will maintain all the installation files and data generated from the runtime.

Cloud Environment Recommendations:

• Azure: Mount from Azure Disk with appropriate redundancy (Premium SSD or Standard SSD)
• AWS: Use EBS volumes with proper backup policies
• GCP: Use Persistent Disks with snapshot backups enabled
• On-premises: Use RAID arrays or SAN storage with regular backup schedules

Why This Matters:

• Prevents data loss during VM restarts or migrations
• Ensures business continuity and data integrity
• Allows for easy backup and disaster recovery procedures
• Maintains application state across infrastructure changes

The following software components are required to be configured in the instance. This is a one-time setup unless there is a version upgrade required.

Unmeshed requires a PostgreSQL V16 or above instance. Please ensure you have a PostgreSQL instance available and accessible from your Unmeshed server.

Download the Unmeshed application bundle and extract it to your installation directory. This will create the core application files and directories needed to run Unmeshed.

After extraction, the following files and directories will be available in your /unmeshed directory:

Configuration Files:

• writer-logback.xml - Logging configuration for the writer service
• engine-logback.xml - Logging configuration for the engine service
• processes.yaml - Process management configuration
• start_monitor.sh - Script to start the monitoring service

Application JARs:

• monitor.jar - Monitoring service application
• engine.jar - Core engine service application
• engine-writer.jar - Engine writer service application

Data Directories:

• integrations/ - Directory containing integration modules
• sqstore/ - Directory for storing sequence data
• shard1/ - Directory for data sharding

Key Directories:

• deno - Contains files related to the Deno runtime
• denobin - Contains the Deno binary file
• zulu22.32.15-ca-jdk22.0.2-linux_x64 - Contains the Java (Zulu) installation
• roles - Contains the roles.json file used for configuring Azure AD role mappings
• ui_htmls - Will hold the UI binaries
• ui_configs - Contains NGINX configuration files

Now that all installation files are in place, we will apply configurations and set up required properties to get the Unmeshed application up and running. The following sections will guide you through the necessary configuration steps.

Update the NGINX configuration file to configure TLS optionally and to point to the Unmeshed UI file. Note the ssl_certificate settings which require a PEM file public and secret.

• Using an Unmeshed-Provided Domain: We can generate a TLS certificate using one of our domains if you provide the server’s IP address. The IP must be publicly resolvable, as DNS resolution will be configured accordingly.

• Using Your Own Domain:
  If you prefer to use your own domain, TLS certificate generation and management must be performed on your end. You will need to ensure that valid certificates are available and correctly configured within your Nginx setup.

Inside /etc/nginx/conf.d/default.conf, ensure that PORT 443 is redirected to 5173 using the following configuration. Use your TLS certificate settings to configure this and update server_name accordingly.

• TLS Certificates: Ensure you have fullchain.pem and privkey.pem files available
• Server Names: Update the server_name directive with your actual domain names
• Port Forwarding: The configuration forwards HTTPS traffic from port 443 to your application running on port 5173
• Security: The configuration includes strong SSL ciphers and protocols for enhanced security
• Root Directory (Optional): For binary updates, you can optionally set the root folder to /unmeshed/uihtml instead of the default /var/www/html. This is only required when updating the application binaries.

Configure the application properties and settings required for Unmeshed to operate properly. The main configuration file is processes.yaml, which was included with the installation files.

The processes.yaml file contains the core configuration for Unmeshed services, including database connections, logging settings, and process management parameters.

The most critical configuration involves the PostgreSQL database properties. While many default values in processes.yaml are sufficient for most deployments, some properties require new values and are self-explanatory.

Key PostgreSQL Properties to Update:

Properties to Customize:

• SPRING_DATASOURCE_URL: Update the host, port, and database name to match your PostgreSQL instance
• SPRING_DATASOURCE_USERNAME: Update to your PostgreSQL username
• SPRING_DATASOURCE_PASSWORD: Update to your PostgreSQL password (consider using VAULT for secure storage)

Note: The example above shows the default values. Replace localhost:5432/postgres with your actual PostgreSQL server details, and update the username and password accordingly.

Review and update other properties as needed for your environment:

• Logging Configuration: Log levels and output paths
• Network Settings: Port configurations and binding addresses
• Security Settings: Authentication and authorization parameters
• Performance Tuning: Memory and thread pool configurations

Important: Make a backup of the original processes.yaml file before making changes:

The Unmeshed application runs as a systemd service using the start_monitor.sh script. This service manages the core Unmeshed processes and ensures they start automatically on system boot.

The start_monitor.sh script contains the configuration for starting the Unmeshed services. Before creating the systemd service, verify the script content and ensure it's properly configured.

Edit the script to set the correct working directory:

Ensure the CURRENT_DIR variable is set correctly:

Important: Do not modify any other parts of this script unless you have specific requirements.

Create a new systemd service file for Unmeshed:

Add the following content to the service file:

Important Notes:

• User Field: Update User=unmeshed to the actual user that should run the process
• Working Directory: Ensure the path matches your installation directory (/unmeshed)
• Permissions: The specified user must have read/write access to the /unmeshed directory

This section covers the maintenance tasks and upgrade procedures for the various components installed on your Unmeshed server.

For Deno and JDK updates, the Unmeshed team will notify you and provide the required steps and download links. These components are managed by the Unmeshed team and should not be updated independently.

PostgreSQL is typically managed externally (cloud provider, database team, etc.) and does not require manual updates from this server. Ensure your database team keeps PostgreSQL updated according to your organization's policies.

For Unmeshed binary version updates, the team will provide you with a download link to a zip file named unmeshed_binary_files_VERSION.zip. You can download this to your server and follow the upgrade steps below.

Important: This upgrade can be performed without stopping the server, but it's crucial to create backups before proceeding in case a rollback is needed.

Before upgrading, create a comprehensive backup of your current installation:

Backup Contents:

• All JAR files (*.jar)
• Configuration files (processes.yaml, start_monitor.sh)
• UI files (/unmeshed/uihtml directory and all contents)

After creating the backup, proceed with the upgrade:

Upgrade Steps Summary:

1. Extract new version to temporary folder
2. Update JAR files - Copy new *.jar files to /unmeshed/
3. Update UI files - Remove old UI files and copy new ones recursively
4. Set permissions - Ensure proper read/execute permissions on UI folders
5. Cleanup - Remove temporary upgrade files
6. Verify - Check that new files are in place

After the files are updated, restart the Unmeshed service to load the new binaries:

:::warning Force Kill Java Processes The pkill -9 java command will terminate all Java processes on the server. Only use this if you're certain no other Java applications are running.

Alternative approach: If you want to be more selective, you can find and kill only the Unmeshed processes:

:::

This section covers common troubleshooting steps when the Unmeshed application is not functioning properly or is inaccessible.

The following log files contain critical information about the application status and any errors:

Start by monitoring the key log files to identify any errors:

If you find errors in the logs and/or the application is not accessible, restart the services:

Alternative selective process killing:

After restarting, check if the application is running:

If the logs show no errors but you see an nginx default message, the UI files may have been cleaned up during an upgrade. Restore them from your last backup:

• Check if the unmeshed user has proper permissions to /unmeshed directory
• Verify that start_monitor.sh has execute permissions
• Check if all required JAR files are present

• Verify nginx is running: sudo systemctl status nginx
• Check nginx configuration: sudo nginx -t
• Ensure firewall allows traffic on ports 443 and 5173

• Verify PostgreSQL is running and accessible
• Check processes.yaml for correct database credentials
• Test database connectivity from the server

If the troubleshooting steps above don't resolve your issue, or if you encounter critical errors that prevent the application from running:

For immediate on-call support:

• Emergency Contact: Contact the Unmeshed support team immediately
• Support Channels: Use your organization's designated support escalation procedures
• Priority: Mark as high priority for immediate attention

Information to provide:

• Server hostname and environment details
• Error messages and log excerpts
• Steps already attempted
• Current application status
• Impact on business operations

Note: The Unmeshed support team has access to additional diagnostic tools and can provide specialized assistance for complex issues that may not be resolvable through standard troubleshooting procedures.

The Unmeshed monitoring service automatically performs comprehensive health checks every few seconds to ensure the application is running optimally and to detect potential issues before they impact operations.

Our monitoring service continuously checks various aspects of the system health without requiring manual intervention:

Application Health:

• Overall application status and responsiveness
• Backend and UI version information
• Service availability and connectivity

Runtime Environment:

• JVM memory usage (heap and non-heap memory)
• Deno runtime health and status
• Python container health status

Database and Integration Health:

• Primary database connectivity and status
• Analytics database performance
• Metadata database status
• Writer integration health and functionality

System Resources:

• CPU utilization and availability
• Memory usage and availability
• Disk space usage across partitions
• Process file counts and system uptime
• Host system uptime and stability

Operational Metrics:

• Running and completed schedule counts
• Process and resiliency file status
• Step file processing status

• Proactive Issue Detection: Identifies problems before they cause service disruption
• Performance Monitoring: Tracks resource usage and system performance
• Automated Alerting: Monitors critical thresholds and system health
• Operational Insights: Provides visibility into application and system status

When the monitoring service detects failures or critical issues, it automatically sends email alerts to the monitoring teams, including the Unmeshed support team. The email alerting configuration is defined in the processes.yaml file, allowing you to customize:

• Alert Recipients: Email addresses for monitoring teams and Unmeshed support
• Alert Thresholds: Conditions that trigger email notifications
• Alert Frequency: How often alerts are sent for persistent issues

This ensures that critical issues are immediately communicated to the appropriate teams for rapid response and resolution.

The monitoring service runs continuously in the background, ensuring that administrators have real-time visibility into the health and performance of their Unmeshed instance.