Elasticsearch and Elastic APM Integration

Elasticsearch and Elastic APM Integration

Logging and Observability Standards

• Logging Standard: All logging must be implemented using Serilog with structured logging.
• Centralized Destination: All logs must be centralized in Elasticsearch, using Elastic APM authentication and Data Streams configuration.
• Enriched Information: Logs must be enriched with context properties such as app-name and app-type.
• Bootstrapping: Serilog initialization must be COMPLETELY OMITTED IN Program.cs and performed through an Extension Method from the Infrastructure layer.

Security and Data Privacy

• Sensitive Information: PII (Personally Identifiable Information), credentials, and tokens must NEVER be logged.
• Sanitize sensitive data before logging.
• Use log filters to exclude sensitive endpoints or data.

Serilog Configuration Extension Method

Create an extension method in Infrastructure/​Extensions/​ExtensionLogging.cs:

using Serilog;
using Serilog.Events;
using Serilog.Sinks.Elasticsearch;
using Elastic.Ingest.Elasticsearch.DataStreams;
using Elastic.Ingest.Elasticsearch;
using Elastic.Transport;

namespace Infrastructure.Extensions
{
    /// <summary>
    /// Extension class for configuring Serilog with Elasticsearch and Elastic APM.
    /// </​summary>
    public static class ExtensionLogging
    {
        /// <summary>
        /// Configures Serilog with Elasticsearch using Elastic APM.
        /// </​summary>
        /// <param name="configuration">Application configuration.</​param>
        public static void AddLog(IConfiguration configuration)
        {
            string? elasticUrl = configuration.GetValue<string>("Global:Elastic:Url");
            string? environment = configuration.GetValue<string>("Global:Elastic:Env");
            string? elasticUser = configuration.GetValue<string>("Global:Elastic:User");
            string? elasticPass = configuration.GetValue<string>("Global:Elastic:Pass");
            string? logLevelEnv = configuration.GetValue<string>("Global:Elastic:LogLevel");

            LogEventLevel logEventLevel = (LogEventLevel)int.Parse(logLevelEnv!);

            Log.Logger = new LoggerConfiguration()
                .Enrich.FromLogContext()
                .Enrich.WithProperty("app-name", configuration.GetSection("Elastic").GetValue<string>("app-name"))
                .Enrich.WithProperty("app-type", configuration.GetSection("Elastic").GetValue<string>("app-type"))
                .WriteTo.Console()
                .WriteTo.Elasticsearch([new Uri(elasticUrl!)], opts =>
                {
                    // Use DataStream for Elastic APM compatibility
                    opts.DataStream = new DataStreamName("app", environment!, "logs");
                    opts.BootstrapMethod = BootstrapMethod.None;
                    opts.MinimumLevel = logEventLevel;
                }, transport =>
                {
                    // Basic Authentication
                    transport.Authentication(new BasicAuthentication(elasticUser!, elasticPass!));
                    // Callback to accept certificates (e.g., in development)
                    transport.ServerCertificateValidationCallback((a, b, c, d) => true);
                })
                .CreateLogger();
        }
    }
}

Program.cs Configuration

Configure Serilog and Elastic APM in Program.cs only for development:

if (app.Environment.IsDevelopment())
{
    builder.Services.AddLogging(loggingBuilder => loggingBuilder.AddSerilog(dispose: true));
    ExtensionLogging.AddLog(builder.Configuration);
    builder.Services.AddAllElasticApm();
}

Required NuGet Packages

• Serilog.AspNetCore
• Serilog.Sinks.Elasticsearch
• Elastic.Apm.NetCoreAll
• Elastic.Ingest.Elasticsearch
• Elastic.Ingest.Elasticsearch.DataStreams
• Elastic.Transport

Configuration Settings (appsettings.json)

{
  "Elastic": {
    "app-name": "your-app-name",
    "app-type": "api",
    "Url": "https://your-elastic-url:9200",
    "Env": "development",
    "User": "elastic_user",
    "Pass": "elastic_password",
    "LogLevel": "2"
  }
}

Log Level Values

• 0 = Verbose
• 1 = Debug
• 2 = Information
• 3 = Warning
• 4 = Error
• 5 = Fatal

Best Practices

• Use structured logging with named properties: Log.Information("User {UserId} logged in", userId);
• Avoid string interpolation in log messages.
• Use log context to enrich logs with request-specific data.
• Configure appropriate log levels for different environments.
• Monitor APM metrics alongside logs for complete observability.