Identity Server 4 Run behind a load balancer

I have Identity Server 4 installed for my project using Entity Framework. I have already configured the service to use the saved grant store and signed certificate.

services.AddIdentityServer()
        .AddSigningCredential(Config.GetSigningCertificate())
        .AddResourceOwnerValidator<ResourceOwnerPasswordValidator>()
        .AddProfileService<ProfileService>()
        .AddConfigurationStore(builder =>
                    builder.UseSqlServer(connectionString, options =>
                        options.MigrationsAssembly(migrationsAssembly)))
        .AddOperationalStore(builder =>
                    builder.UseSqlServer(connectionString, options =>
                        options.MigrationsAssembly(migrationsAssembly)));

      

        
        
        
      

    

Here is the configuration for the service.

The problem is that I am running my server behind a load balancer, for example for two identical instances handling the whole request, the server where the user is not logged in failed to decode the JWT token resulting in 401 unauthorized errors.

I am guessing that the method of defining tokens or labeling them is the problem, but I cannot find a way to solve this problem.

Here's the rest of my configuration.

Configuration:

app.UseIdentityServerAuthentication(new IdentityServerAuthenticationOptions
{
      Authority = url,
      // Authority = "http://localhost:5000",
      AllowedScopes = { "WebAPI" },
      RequireHttpsMetadata = false,
      AutomaticAuthenticate = true,
      AutomaticChallenge = true,

});

      

        
        
        
      

    

Customer:

new Client
{
     ClientId = "Angular2SPA",
     AllowedGrantTypes = GrantTypes.ResourceOwnerPassword, // Resource Owner Password Credential grant.
     AllowAccessTokensViaBrowser = true,
     RequireClientSecret = false, // This client does not need a secret to request tokens from the token endpoint.
     AccessTokenLifetime = 7200, // Lifetime of access token in seconds.
     AllowedScopes = {
                       IdentityServerConstants.StandardScopes.OpenId, // For UserInfo endpoint.
                       IdentityServerConstants.StandardScopes.Profile,
                       "roles",
                       "WebAPI"
                      },
     AllowOfflineAccess = true, // For refresh token.
     AccessTokenType = AccessTokenType.Jwt

}

      

        
        
        
      

    

I also implemented my own IResourceOwnerPasswordValidator and IProfileService.

Any idea why this is happening?