Secrets

Horde implements an API for retrieving secrets that can be stored in its own configuration file or obtained from an external source. Marshalling data through Horde allows access to be controlled using Horde's permissions model, and for automated processes to impersonate the user that requested them.

At the moment, Horde only supports the AWS Parameter Store and HashiCorp Vault as external secret providers out of the box, though other implementations can be added through the ISecretProvider interface.

Configuring secrets

Secrets are defined in the secrets list of the globals.json file. Each entry includes an identifier for the secret (id), a set of key value pairs (data) and an ACL controlling who can access it.

Additional keys and values may be merged from external providers by adding entries to the sources array. Each entry here contains the name of the provider to obtain the secret from, and a path used to locate the secret in a provider-specific syntax.

Secrets from external providers may take two forms, determined by the format property.

text secrets are simple string values which are added to the secret's set of key/value pairs using the specified key property.
json secrets are parsed as JSON objects and merged into the secret's key/value pairs using property names as keys.

An example secret may be configured as follows:

 "secrets": [
     {
         "id": "horde-secrets",

         // Some hard-coded property values        
         "data": {
             "aws-region": "us-east-1"
         },

         // Some values read from the AWS parameter store
         "sources": [

             // Read a single secret from the AWS parameter store and assign it to "aws-secret-access-key"
             {
                 "provider": "AwsParameterStore",
                 "key": "aws-secret-access-key",
                 "path": "name-of-secret-in-parameter-store"
             },

             // Read a JSON object from the AWS parameter store and merge all the key/value pairs into this secret.
             {
                 "provider": "AwsParameterStore",
                 "format": "json",
                 "path": "name-of-secret-in-parameter-store"
             },
         ],

         // Only allow Horde agents to access this
         "acl": {
             "entries": [
                 {
                     "claim": {
                         "type": "http://epicgames.com/ue/horde/role",
                         "value": "agent"
                     },
                     "actions": [
                         "ViewSecret"
                     ]
                 }
             ]
         }
     }
 ]

Secrets are queried from the external provider when requested by a user, and are not cached by Horde.

Secrets from external providers that require additional options, for example HashiCorp Vault to set the endpoint and credentials, will use a providerConfig property instead of name property to provide the data. For example to fetch three secrets from HashiCorp Vault using different authentication methods:

Use an AWS ARN to pass to AWS Security Token Service AssumeRole
Use AWS default credential search
A pre-shared token provided by the Vault server

 "secrets": [
     {
         "id": "horde-secrets",

         // Some values read from HashiCorp Vault
         "sources": [
             {
                 "providerConfig": "hcp-vault-aws-assume-role",
                 // The Vault API path to the secret
                 "path": "/v1/secret/data/first-secret",
                 // Format must be json for Vault because a path in Vault returns a JSON document
                 "format": "json"
             },

             {
                 "providerConfig": "hcp-vault-aws",
                 "format": "json",
                 "path": "/v1/secret/data/second-secret",
             },

             {
                 "providerConfig": "hcp-vault-pre-shared-key",
                 "path": "/v1/secret/data/third-secret",
                 "format": "json"
             }
         ],

         // Only allow Horde agents to access this
         "acl": {
             "entries": [
                 {
                     "claim": {
                         "type": "http://epicgames.com/ue/horde/role",
                         "value": "agent"
                     },
                     "actions": [
                         "ViewSecret"
                     ]
                 }
             ]
         }
     }
 ],
 "plugins": {
     "secrets": {
         "providerConfigs": [
             {
                 "name": "hcp-vault-aws-assume-role",
                 "provider": "HcpVault",
                 "hcpVault": {
                     "credentials": "awsauth",
                     "endpoint": "https://vault.example.com",
                     "awsIamServerId": "vault.example.com",
                     "awsArnRole": "arn:aws:iam::1234567890:role/vault-example-role",
                     "role": "vault-role-name"
                 }
             },
             {
                 "name": "hcp-vault-aws",
                 "provider": "HcpVault",
                 "hcpVault": {
                     "credentials": "awsauth",
                     "endpoint": "https://vault.example.com",
                     "awsIamServerId": "vault.example.com",
                     "role": "vault-role-name"
                 }
             },
             {
                 "name": "hcp-vault-pre-shared-key",
                 "provider": "HcpVault",
                 "hcpVault": {
                     "credentials": "presharedkey",
                     "endpoint": "https://vault.example.com",
                     "preSharedKey": "the-key-provided-by-vault"
                 }
             }
         ]
     }
 }

Using secrets

The most common use case for secrets is during build automation pipelines. In this scenario, the Horde Server URL and credentials are taken from environment variables injected automatically by the Horde Agent, and allow the pipeline to request secrets on behalf of the user starting the job with little additional configuration.

There are three common ways to use secrets:

1. Using the Horde-GetSecrets BuildGraph task

This task takes a file as a parameter, and will read it in, expand any secrets in the form {{ secret-name.property }} with their values from Horde, and write it back out again. Rather than updating an existing file, you can also put the template in a BuildGraph property and expand that instead, as follows:

<Property Name="AwsEnvironmentText" Multiline="true">
   AWS_REGION={{horde-secrets.aws-region}}
   AWS_ACCESS_KEY_ID={{horde-secrets.aws-access-key-id}}
   AWS_SECRET_ACCESS_KEY={{horde-secrets.aws-secret-access-key}}
</Property>
<Horde-GetSecrets File="credentials.txt" Text="$(AwsEnvironmentText)"/>

2. Using the Horde-SetSecretEnvVar BuildGraph task

This task sets an environment variable to the value of a secret at runtime. Environment variables are inherited by child processes but not set at the system level, so the environment variable will contain that secret until the end of the current step.

<Horde-SetSecretEnvVar Name="AWS_SECRET_ACCESS_KEY" Secret="horde-secrets.aws-secret-access-key"/>

3. Using the Horde API

The HTTP secrets endpoint is listed in Horde's API documentation, and AutomationTool includes the following utility methods for querying them at runtime:

IReadOnlyDictionary<string, string> secret = await CommandUtils.GetHordeSecretAsync(new SecretId("my-secret-name"));

string propertyValue = await CommandUtils.GetHordeSecretAsync(new SecretId("my-secret-name"), "propertyName")

Implement a custom external provider

To implement a custom external provider instead of using AWS or HCP Vault. Create a file in the folder Engine\Source\Programs\Horde\Plugins\Secrets\HordeServer.Secrets\Secrets\Providers with a suitable name for the provider e.g. MySecretProvider.cs that provides an implementation of HordeServer.Secrets.ISecretProvider.

 namespace HordeServer.Secrets.Providers
 {
     public class MySecretProvider : ISecretProvider
     {
         // The name must be unique across all providers
         public string Name => "MySecretStore";

         // If access to the provider requires a HTTP client
         private readonly IHttpClientFactory _httpClientFactory;

         private readonly ILogger _logger;

         public MySecretProvider(IHttpClientFactory httpClientFactory, ILogger<HcpVaultSecretProvider> logger)
         {
              _httpClientFactory = httpClientFactory;
              _logger = logger;
         }

         private HttpClient GetHttpClient()
         {
             return _httpClientFactory.CreateClient("HordeSecretMySecretProvider");
         }

         public async Task<string> GetSecretAsync(string path, SecretProviderConfig? config, CancellationToken cancellationToken)
         {
             string secretValue;
             // Implement how to get the secret for the give 'path' from your provider
             return secretValue;
         }
     }
 }

Add the provider to the service collection in the ConfigureServices method found in Engine\Source\Programs\Horde\Plugins\Secrets\HordeServer.Secrets\SecretsPlugin.cs.

 public void ConfigureServices(IServiceCollection services)
 {
     // ...

     services.AddSingleton<ISecretProvider, MySecretProvider>();
 }

To test on a local workstation unit tests can be added to Engine\Source\Programs\Horde\Plugins\Secrets\HordeServer.Secrets.Tests\Secrets\SecretProviderTests.cs.

Edit the globals.json file to add a secret to use the new provider as shown in the earlier examples. In the examples where there is "provider": "AwsParameterStore" use "provider": "MySecretStore" instead.

Rebuild and deploy the Horde server. Once deployed can use Swagger and the endpoint /api/v1/secrets/{secretId} to get a secret declared in globals.json.

Add additional configuration for the external provider

If the external provider requires additional configuration for example different secrets are stored at different domains or an access token is required then a provider configuarion can be implemented.

Create a file in the folder Engine\Source\Programs\Horde\Plugins\Secrets\HordeServer.Secrets\Secrets\Providers with a suitable name for the provider configuration e.g. MySecretProviderConfig.cs.

 namespace HordeServer.Secrets.Providers
 {
     public class MySecretProviderConfig
     {
         public string? DomainName { get; set; }
         public string? LoginToken { get; set; }
     }
 }

Extend the class HordeServer.Secrets.SecretProviderConfig in Engine\Source\Programs\Horde\Plugins\Secrets\HordeServer.Secrets\Secrets\SecretProviderConfig.cs to have a MySecretProviderConfig member.

 public class SecretProviderConfig
 {
     // ...

     public MySecretProviderConfig? MyConfig { get; set; }
 }

Edit the GetSecretAsync method created earlier to use this new configuration to get the secret. For example if the configuraion was to be used in a HTTP GET request it might look something like this

 public async Task<string> GetSecretAsync(string path, SecretProviderConfig? config, CancellationToken cancellationToken)
 {
     if (config?.MyConfig == null)
     {
         throw new InvalidOperationException($"Unable to fetch secret {path} from My Secret Provider (No Config)");
     }
     if (config.MyConfig.DomainName == null)
     {
         throw new InvalidOperationException($"Unable to fetch secret {path} from My Secret Provider (No Domain Name)");
     }
     if (config.MyConfig.LoginToken == null)
     {
         throw new InvalidOperationException($"Unable to fetch secret {path} from My Secret Provider (No Login Token)");
     }

     Uri uri = new("https://" + config.MyConfig.DomainName + ".example.com", path);
     using HttpRequestMessage request = new(HttpMethod.Get, uri);
     HttpClient httpClient = GetHttpClient();
     httpClient.DefaultRequestHeaders.Add("X-Token", config.MyConfig.LoginToken);
     using HttpResponseMessage httpResponseMessage = await httpClient.SendAsync(request, cancellationToken);

     string secretValue;
     // Implement how to get the secret from the repsone message
     return secretValue;
 }

Edit the globals.json file and add the configuration options to plugins.secrets.providerConfigs with an object for each combination of configuration options. For the secrets replace "provider": "MySecretStore" with the key "providerConfig" and its value the unique name of the object in plugins.secrets.providerConfigs. See the earlier examples of configuring HCP Vault for more details.

13 KiB Raw Blame History