If you have an elixir app running on Fly.io, sending your telemetry data to Grafana cloud is easy.

Lets start with Logs.

Logs

In order to ship logs to Grafana Cloud Loki, another application, called logshipper, is required. It is all detailed in the documentation, but the TLDR; is:

• 
  Create a new directory

   mkdir logshipper

• 
  Create a new app, but don’t launch yet

   fly launch --no-deploy --image ghcr.io/superfly/fly-log-shipper:latest</code></pre></li><li> Configure your org and access token <pre><code> fly secrets set ORG=personal fly secrets set ACCESS_TOKEN=$(fly auth token)</code></pre></li><li> Configure your Loki credentials. These can be found in the Grafana Cloud Portal, Loki section. You may have to generate an API key, but all the data should be there for you to use here. <pre><code> fly secrets set LOKI_URL= fly secrets set LOKI_USERNAME= fly secrets set LOKI_PASSWORD=</code></pre></li><li> Add this to the newly generated fly.toml file: <pre><code>[[services]] http_checks = [] internal_port = 8686</code></pre></li><li> Deploy <code class="inline">flyctl deploy</code></li></ul><p> Once deployed, this should start sending all the logs from all your fly apps in the configured organization to Grafana Cloud.</p><p> More configuration options for the logshipper app can be found in the  <a target="_blank" href="https://github.com/superfly/fly-log-shipper#provider-configuration">github repo</a></p><p> Lastly, I’d recommend shipping your logs in JSON format using something similar to  <a target="_blank" href="https://hex.pm/packages/logger_json">logger_json</a>. </p><h2> Traces</h2><p> Traces are fairly easy to send to Grafana Cloud Tempo.</p><p> Start by pulling in the open_telemetry libraries.</p><pre><code class="makeup elixir"><span># ./mix.exs</span><span></span><span>defp</span><span></span><span>deps</span><span></span><span>do</span><span></span><span>[</span><span></span><span>...</span><span></span><span>{</span><span>:opentelemetry_exporter</span><span>,</span><span></span><span>&quot;~&gt; 1.0&quot;</span><span>}</span><span>,</span><span></span><span>{</span><span>:opentelemetry</span><span>,</span><span></span><span>&quot;~&gt; 1.0&quot;</span><span>}</span><span>,</span><span></span><span>{</span><span>:opentelemetry_api</span><span>,</span><span></span><span>&quot;~&gt; 1.0&quot;</span><span>}</span><span>,</span><span></span><span>{</span><span>:opentelemetry_ecto</span><span>,</span><span></span><span>&quot;~&gt; 1.0&quot;</span><span>}</span><span>,</span><span></span><span>{</span><span>:opentelemetry_liveview</span><span>,</span><span></span><span>&quot;~&gt; 1.0.0-rc.4&quot;</span><span>}</span><span>,</span><span></span><span>{</span><span>:opentelemetry_phoenix</span><span>,</span><span></span><span>&quot;~&gt; 1.0&quot;</span><span>}</span><span>,</span><span></span><span>{</span><span>:opentelemetry_cowboy</span><span>,</span><span></span><span>&quot;~&gt; 0.2&quot;</span><span>}</span><span></span><span>]</span><span></span><span>end</span></code></pre><p> Add some configuration to <code class="inline">runtime.exs</code></p><pre><code class="makeup elixir"><span># ./config/runtime.exs</span><span></span><span>if</span><span></span><span>config_env</span><span>(</span><span>)</span><span></span><span>==</span><span></span><span>:prod</span><span></span><span>do</span><span></span><span>otel_auth</span><span></span><span>=</span><span></span><span>System</span><span>.</span><span>get_env</span><span>(</span><span>&quot;OTEL_AUTH&quot;</span><span>)</span><span></span><span>||</span><span></span><span>raise</span><span></span><span>&quot;&quot;&quot; OTEL_AUTH is a required variable &quot;&quot;&quot;</span><span></span><span>config</span><span></span><span>:opentelemetry_exporter</span><span>,</span><span></span><span>otlp_protocol</span><span>:</span><span></span><span>:grpc</span><span>,</span><span></span><span>otlp_traces_endpoint</span><span>:</span><span></span><span>System</span><span>.</span><span>fetch_env!</span><span>(</span><span>&quot;OTLP_ENDPOINT&quot;</span><span>)</span><span>,</span><span></span><span>otlp_headers</span><span>:</span><span></span><span>[</span><span>{</span><span>&quot;Authorization&quot;</span><span>,</span><span></span><span>&quot;Basic </span><span>#{</span><span>otel_auth</span><span>}</span><span>&quot;</span><span>}</span><span>]</span><span></span><span>end</span></code></pre><p> Next, setup the environment variables.</p><ul><li> The value required for <code class="inline">OTLP_ENDPOINT</code> can be found on the Grafana Cloud Portal Tempo section, and will look something like: <code class="inline">https://tempo-us-central1.grafana.net/tempo</code> (your url may be different) </li><li> The value for <code class="inline">OTEL_AUTH</code> is a base64 encoded value of <code class="inline">{username}:{api token}</code> which can be easily obtained using: <pre><code> echo -n &#39;username:password&#39; | base64

  
  (replace username and password with the actual vaules)
• 
  And you’ll need the data source name (found on the same Grafana Cloud Portal page) which will be used to set the value of OTEL_RESOURCE_ATTRIBUTES

All of these values can be set with one command:

flyctl secrets set OTLP_ENDPOINT=https://your_endpoint OTEL_RESOURCE_ATTRIBUTES=your_datasource_name OTEL_AUTH=your_base64_encoded_string

After setting these values and deploying the application, traces should start showing in Grafana!

Metrics

For Metrics, I really like to use Prometheus and I find the easiest way to get started with Prometheus is using prom_ex.

Once prom_ex is installed and running, it just needs to be exposed so that Fly can scrape it. As documented by Fly, just add the following to your fly.toml file:

[metrics]

port = 9091

path = "/metrics"

Finally, setup the Prometheus data source in Grafana Cloud with the following properties:

• 
  HTTP -> URL “ https://api.fly.io/prometheus/<org-slug>/“ where org_slug is your org
• 
  Custom HTTP Headers -> + Add Header:
  • 
    Header: Authorization, Value: Bearer <token> where token is the result of flyctl auth token

You should now see all of the fly metrics and prom_ex defined metrics in Grafana!

Wrap Up

I wrote this because I wanted to add observability to my Elixir/Phoenix apps that run on Fly and finding the information I needed was scattered throughout the docs. Hopefully others find this useful.

Happy Observing!