he great thing about Loki is that we can access all of our logs through one interface, search them, analyze them, create dashboards from them to better understand what is happening inside the application. The problem is we have A LOT of logging data. Our Loki data store is up to over 11TB now to maintain 6 months worth of logging data. With that much unstructured data, doing anything useful with it becomes difficult, slow and uses ever more processing power.Structuring our logs allows us to use Loki's labeling powers, like indexing for databases. Instead of having to read ALL of the logs, we can read the much smaller "indexes" and get the answer much faster, and with less resource impact to the storage systems. It also allows us to analyze our logs for reporting, troubleshooting, even for prioritizing development work. Small queries can tell us how often a specific bug is being triggered, how many times it's happened, etc. We can quickly answer questions like, can users log in to <app>? and many equally impactful others. We can add the answers to common questions to project dashboards. I'm pretty sure we can embed those into Confluence wiki pages to increase visibility to specific questions and answers (dashboard widgets).A few sub-topics. We don't have to cover them all this week, we can spread it out and we can discuss details within the different project teams.

• Formats. What do you like, what do you dislike?

• Challenges. Which apps do we not have any control over output? Which apps are going to require more work to reformat? Which apps require a specific person's time? Other challenges?

• Verbosity. This is a specific challenge, because 11TB for 6 months is a lot. This is mostly a Hyrax issue, so Drupal, Oregon Explorer, etc. generate only a fraction of what SA@OSU and OD2 generate. This is a complex issue that we won't tackle in one meeting.

  • What can we do while we're thinking about formatting that can help lower the data volume?

  • What would the level of effort be to implement dynamic logging? (Utilize logging "levels" so that log entries only needed for debugging have a level of DEBUG, etc. Classify entries that we always want to see so they're always logged no matter the level. If it's not already there, add logging level to the ConfigMap with all of our runtime configurations, we can enable the noisiest logging with a kubectl ... rollout restart command after a config change.

  • What logging frameworks can we use to condense the dozens of lines Rails likes to emit down to a structured summary per-request?

• Retention. How long do we need (and want) to keep logging data? Do we have different requirements for different apps?

• Next Steps. Which apps do we debug more often, and would benefit most? Which apps are low/medium/high effort? When can we fit this work into future work cycles?

A few other notes:

• Drupal site log reformatting is VERY low effort. All logging is done by the web server and they all allow for formats Loki understands natively (JSON, key="val", etc.)

• logcli is a simple tool for developers, available on most platforms, for doing command line log queries. Using labels is relatively straightforward, for example: {req_id="<id>", message=~".*<string I'm looking for>"} will return any lines matching that request ID and the regular expression in the message. I don't think we can actually index the entire message, there would be way too many different values but you get the idea.