Logging Strategy (high costs for storing all logs)
In our organization, we use a GCP setup with Kubernetes. We generate tons of firewall logs as we provide a digital service that generates a high volume of requests from our users. Storing all these logs is quite costly, especially when you consider how long it takes to detect an attack, resulting in a long retention period. Using the firewall logs for troubleshooting is quite rare (I don't remember us having this case).
Our goal is to reduce the cost of storing these logs. Can you recommend a technique that maintains visibility (in terms of security) on the one hand and reduces the amount of logs on the other? I have heard of different approaches: aggregation, deploying an IDS behind the firewalls and using these logs instead, labeling the severity of logs based on risk and omitting low severity logs. But I would like to hear your opinion on different approaches.
We are ISO 27001 certified and subject to GDPR if this information is of interest.
