in AWS, devops |

Store aws beanstalk symfony and apache logs in cloudwatch logs

Do you have a symfony app deployed with autoscale capabilities in aws beanstalk and you don’t have a central logging server? Then you may find this post especially relevant.

When your app scale up and down your server created and destroyed dynamically. During this process you loose your logs. This logs may be extremely useful if something goes wrong in your system. Moreover it may be required by local legislation to keep a copy of it.

This will be a recipe on how to setup Cloudwatch logs as a central log storage in AWS. The example is based on a symfony app but with minor adjustments it will work for any LAMP project.

Prerequisites:

  • AWS Beanstalk project ready to deploy
  • OS: 64bits Amazon Linux 2016.09 (It may works in others as well)
  • Access to edit roles in AWS Identity Access Management

Step 1: Install awslogs

First we need to install awslogs package into your EC2 instances. In Beanstalk we can do it using the .ebextensions folder. Customizing your linux server in beanstalk

packages:
    yum:
        awslogs: []

Step 2: Log streaming setup

Second we are going use .ebxtensions to configure awslogs and setup which logs of our server do we want to be processed. For that purpose we create a file named .ebextensions/log-streaming.config and add the content above.

There is a few important considerations and changes we have to do in order to fit our application.

  1. /etc/awslogs/awscli.conf – region
    We have to adjust our cloudwatch region in the file awscli.conf . In the example is setup in eu-wset-1
  2. /etc/awslogs/config/app.conf – commands.log
    In the project I based this example I have a symfony monolog stream that logs the Symfony cli commands in app/logs/commands.log for other projects it may not be the case. We can rename this logs to our custom logs or delete if we do not have it.
option_settings:
  aws:elasticbeanstalk:cloudwatch:logs:
    StreamLogs: true
    DeleteOnTerminate: false
    RetentionInDays: 180
files:
  "/etc/awslogs/awscli.conf":
      mode: "000644"
      owner: root
      group: root
      content: |
        [plugins]
        cwlogs = cwlogs
        [default]
        region = eu-west-1

 "/etc/awslogs/config/app.conf":
     mode: "000644"
     owner: root
     group: root
     content: |
       [/var/app/current/app/logs/prod.log]
       log_group_name=/aws/elasticbeanstalk/yourAppName/app/prod.log
       log_stream_name={instance_id}
       file=/var/app/current/app/logs/prod.log

       [/var/app/current/app/logs/commands.log]
       log_group_name=/aws/elasticbeanstalk/yourAppName/app/commands.log
       log_stream_name={instance_id}
       file=/var/app/current/app/logs/commands.log

  "/etc/awslogs/config/beanstalklogs.conf":
      mode: "000644"
      owner: root
      group: root
      content: |
        [/var/log/httpd/error_log]
        log_group_name=/aws/elasticbeanstalk/yourAppName/apache/error.log
        log_stream_name={instance_id}
        file=/var/log/httpd/error_log*

        [/var/log/httpd/access_log]
        log_group_name=/aws/elasticbeanstalk/yourAppName/apache/access.log
        log_stream_name={instance_id}
        file=/var/log/httpd/access_log*

        [/var/log/eb-activity.log]
        log_group_name=/aws/elasticbeanstalk/yourAppName/deployment/activity.log
        log_stream_name={instance_id}
        file=/var/log/eb-activity.log*

container_commands:
  10logs:
    command: sudo service awslogs restart

Step 3: Grant access to cloudwatch logs

Third we need to extend the default logs our aws-beanstalk-ec2-role as the default setup does not allow to create log streams and log groups.

IAM may looks complicated the first time we use it so I will guide step by step.

First you have to create a policy that allow the basic actions to create logging groups, log streams and upload the logs. For that we go into IAM console and once there we create our own policy (IAM > Policies > Create policy > Create your own policy ). I named it AWSElasticBeanstalkCloudWatchLogsExtended. The follow JSON is the policy document.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CloudWatchLogsAccess",
            "Action": [
                "logs:PutLogEvents",
                "logs:CreateLogStream",
                "logs:CreateLogGroup",
                "logs:GetLogEvents",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams",
                "logs:PutRetentionPolicy"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:logs:*:*:log-group:/aws/elasticbeanstalk*"
            ]
        }
    ]
}

Second once created we have to attache our policy to our aws-elasticbeanstalk-ec2-role. IAM > Roles > aws-elasticbeanstalk-ec2-role > Attach police .

Related link: IAM Documentation

 

Step 4: Deploy your application

Deploy your application, check /var/logs/awslogs.log in your web servers and check your logs in aws cloudwatch logs.

Conclusion

In conclusion we should have our app logs streamed into our aws cloudwatch logs. Therefore we can create dashboards, alarms, metrics based on our logs as well as keeping them even when our application downscale.

How do you manage your logs?

If you like this post, comment and share it.

Published

Write a Comment

Comment