inspection fails with inspec 2.x

[mattagohni]

Hello everyone,

first of all, thanks for these controls, this helps alot. Unfortunately I have problems running this profile after upgrading to inspec 2.x.

It seems to be, that every control, which checks the content of a file, fails. After downgrading to inspec 1.51.25 the controls worked fine again.

I ran the control from my local machine (running macOS) against a EC2-Instance via ssh. Here is the command I run:

inspec exec -i ssh_key  -t ssh://ubuntu@x.x.x.x spec/webserver --sudo

Output with inspec v2.045/2.1.83

Profile: DevSec Nginx Baseline (nginx-baseline)
Version: 2.0.2
Target:  ssh://ubuntu@x.x.x.x:22

  ×  nginx-01: Running worker process as non-privileged user (1 failed)
     ✔  User www-data should exist
     ×  Parse Config File /etc/nginx/nginx.conf user should eq "www-data"
     
     expected: "www-data"
          got: nil
     
     (compared using ==)

     ✔  Parse Config File /etc/nginx/nginx.conf group should not eq "root"
  ✔  nginx-02: Check NGINX config file owner, group and permissions.
     ✔  File /etc/nginx/nginx.conf should be owned by "root"
     ✔  File /etc/nginx/nginx.conf should be grouped into "root"
     ✔  File /etc/nginx/nginx.conf should not be readable by others
     ✔  File /etc/nginx/nginx.conf should not be writable by others
     ✔  File /etc/nginx/nginx.conf should not be executable by others
  ✔  nginx-03: Nginx default files
     ✔  File /etc/nginx/conf.d/default.conf should not be file
     ✔  File /etc/nginx/sites-enabled/default should not be file
     ✔  File /etc/nginx/nginx.conf should be file
     ✔  File /etc/nginx/conf.d/90.hardening.conf should be file
  ✔  nginx-04: Check for multiple instances
     ✔  Command ps aux | egrep "nginx: master" | egrep -v "grep" | wc -l stdout should match /^1$/
  ×  nginx-05: Disable server_tokens directive
     ×  Parse Config File /etc/nginx/nginx.conf server_tokens should eq "off"
     
     expected: "off"
          got: nil
     
     (compared using ==)

  ×  nginx-06: Prevent buffer overflow attacks (4 failed)
     ×  Parse Config File /etc/nginx/nginx.conf client_body_buffer_size should eq "1k"
     
     expected: "1k"
          got: nil
     
     (compared using ==)

     ×  Parse Config File /etc/nginx/nginx.conf client_max_body_size should eq "1k"
     
     expected: "1k"
          got: nil
     
     (compared using ==)

     ×  Parse Config File /etc/nginx/conf.d/90.hardening.conf client_header_buffer_size should eq "1k"
     
     expected: "1k"
          got: nil
     
     (compared using ==)

     ×  Parse Config File /etc/nginx/conf.d/90.hardening.conf large_client_header_buffers should eq "2 1k"
     
     expected: "2 1k"
          got: nil
     
     (compared using ==)

  ×  nginx-07: Control simultaneous connections (2 failed)
     ×  Parse Config File /etc/nginx/conf.d/90.hardening.conf limit_conn_zone should eq "$binary_remote_addr zone=default:10m"
     
     expected: "$binary_remote_addr zone=default:10m"
          got: nil
     
     (compared using ==)

     ×  Parse Config File /etc/nginx/conf.d/90.hardening.conf limit_conn should eq "default 5"
     
     expected: "default 5"
          got: nil
     
     (compared using ==)

  ×  nginx-08: Prevent clickjacking
     ×  Parse Config File /etc/nginx/conf.d/90.hardening.conf add_header should include "X-Frame-Options SAMEORIGIN"
     expected nil to include "X-Frame-Options SAMEORIGIN", but it does not respond to `include?`
  ×  nginx-09: Enable Cross-site scripting filter
     ×  Parse Config File /etc/nginx/conf.d/90.hardening.conf add_header should include "X-XSS-Protection \"1; mode=block\""
     expected nil to include "X-XSS-Protection \"1; mode=block\"", but it does not respond to `include?`
  ×  nginx-10: Disable content-type sniffing
     ×  Parse Config File /etc/nginx/conf.d/90.hardening.conf add_header should include "X-Content-Type-Options nosniff"
     expected nil to include "X-Content-Type-Options nosniff", but it does not respond to `include?`

Profile Summary: 3 successful controls, 5 control failures, 0 controls skipped
Test Summary: 12 successful, 5 failures, 0 skipped

inspec v1.51.25:

Profile: DevSec Nginx Baseline (nginx-baseline)
Version: 2.0.2
Target:  ssh://ubuntu@x.x.x.x:22

    ✔  nginx-01: Running worker process as non-privileged user
     ✔  User www-data should exist
     ✔  Parse Config File /etc/nginx/nginx.conf user should eq "www-data"
     ✔  Parse Config File /etc/nginx/nginx.conf group should not eq "root"
  ✔  nginx-02: Check NGINX config file owner, group and permissions.
     ✔  File /etc/nginx/nginx.conf should be owned by "root"
     ✔  File /etc/nginx/nginx.conf should be grouped into "root"
     ✔  File /etc/nginx/nginx.conf should not be readable by others
     ✔  File /etc/nginx/nginx.conf should not be writable by others
     ✔  File /etc/nginx/nginx.conf should not be executable by others
  ✔  nginx-03: Nginx default files
     ✔  File /etc/nginx/conf.d/default.conf should not be file
     ✔  File /etc/nginx/sites-enabled/default should not be file
     ✔  File /etc/nginx/nginx.conf should be file
     ✔  File /etc/nginx/conf.d/90.hardening.conf should be file
  ✔  nginx-04: Check for multiple instances
     ✔  Command ps aux | egrep "nginx: master" | egrep -v "grep" | wc -l stdout should match /^1$/
  ✔  nginx-05: Disable server_tokens directive
     ✔  Parse Config File /etc/nginx/nginx.conf server_tokens should eq "off"
  ✔  nginx-08: Prevent clickjacking
     ✔  Parse Config File /etc/nginx/conf.d/90.hardening.conf add_header should include "X-Frame-Options SAMEORIGIN"
  ✔  nginx-09: Enable Cross-site scripting filter
     ✔  Parse Config File /etc/nginx/conf.d/90.hardening.conf add_header should include "X-XSS-Protection \"1; mode=block\""
  ✔  nginx-10: Disable content-type sniffing
     ✔  Parse Config File /etc/nginx/conf.d/90.hardening.conf add_header should include "X-Content-Type-Options nosniff"

Profile Summary: 8 successful controls, 0 control failures, 0 controls skipped
Test Summary: 17 successful, 0 failures, 0 skipped

Anyone an idea, what's going on here?

[mattagohni]

Little update on this one, the problem seems to be a deprecated method call which is in v2.0.2 of the baseline. After forking the repository and using a version with the replaced method call, I got it to work.

Since the deprecated method call is fixed in v2.1.0 I upgraded to this one and skipped controls which are making problems for us at the moment.

[chris-rock]

Thank you @solutionDrive-Alt for providing the details. Does 2.1.0 of this baseline still having an issue? Please let us know what is difficult. Also any PRs to improve the baselines are welcome!

[mattagohni]

I was trying to open a PR for this, but it seems impossible to make PR against the 2.0.2 version of the baseline. I think this would require some git magic.. :-) (altering history and so on) But of course we are willing to provide some PRs in the future, since we are using more and more stuff from devsec.

The problem we have with the latest baseline is the following:
For 2.1.0 the corresponding ansible-role seems to configure nginx not as save/strict as the baseline requires it. I'm not totally in the details, since the integration of the nginx-baseline and its ansible-role was not done by me in our projects, but I will have a look at this in the next days/weeks.

A colleague of mine mentioned that a PR exits within the repository for the ansible-role, which would address most of the issues. But these issues are not related to the problem I was mentioning above.
For example does the nginx-baseline a very strict test for TLS, where the ansible-role allows more versions in its default-configuration.

Here for example my output running baseline 2.1.0 with inspec 2.1.83 against a server provisioned with the nginx-hardening-role 1.0.2:

Profile: DevSec Nginx Baseline (nginx-baseline)
Version: 2.1.0
Target:  ssh://ubuntu@x.x.x.x:22

  ✔  nginx-01: Running worker process as non-privileged user
     ✔  User www-data should exist
     ✔  Parse Config File /etc/nginx/nginx.conf user should eq "www-data"
     ✔  Parse Config File /etc/nginx/nginx.conf group should not eq "root"
  ✔  nginx-02: Check NGINX config file owner, group and permissions.
     ✔  File /etc/nginx/nginx.conf should be owned by "root"
     ✔  File /etc/nginx/nginx.conf should be grouped into "root"
     ✔  File /etc/nginx/nginx.conf should not be readable by others
     ✔  File /etc/nginx/nginx.conf should not be writable by others
     ✔  File /etc/nginx/nginx.conf should not be executable by others
  ✔  nginx-03: Nginx default files
     ✔  File /etc/nginx/conf.d/default.conf should not be file
     ✔  File /etc/nginx/sites-enabled/default should not be file
     ✔  File /etc/nginx/nginx.conf should be file
     ✔  File /etc/nginx/conf.d/90.hardening.conf should be file
  ✔  nginx-04: Check for multiple instances
     ✔  Command ps aux | egrep "nginx: master" | egrep -v "grep" | wc -l stdout should match /^1$/
  ✔  nginx-05: Disable server_tokens directive
     ✔  Parse Config File /etc/nginx/nginx.conf server_tokens should eq "off"
  ×  nginx-06: Prevent buffer overflow attacks (2 failed)
     ×  Parse Config File /etc/nginx/nginx.conf client_body_buffer_size should eq "1k"
     
     expected: "1k"
          got: "1M"
     
     (compared using ==)

     ×  Parse Config File /etc/nginx/nginx.conf client_max_body_size should eq "1k"
     
     expected: "1k"
          got: "1M"
     
     (compared using ==)

     ✔  Parse Config File /etc/nginx/conf.d/90.hardening.conf client_header_buffer_size should eq "1k"
     ✔  Parse Config File /etc/nginx/conf.d/90.hardening.conf large_client_header_buffers should eq "2 1k"
  ×  nginx-07: Control simultaneous connections (1 failed)
     ✔  Parse Config File /etc/nginx/conf.d/90.hardening.conf limit_conn_zone should eq "$binary_remote_addr zone=default:10m"
     ×  Parse Config File /etc/nginx/conf.d/90.hardening.conf limit_conn should eq "default 5"
     
     expected: "default 5"
          got: "default 1024"
     
     (compared using ==)

  ✔  nginx-08: Prevent clickjacking
     ✔  Parse Config File /etc/nginx/conf.d/90.hardening.conf add_header should include "X-Frame-Options SAMEORIGIN"
  ✔  nginx-09: Enable Cross-site scripting filter
     ✔  Parse Config File /etc/nginx/conf.d/90.hardening.conf add_header should include "X-XSS-Protection \"1; mode=block\""
  ✔  nginx-10: Disable content-type sniffing
     ✔  Parse Config File /etc/nginx/conf.d/90.hardening.conf add_header should include "X-Content-Type-Options nosniff"
  ✔  nginx-11: Disable content-type sniffing
     ✔  Parse Config File /etc/nginx/conf.d/90.hardening.conf add_header should include "X-Content-Type-Options nosniff"
  ×  nginx-13: Add HSTS Header
     ×  File /etc/nginx/conf.d/90.hardening.conf content should match /^\s*add_header Strict-Transport-Security max-age=15768000;$/
     expected "# Ansible managed\n# Additional configuration for Nginx.\n\nclient_header_buffer_size 1k;\nlarge_cli...IGIN;\nadd_header X-Content-Type-Options nosniff;\nadd_header X-XSS-Protection \"1; mode=block\";\n" to match /^\s*add_header Strict-Transport-Security max-age=15768000;$/
     Diff:
     @@ -1,2 +1,18 @@
     -/^\s*add_header Strict-Transport-Security max-age=15768000;$/
     +# Ansible managed
     +# Additional configuration for Nginx.
     +
     +client_header_buffer_size 1k;
     +large_client_header_buffers 2 1k;
     +client_body_timeout 10;
     +client_header_timeout 10;
     +send_timeout 10;
     +limit_conn_zone $binary_remote_addr zone=default:10m;
     +limit_conn default 1024;
     +ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
     +ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA;
     +ssl_dhparam /etc/nginx/dh2048.pem;
     +ssl_prefer_server_ciphers on;
     +add_header X-Frame-Options SAMEORIGIN;
     +add_header X-Content-Type-Options nosniff;
     +add_header X-XSS-Protection "1; mode=block";

  ×  nginx-14: Disable insecure HTTP-methods
     ×  File /etc/nginx/nginx.conf content should match /^\s*if\s+\(\$request_method\s+\!\~\s+\^\(GET\|HEAD\|POST\)\$\)\{?$/
     expected "user  www-data;\n\nerror_log  /var/log/nginx/error.log warn;\npid        /run/nginx.pid;\n\nworker_p...n;\n\n\n\n    include /etc/nginx/conf.d/*.conf;\n    include /etc/nginx/sites-enabled/*;\n\n    }\n" to match /^\s*if\s+\(\$request_method\s+\!\~\s+\^\(GET\|HEAD\|POST\)\$\)\{?$/
     Diff:
     @@ -1,2 +1,48 @@
     -/^\s*if\s+\(\$request_method\s+\!\~\s+\^\(GET\|HEAD\|POST\)\$\)\{?$/
     +user  www-data;
     +
     +error_log  /var/log/nginx/error.log warn;
     +pid        /run/nginx.pid;
     +
     +worker_processes  1;
     +
     +
     +events {
     +    worker_connections  1024;
     +    multi_accept off;
     +}
     +
     +http {
     +client_body_buffer_size 1M;
     +    
     +    include       /etc/nginx/mime.types;
     +    default_type  application/octet-stream;
     +
     +    server_names_hash_bucket_size 64;
     +
     +client_max_body_size 1M;
     +
     +    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
     +                       '$status $body_bytes_sent "$http_referer" '
     +                       '"$http_user_agent" "$http_x_forwarded_for"'
     +;
     +
     +    access_log  /var/log/nginx/access.log main buffer=16k flush=2m;
     +
     +    sendfile        on;
     +    tcp_nopush      on;
     +    tcp_nodelay     on;
     +
     +keepalive_timeout 5 5;
     +    keepalive_requests 100;
     +
     +server_tokens off;
     +
     +    # gzip on;
     +
     +
     +
     +    include /etc/nginx/conf.d/*.conf;
     +    include /etc/nginx/sites-enabled/*;
     +
     +    }

  ×  nginx-15: Content-Security-Policy
     ×  Parse Config File /etc/nginx/conf.d/90.hardening.conf content should match /^\s*add_header Content-Security-Policy "script-src 'self'; object-src 'self'";$/
     expected "# Ansible managed\n# Additional configuration for Nginx.\n\nclient_header_buffer_size 1k;\nlarge_cli...IGIN;\nadd_header X-Content-Type-Options nosniff;\nadd_header X-XSS-Protection \"1; mode=block\";\n" to match /^\s*add_header Content-Security-Policy "script-src 'self'; object-src 'self'";$/
     Diff:
     @@ -1,2 +1,18 @@
     -/^\s*add_header Content-Security-Policy "script-src 'self'; object-src 'self'";$/
     +# Ansible managed
     +# Additional configuration for Nginx.
     +
     +client_header_buffer_size 1k;
     +large_client_header_buffers 2 1k;
     +client_body_timeout 10;
     +client_header_timeout 10;
     +send_timeout 10;
     +limit_conn_zone $binary_remote_addr zone=default:10m;
     +limit_conn default 1024;
     +ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
     +ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA;
     +ssl_dhparam /etc/nginx/dh2048.pem;
     +ssl_prefer_server_ciphers on;
     +add_header X-Frame-Options SAMEORIGIN;
     +add_header X-Content-Type-Options nosniff;
     +add_header X-XSS-Protection "1; mode=block";

  ×  nginx-16: Set cookie with HttpOnly and Secure flag
     ×  Parse Config File /etc/nginx/conf.d/90.hardening.conf content should match /^\s*set_cookie_flag * HttpOnly secure;$/
     expected "# Ansible managed\n# Additional configuration for Nginx.\n\nclient_header_buffer_size 1k;\nlarge_cli...IGIN;\nadd_header X-Content-Type-Options nosniff;\nadd_header X-XSS-Protection \"1; mode=block\";\n" to match /^\s*set_cookie_flag * HttpOnly secure;$/
     Diff:
     @@ -1,2 +1,18 @@
     -/^\s*set_cookie_flag * HttpOnly secure;$/
     +# Ansible managed
     +# Additional configuration for Nginx.
     +
     +client_header_buffer_size 1k;
     +large_client_header_buffers 2 1k;
     +client_body_timeout 10;
     +client_header_timeout 10;
     +send_timeout 10;
     +limit_conn_zone $binary_remote_addr zone=default:10m;
     +limit_conn default 1024;
     +ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
     +ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA;
     +ssl_dhparam /etc/nginx/dh2048.pem;
     +ssl_prefer_server_ciphers on;
     +add_header X-Frame-Options SAMEORIGIN;
     +add_header X-Content-Type-Options nosniff;
     +add_header X-XSS-Protection "1; mode=block";

the controls 6 and 7 are failing, because we have tweaked these values for our projects. Of course we should be able to override these values in Ansible with the proper ones, which are required by the baseline. But I think it would be better if the default-values of the ansible-role match the required values by the baseline.

This issue I opened is only a problem when using baseline v2.0.2 and inspec > 2. With inspec1.x the baseline works fine, despite the deprecated-message.

[DEPRECATION] `:assignment_re` is deprecated in favor of `:assignment_regex` and will be removed in the next major version.