diff --git a/ansible/README-os-idr.md b/ansible/README-os-idr.md index 5b1d50072..f1887c38a 100644 --- a/ansible/README-os-idr.md +++ b/ansible/README-os-idr.md @@ -10,84 +10,67 @@ Most of these scripts should also work on other platforms, providing the VM is b The guest must be running CentOS 7. -Guide for the Impatient ------------------------ +Openstack: Creation of instances, volumes and security groups +------------------------------------------------------------- -Setup your OpenStack environment variables, and run: +[Setup your OpenStack environment variables](http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html), edit the variables in `os-idr-playbooks/os-idr-create-example.yml` (especially `idr_vm_keyname` and `idr_environment`), then run: - ansible-playbook -i inventory -e omero_vm_name=FOO -e omero_vm_key_name=YOUR_KEY os-idr-uod.yml + ansible-playbook os-idr-playbooks/os-idr-create-example.yml -`os-idr-playbooks/os-omero.yml` -------------------------------- - -This is the Ansible playbook that will be run to setup OMERO. -You can also run it manually to install OMERO on localhost. - - -`os-idr-playbooks/os-create.yml` --------------------------------- +Openstack: Installing the IDR +----------------------------- -This playbook will connect to OpenStack and spin up a VM. -The Ansible modules in this playbook require the `shade` Python module. +Find the floating IP of the proxy/bastion server. +Set `BASTION_IP` to the IP, and `IDR_ENVIRONMENT` to match the value from above. +Run: -Before running the playbook you must [setup your OpenStack environment variables](http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html). -You can override variables at the command line, for example (note double quoting is necessary if spaces are present): + BASTION_IP=10.0.0.0 + IDR_ENVIRONMENT=idr + ansible-playbook \ + -i inventory/openstack-private.py \ + -u centos \ + -e idr_environment=$IDR_ENVIRONMENT \ + -e idr_nginx_ssl_self_signed=True \ + -e ansible_ssh_common_args="'-o ProxyCommand=\\\"ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -W %h:%p -q centos@$BASTION_IP\\\" -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no'" \ + idr-playbooks/os-idr-volumes.yml \ + idr-playbooks/idr-dundee-nfs.yml \ + idr-playbooks/idr-ebi-nfs.yml \ + idr-playbooks/idr.yml \ + idr-playbooks/idr-docker.yml - ansible-playbook os-idr-playbooks/os-create.yml -e omero_vm_name=FOO \ - -e omero_vm_key_name=YOURKEY -e "omero_vm_flavour='m2.xxlarge'" -If this step fails it could be due to an incorrect variable, the Ansible `os_server` module usually gives an uninformative "Error in creating instance" message. -If the VM was created the floating IP of the VM will be printed out. +Deploying the IDR on existing infrastructure +-------------------------------------------- -To delete the VM and related security group: +If you have already created your servers and just wish to install a plain IDR then run: - ansible-playbook os-idr-playbooks/os-delete.yml -e omero_vm_name=FOO + ansible-playbook \ + -i inventory \ + -u centos \ + -e idr_environment=$IDR_ENVIRONMENT \ + -e idr_nginx_ssl_self_signed=True \ + idr-playbooks/idr-omero.yml -If another instance is using the OMERO security group, the task will fail but can be safely ignored. +where `inventory` contains groups described in the following section. -Inventory ---------- +`idr-playbooks/idr-omero.yml` +----------------------------- -This directory takes advantage of an -[Ansible dynamic inventory script for OpenStack (`openstack.py`)](http://docs.ansible.com/ansible/intro_dynamic_inventory.html#example-openstack-external-inventory-script) -instead of having to manage an inventory file when using Ansible to push out changes. -For example: +This is the Ansible playbook that will be run to setup OMERO. +This can be run independently of the openstack playbooks providing you have an inventory with groups: +- `{{ idr_environment }}-data-hosts` +- `{{ idr_environment }}-omero-hosts` +- `{{ idr_environment }}-proxy-hosts` - ansible-playbook -i inventory -l os-image-centos os-idr-playbooks/os-omero.yml -vv -Variables for the groups defined in `os-idr-playbooks/os-create.yml` as `omero_vm_groups` can be added under inventory/variables. +TODO: explain other `idr-playbooks/*.yml` playbooks Deploying the IDR ================= - -The production IDR is setup using a private configuration repository. -Replace `{{ inventory_dir }}` with the path to the inventory directory. -You can use `inventory` in this directory if you have configured the required variables, such as by creating a group_vars file if necessary in `{{ inventory_dir }}/group_vars/`, e.g. `{{ inventory_dir }}/group_vars/os-idr.yml` -This should match the value of the `idr_environment` variable (default `os-idr`), and can be used to support multiple deployment environments with different variables. - -Decide on your openstack dynamic inventory. -If you are using a single floating IP use `{{ inventory_dir }}/openstack-private.py`. -using private internal IPs and a gateway server on the Openstack cloud. -If you are using floating IPs for all instances you can optionally use `{{ inventory_dir }}/openstack.py` instead. - -Select your playbook, for instance `os-idr-uod.yml` for the Dundee cloud. - -For example (using the default `os-idr` host-group and variables): - - ansible-playbook -i {{ inventory_dir }}/openstack-private.py os-idr-uod.yml - -e vm_key_name="KEY_NAME" -e vm_prefix=PREFIX - -Or using a custom group called `os-idrstaging` with additional variable overrides: - - ansible-playbook -i {{ inventory_dir }}/openstack-private.py os-idr-uod.yml - -e vm_key_name="KEY_NAME" -e vm_prefix=PREFIX - -e @vars/test-overrides.yml -e idr_environment=os-idrstaging - - Component playbooks ------------------- diff --git a/ansible/Vagrantfile b/ansible/Vagrantfile index b89dc4581..e244bd6bc 100644 --- a/ansible/Vagrantfile +++ b/ansible/Vagrantfile @@ -63,14 +63,16 @@ Vagrant.configure(2) do |config| ansible.playbook = "idr-playbooks/idr-omero.yml" ansible.limit = "idr" ansible.groups = { - "database-hosts" => ["idr-database"], - "omero-hosts" => ["idr-omero"], - "proxy-hosts" => ["idr-gateway"], - "idr:children" => ["database-hosts", "omero-hosts", "proxy-hosts"], + "idr-database-hosts" => ["idr-database"], + "idr-omero-hosts" => ["idr-omero"], + "idr-proxy-hosts" => ["idr-gateway"], + "idr:children" => ["idr-database-hosts", "idr-omero-hosts", "idr-proxy-hosts"], "idr:vars" => { # Vagrant uses eth0 for NAT, eth1 for private guest network "idr_net_iface" => "eth1", "idr_nginx_ssl_self_signed" => "True", + "omero_release" => "0.0.7-rc1", + "omero_omego_additional_args" => "--downloadurl https://downloads.openmicroscopy.org/idr", } } end diff --git a/ansible/idr-playbooks/files/IDR-OMERO-52-omero.j2 b/ansible/idr-playbooks/files/IDR-OMERO-52-omero.j2 index 672219146..8dab32b13 100644 --- a/ansible/idr-playbooks/files/IDR-OMERO-52-omero.j2 +++ b/ansible/idr-playbooks/files/IDR-OMERO-52-omero.j2 @@ -26,7 +26,7 @@ config set omero.web.secure_proxy_ssl_header '["HTTP_X_FORWARDED_PROTO_OMERO_WEB config set omero.web.public.enabled True config set omero.web.public.server_id 1 config set omero.web.public.user public -config set omero.web.public.password {{ idr_secret_omero_web_public_password | default("") }} +config set omero.web.public.password {{ omero_web_public_password }} config set omero.web.public.url_filter ^/(webadmin/myphoto/|webclient/(?!(action|annotate_(file|tags|comment|rating|map)|script_ui|ome_tiff|figure_script))|webgateway/(?!(archived_files|download_as))) # Group/User drop down menu diff --git a/ansible/idr-playbooks/group_vars/omero-hosts.yml b/ansible/idr-playbooks/group_vars/omero-hosts.yml index 70996a47c..b594e8e12 100644 --- a/ansible/idr-playbooks/group_vars/omero-hosts.yml +++ b/ansible/idr-playbooks/group_vars/omero-hosts.yml @@ -13,10 +13,11 @@ omero_omego_additional_args: "--downloadurl https://downloads.openmicroscopy.org # Disable database backups omero_database_backupdir: -omero_upgrade: True # Recursively chown data dir (this may take a very long time) #omero_datadir_chown: True +omero_web_public_password: "{{ idr_secret_omero_web_public_password | default('public') }}" + omero_prestart_file: "{{ playbook_dir }}/files/IDR-OMERO-52-omero.j2" omero_logmonitor_slack_token: "{{ idr_secret_omero_logmonitor_slack_token | default(None) }}" diff --git a/ansible/idr-playbooks/group_vars/proxy-hosts.yml b/ansible/idr-playbooks/group_vars/proxy-hosts.yml index c7a11d782..71559a38f 100644 --- a/ansible/idr-playbooks/group_vars/proxy-hosts.yml +++ b/ansible/idr-playbooks/group_vars/proxy-hosts.yml @@ -54,4 +54,4 @@ nginx_proxy_direct_locations: #nginx_proxy_block_locations: #- "^~ /login" -nginx_proxy_set_header_host: 'idr-demo.openmicroscopy.org' +#nginx_proxy_set_header_host: 'idr-demo.openmicroscopy.org' diff --git a/ansible/idr-playbooks/idr-dundee-nfs.yml b/ansible/idr-playbooks/idr-dundee-nfs.yml index c354561b2..62eb8833a 100644 --- a/ansible/idr-playbooks/idr-dundee-nfs.yml +++ b/ansible/idr-playbooks/idr-dundee-nfs.yml @@ -2,7 +2,7 @@ # The default is to use NFS, if you are using samba you must install the # dependencies (cifs-utils) yourself. -- hosts: uod-nfs +- hosts: "{{ idr_environment | default('idr') }}-uod-nfs" vars: idr_mountpoint: /uod/idr diff --git a/ansible/idr-playbooks/idr-local-files.yml b/ansible/idr-playbooks/idr-local-files.yml index 157a4e07b..0604ff083 100644 --- a/ansible/idr-playbooks/idr-local-files.yml +++ b/ansible/idr-playbooks/idr-local-files.yml @@ -5,7 +5,7 @@ # This should be cleaned up and moved/combined into an appropriate role # Variables should be in a private group_vars file -- hosts: proxy-hosts +- hosts: "{{ idr_environment | default('idr') }}-proxy-hosts" tasks: diff --git a/ansible/idr-playbooks/idr-local-users.yml b/ansible/idr-playbooks/idr-local-users.yml index 818d47292..ad789e74f 100644 --- a/ansible/idr-playbooks/idr-local-users.yml +++ b/ansible/idr-playbooks/idr-local-users.yml @@ -2,7 +2,7 @@ # Playbook for creating local user accounts on Openstack instances # Variables should be in a private group_vars file -- hosts: database-hosts, omero-hosts, proxy-hosts +- hosts: "{{ idr_environment | default('idr') }}-database-hosts, {{ idr_environment | default('idr') }}-omero-hosts, {{ idr_environment | default('idr') }}-proxy-hosts" roles: - role: sudoers # sudoers_individual_commands: diff --git a/ansible/idr-playbooks/idr-monitoring.yml b/ansible/idr-playbooks/idr-monitoring.yml index 54ffbe656..16fab3afa 100644 --- a/ansible/idr-playbooks/idr-monitoring.yml +++ b/ansible/idr-playbooks/idr-monitoring.yml @@ -1,6 +1,6 @@ # Monitoring playbook -- hosts: omero-hosts +- hosts: "{{ idr_environment | default('idr') }}-omero-hosts" pre_tasks: - name: Get short hostname (not the same as the Ansible hostname vars) diff --git a/ansible/idr-playbooks/idr-omero.yml b/ansible/idr-playbooks/idr-omero.yml index f4aca3fbb..11295d6fe 100644 --- a/ansible/idr-playbooks/idr-omero.yml +++ b/ansible/idr-playbooks/idr-omero.yml @@ -12,8 +12,7 @@ # - `idr_net_iface=iface` if your servers use a network interface other # then eth0 for inter-machine networking - -- hosts: database-hosts +- hosts: "{{ idr_environment | default('idr') }}-database-hosts" roles: - role: postgresql @@ -29,7 +28,7 @@ postgresql_server_chown_datadir: True -- hosts: omero-hosts +- hosts: "{{ idr_environment | default('idr') }}-omero-hosts" pre_tasks: @@ -44,15 +43,21 @@ become: yes - set_fact: - # omero_db_host_ansible: "{{ hostvars[groups['database-hosts'][0]]['ansible_ssh_host'] | default(hostvars[groups['database-hosts'][0]]['ansible_host']) }}" - # omero_db_host_ansible: "{{ hostvars[groups['database-hosts'][0]]['ansible_' + (idr_net_iface | default('eth0'))]['ipv4']['address']}}" + omero_db_host_ansible: "{{ hostvars[groups[idr_environment | default('idr') + '-database-hosts'][0]]['ansible_' + (idr_net_iface | default('eth0'))]['ipv4']['address']}}" roles: - - { role: upgrade-distpackages, tags: "upgrade-distpackages" } - - { role: versioncontrol-utils, tags: "versioncontrol-utils" } - - { role: omero-server, tags: "omero-server" } - - { role: python-pydata, tags: "python-pydata" } - - { role: omero-web-apps, tags: "omero-web-apps" } + - role: basedeps + tags: "basedeps" + - role: cli-utils + tags: "cli-utils" + - role: versioncontrol-utils + tags: "versioncontrol-utils" + - role: omero-server + tags: "omero-server" + - role: python-pydata + tags: "python-pydata" + - role: omero-web-apps + tags: "omero-web-apps" vars: omero_dbhost: "{{ omero_db_host_ansible }}" @@ -110,13 +115,12 @@ # Additional vars are in group_vars/omero-hosts.yml -- hosts: proxy-hosts +- hosts: "{{ idr_environment | default('idr') }}-proxy-hosts" pre_tasks: - set_fact: - # omero_omero_host_ansible: "{{ hostvars[groups['omero-hosts'][0]]['ansible_ssh_host'] | default(hostvars[groups['omero-hosts'][0]]['ansible_host']) }}" - # omero_omero_host_ansible: "{{ hostvars[groups['omero-hosts'][0]]['ansible_' + (idr_net_iface | default('eth0'))]['ipv4']['address']}}" + omero_omero_host_ansible: "{{ hostvars[groups[idr_environment | default('idr') + '-omero-hosts'][0]]['ansible_' + (idr_net_iface | default('eth0'))]['ipv4']['address']}}" roles: - role: nginx-ssl-selfsigned diff --git a/ansible/idr-playbooks/idr-user-utils.yml b/ansible/idr-playbooks/idr-user-utils.yml deleted file mode 100644 index 114054bfb..000000000 --- a/ansible/idr-playbooks/idr-user-utils.yml +++ /dev/null @@ -1,33 +0,0 @@ ---- -# Playbook for accessing idr metadata - -- hosts: idr-hosts - - roles: - - role: versioncontrol-utils - - role: python-pydata - - tasks: - - - name: Install screen - become: yes - yum: - pkg: screen - state: present - - - name: Create metadata directory - become: yes - file: - path: /opt/idr-metadata - owner: omero - group: omero - recurse: yes - state: directory - - - name: Clone metadata repo - become: yes - become_user: omero - git: - repo: git://github.com/snoopycrimecop/idr-metadata.git - dest: /opt/idr-metadata - version: merge/trigger diff --git a/ansible/idr-playbooks/idr.yml b/ansible/idr-playbooks/idr.yml index 178cfb134..00f594f60 100644 --- a/ansible/idr-playbooks/idr.yml +++ b/ansible/idr-playbooks/idr.yml @@ -1,5 +1,6 @@ -# Runs all public playbooks for setting up the IDR infrastructure -# The remaining playbooks require additional private configuration +# Runs all public playbooks for setting up the IDR infrastructure in any +# environment. This does not run any storage/networking/cloud specific +# tasks, nor does it run playbooks requiring private configuration - include: idr-local-users.yml - include: idr-omero.yml - include: idr-local-files.yml diff --git a/ansible/idr-playbooks/os-idr-volumes.yml b/ansible/idr-playbooks/os-idr-volumes.yml new file mode 100644 index 000000000..d457b9602 --- /dev/null +++ b/ansible/idr-playbooks/os-idr-volumes.yml @@ -0,0 +1,27 @@ +--- +# Initialise openstack volumes from inside VMs if necessary + +- hosts: "{{ idr_environment | default('idr') }}-database-hosts" + roles: + - role: storage-volume-initialise + storage_volume_initialise_device: "{{ database_db_vol_dev | default('/dev/vdb') }}" + storage_volume_initialise_mount: /var/lib/pgsql + +- hosts: "{{ idr_environment | default('idr') }}-omero-hosts" + roles: + - role: storage-volume-initialise + storage_volume_initialise_device: "{{ omero_data_vol_dev | default('/dev/vdb') }}" + storage_volume_initialise_mount: /data + +- hosts: "{{ idr_environment | default('idr') }}-proxy-hosts" + roles: + - role: storage-volume-initialise + storage_volume_initialise_device: "{{ gateway_nginxcache_vol_dev | default('/dev/vdb') }}" + storage_volume_initialise_mount: /var/cache/nginx + +# Use this group for any other IDR VMs that should have a volume mounted on /data +- hosts: "{{ idr_environment | default('idr') }}-data-hosts" + roles: + - role: storage-volume-initialise + storage_volume_initialise_device: "{{ data_vol_dev | default('/dev/vdb') }}" + storage_volume_initialise_mount: /data diff --git a/ansible/os-idr-ebi.yml b/ansible/os-idr-ebi.yml index a93e9721c..8eb891a71 100644 --- a/ansible/os-idr-ebi.yml +++ b/ansible/os-idr-ebi.yml @@ -4,7 +4,7 @@ - include: os-idr-playbooks/os-create.yml vars: - omero_vm_extra_groups: "ebi-nfs,idr-hosts" + omero_vm_extra_groups: "{{ idr_environment | default('idr') }}-ebi-nfs" os_cloud_provider: ebi - include: os-idr-playbooks/os-volumes.yml @@ -14,19 +14,19 @@ - include: idr-playbooks/idr-local-users.yml # Variables for this section are in a private file -- hosts: database-hosts +- hosts: "{{ idr_environment | default('idr') }}-database-hosts" roles: - role: storage-volume-initialise storage_volume_initialise_device: /dev/vdb storage_volume_initialise_mount: /var/lib/pgsql -- hosts: omero-hosts +- hosts: "{{ idr_environment | default('idr') }}-omero-hosts" roles: - role: storage-volume-initialise storage_volume_initialise_device: /dev/vdb storage_volume_initialise_mount: /data -- hosts: proxy-hosts +- hosts: "{{ idr_environment | default('idr') }}-proxy-hosts" roles: - role: storage-volume-initialise storage_volume_initialise_device: /dev/vdb @@ -39,6 +39,4 @@ - include: idr-playbooks/idr-local-files.yml -- include: idr-playbooks/idr-user-utils.yml - #- include: idr-playbooks/idr-monitoring.yml diff --git a/ansible/os-idr-playbooks/os-create.yml b/ansible/os-idr-playbooks/os-create.yml index 66cc478ec..99ab19a0f 100644 --- a/ansible/os-idr-playbooks/os-create.yml +++ b/ansible/os-idr-playbooks/os-create.yml @@ -7,16 +7,26 @@ connection: local #gather_facts: false + pre_tasks: + + - fail: + msg: "vm_key_name is required" + when: vm_key_name is undefined or not vm_key_name + + - set_fact: + idr_environment: idr + when: idr_environment is undefined + + vars: # idr_environment: All VMs will be put into this group, which should have # a matching group_vars file - omero_vm_groups: "ansible-managed,os-image-centos,omero-hosts,{{ idr_environment | default('os-idr') }}" + omero_vm_groups: "ansible-managed,os-image-centos,omero-hosts,{{ idr_environment }}-omero-hosts,{{ idr_environment }}-hosts" omero_vm_extra_groups: "" - gateway_vm_groups: "ansible-managed,os-image-centos,proxy-hosts,{{ idr_environment | default('os-idr') }}" - database_vm_groups: "ansible-managed,os-image-centos,database-hosts,{{ idr_environment | default('os-idr') }}" + gateway_vm_groups: "ansible-managed,os-image-centos,proxy-hosts,{{ idr_environment }}-proxy-hosts,{{ idr_environment }}-hosts" + database_vm_groups: "ansible-managed,os-image-centos,database-hosts,{{ idr_environment }}-database-hosts,{{ idr_environment }}-hosts" - #vm_prefix: #vm_key_name: ignore_internal_known_hosts: True @@ -26,17 +36,6 @@ vars_files: - [ "{{ inventory_dir }}/vars/os-create-{{ os_cloud_provider }}.yml", "vars/os-create-default.yml" ] - pre_tasks: - - - fail: - msg: "vm_key_name is required" - when: vm_key_name is undefined or not vm_key_name - - - fail: - msg: "vm_prefix is required" - when: vm_prefix is undefined or not vm_prefix - - tasks: # If True (default) a single gateway will be setup diff --git a/ansible/os-idr-playbooks/os-delete.yml b/ansible/os-idr-playbooks/os-delete.yml index 0255c6dd5..a3f0baf4e 100644 --- a/ansible/os-idr-playbooks/os-delete.yml +++ b/ansible/os-idr-playbooks/os-delete.yml @@ -8,12 +8,12 @@ tasks: - fail: - msg: "vm_prefix is required" - when: vm_prefix is undefined or not vm_prefix + msg: "idr_environment is required" + when: idr_environment is undefined or not idr_environment - name: Remove instances os_server: - name: "{{ vm_prefix }}-{{ item }}" + name: "{{ idr_environment }}-{{ item }}" state: absent with_items: - database @@ -22,7 +22,7 @@ - name: Remove instances os_server: - name: "{{ vm_prefix }}-{{ item }}" + name: "{{ idr_environment }}-{{ item }}" state: absent with_items: - database @@ -35,9 +35,9 @@ display_name: "{{ item }}" state: absent with_items: - - "{{ vm_prefix }}-omero-data" - - "{{ vm_prefix }}-database-db" - - "{{ vm_prefix }}-gateway-nginxcache" + - "{{ idr_environment }}-omero-data" + - "{{ idr_environment }}-database-db" + - "{{ idr_environment }}-gateway-nginxcache" # Can't remove security group unless nothing is using it - name: Remove OMERO external access security group diff --git a/ansible/os-idr-playbooks/os-idr-create-example.yml b/ansible/os-idr-playbooks/os-idr-create-example.yml new file mode 100644 index 000000000..21f8e85b5 --- /dev/null +++ b/ansible/os-idr-playbooks/os-idr-create-example.yml @@ -0,0 +1,73 @@ +--- +# Example playbook for creating OpenStack IDR VMs +# You will need to change the variables in the `vars` section depending on +# your openstack environment. For more fine grained control set the vars +# directly in each role + +- hosts: localhost + connection: local + + vars: + - idr_environment: idr + #- idr_vm_keyname: VM_KEYNAME + - vm_image: CentOS 7 1604 + - vm_flavour: m1.large + + + roles: + + ############################################################ + # Security groups + + - role: openstack-idr-security-groups + + + ############################################################ + # Instances + + # Dedicated database server + - role: openstack-idr-instance + idr_vm_name: "{{ idr_environment }}-database" + idr_vm_image: "{{ vm_image }}" + idr_vm_flavour: "{{ vm_flavour }}" + idr_vm_database: True + + # OMERO server + - role: openstack-idr-instance + idr_vm_name: "{{ idr_environment }}-omero" + idr_vm_image: "{{ vm_image }}" + idr_vm_flavour: "{{ vm_flavour }}" + idr_vm_omero: True + #idr_vm_extra_groups: + #idr_vm_private_networks: + + # Proxy server, doubles up as a bastion server + - role: openstack-idr-instance + idr_vm_name: "{{ idr_environment }}-proxy" + idr_vm_image: "{{ vm_image }}" + idr_vm_flavour: "{{ vm_flavour }}" + idr_vm_proxy: True + idr_vm_bastion: True + idr_vm_assign_floating_ip: True + + + ############################################################ + # Volumes + + - role: openstack-volume-storage + openstack_volume_size: 100 + openstack_volume_vmname: "{{ idr_environment }}-database" + openstack_volume_name: db + openstack_volume_device: /dev/vdb + + - role: openstack-volume-storage + openstack_volume_size: 100 + openstack_volume_vmname: "{{ idr_environment }}-omero" + openstack_volume_name: data + openstack_volume_device: /dev/vdb + + - role: openstack-volume-storage + openstack_volume_size: 20 + openstack_volume_vmname: "{{ idr_environment }}-proxy" + openstack_volume_name: nginxcache + openstack_volume_device: /dev/vdb diff --git a/ansible/os-idr-playbooks/vars/os-create-default.yml b/ansible/os-idr-playbooks/vars/os-create-default.yml index 275bee279..484fe9332 100644 --- a/ansible/os-idr-playbooks/vars/os-create-default.yml +++ b/ansible/os-idr-playbooks/vars/os-create-default.yml @@ -7,8 +7,8 @@ omero_vm_flavour: m2.large gateway_vm_flavour: m1.large database_vm_flavour: m1.large -omero_vm_name: "{{ vm_prefix }}-omero" -gateway_vm_name: "{{ vm_prefix }}-gateway" -database_vm_name: "{{ vm_prefix }}-database" +omero_vm_name: "{{ idr_environment }}-omero" +gateway_vm_name: "{{ idr_environment }}-gateway" +database_vm_name: "{{ idr_environment }}-database" # Assume there's only one network in this tenancy so no need to specify network diff --git a/ansible/os-idr-uod.yml b/ansible/os-idr-uod.yml index 73ec16858..b564a71a4 100644 --- a/ansible/os-idr-uod.yml +++ b/ansible/os-idr-uod.yml @@ -4,7 +4,7 @@ - include: os-idr-playbooks/os-create.yml vars: - omero_vm_extra_groups: "uod-nfs,idr-hosts" + omero_vm_extra_groups: "{{ idr_environment | default('idr') }}-uod-nfs" os_cloud_provider: uod - include: os-idr-playbooks/os-volumes.yml @@ -14,19 +14,19 @@ - include: idr-playbooks/idr-local-users.yml # Variables for this section are in a private file -- hosts: database-hosts +- hosts: "{{ idr_environment | default('idr') }}-database-hosts" roles: - role: storage-volume-initialise storage_volume_initialise_device: /dev/vdb storage_volume_initialise_mount: /var/lib/pgsql -- hosts: omero-hosts +- hosts: "{{ idr_environment | default('idr') }}-omero-hosts" roles: - role: storage-volume-initialise storage_volume_initialise_device: /dev/vdb storage_volume_initialise_mount: /data -- hosts: proxy-hosts +- hosts: "{{ idr_environment | default('idr') }}-proxy-hosts" roles: - role: storage-volume-initialise storage_volume_initialise_device: /dev/vdb @@ -39,6 +39,4 @@ - include: idr-playbooks/idr-local-files.yml -- include: idr-playbooks/idr-user-utils.yml - #- include: idr-playbooks/idr-monitoring.yml diff --git a/ansible/roles/omero-web-runtime/tasks/main.yml b/ansible/roles/omero-web-runtime/tasks/main.yml index 3ba16e672..fbd888c11 100644 --- a/ansible/roles/omero-web-runtime/tasks/main.yml +++ b/ansible/roles/omero-web-runtime/tasks/main.yml @@ -48,8 +48,12 @@ - name: omero | install selinux utilities become: yes yum: - name: libselinux-python + name: "{{ item }}" state: present + with_items: + - libselinux-python + - libsemanage-python + - policycoreutils-python when: omero_selinux_setup - name: omero web | selinux booleans @@ -71,4 +75,4 @@ proto: tcp setype: http_port_t state: present - when: omero_selinux_setup \ No newline at end of file + when: omero_selinux_setup diff --git a/ansible/roles/openstack-idr-instance/README.md b/ansible/roles/openstack-idr-instance/README.md new file mode 100644 index 000000000..210cbac75 --- /dev/null +++ b/ansible/roles/openstack-idr-instance/README.md @@ -0,0 +1,44 @@ +Openstack IDR Instance +====================== + +Create an Openstack VM for use with the IDR playbooks. + + +Role Variables +-------------- + +Defaults: `defaults/main.yml` + +Required variables: +- `idr_vm_name`: VM hostname +- `idr_vm_image`: Openstack base image +- `idr_vm_keyname`: Openstack SSH key +- `idr_vm_flavour`: Openstack flavour + +Optional variables: +- `idr_vm_private_networks`: Use this network instead of the default one +- `idr_vm_assign_floating_ip`: Assign a floating IP, default `False` +- `idr_environment`: Use this as a group prefix. You should almost always set this to something other than the default `idr` + +Booleans indicating the purpose of this server: +- If any of these are `True` they will be used to automatically set the security groups and host-groups for this VM, default `False`. Multiple may be set to `True` if a server has multiple purposes. + - `idr_vm_database`: An IDR database server + - `idr_vm_omero`: An IDR OMERO server + - `idr_vm_proxy`: An IDR web proxy server + +Advanced settings: +- `idr_vm_groups`: A list of host-groups, default depends on the above booleans +- `idr_vm_extra_groups`: A list of host-groups in addition to the above default +- `idr_vm_security_groups`: A list of security groups, default depends on the above booleans + + +Development +----------- + +See the warning in `tasks/main.yml` before making changes. + + +Author Information +------------------ + +ome-devel@lists.openmicroscopy.org.uk diff --git a/ansible/roles/openstack-idr-instance/defaults/main.yml b/ansible/roles/openstack-idr-instance/defaults/main.yml new file mode 100644 index 000000000..045a549b7 --- /dev/null +++ b/ansible/roles/openstack-idr-instance/defaults/main.yml @@ -0,0 +1,82 @@ +--- +# defaults file for roles/openstack-idr-instance +# This contains a lot of the logic for setting the host and security +# groups based on the purpose of the VM, and is specific to the IDR +# playbooks + +# Required vars: +#idr_vm_name +#idr_vm_image +#idr_vm_key_name +#idr_vm_flavour + +# Optional, default(omit) +#idr_vm_private_network + +idr_vm_assign_floating_ip: False + +# idr_environment: All VMs will be put into this group +idr_environment: idr + +# Booleans indicating the purpose of this server (multiple may be True) +idr_vm_database: False +idr_vm_omero: False +idr_vm_proxy: False +idr_vm_dockermanager: False +idr_vm_dockerworker: False +idr_vm_bastion: False + +# Default groups depend on the purpose of this server +idr_vm_groups: > + {{ + (idr_vm_database | ternary(idr_vm_default_groups_database, [])) + + (idr_vm_omero | ternary(idr_vm_default_groups_omero, [])) + + (idr_vm_proxy | ternary(idr_vm_default_groups_proxy, [])) + + (idr_vm_dockermanager | ternary(idr_vm_default_groups_dockermanager, [])) + + (idr_vm_dockerworker | ternary(idr_vm_default_groups_dockerworker, [])) + + (idr_vm_bastion | ternary(idr_vm_default_groups_bastion, [])) + }} + +idr_vm_extra_groups: [] + +# Default security groups depend on the purpose of this server +idr_vm_security_groups: > + {{ + ['default'] + + (idr_vm_omero | ternary(['idr-omero-external'], [])) + + (idr_vm_proxy | ternary(['idr-web-external'], [])) + + (idr_vm_bastion | ternary(['idr-bastion-external'], [])) + }} + +idr_vm_default_groups_database: +- database-hosts +- "{{ idr_environment }}-database-hosts" +- "{{ idr_environment }}-hosts" + +idr_vm_default_groups_omero: +- omero-hosts +- "{{ idr_environment }}-omero-hosts" +- "{{ idr_environment }}-hosts" + +idr_vm_default_groups_proxy: +- proxy-hosts +- "{{ idr_environment }}-proxy-hosts" +- "{{ idr_environment }}-hosts" + +idr_vm_default_groups_bastion: +- bastion-hosts +- "{{ idr_environment }}-bastion-hosts" +- "{{ idr_environment }}-hosts" + +idr_vm_default_groups_dockermanager: +- dockermanager-hosts +- "{{ idr_environment }}-dockermanager-hosts" +- "{{ idr_environment }}-docker-hosts" +- "{{ idr_environment }}-hosts" +- "{{ idr_environment }}-data-hosts" + +idr_vm_default_groups_dockerworker: +- dockerworker-hosts +- "{{ idr_environment }}-dockerworker-hosts" +- "{{ idr_environment }}-docker-hosts" +- "{{ idr_environment }}-hosts" diff --git a/ansible/roles/openstack-idr-instance/tasks/main.yml b/ansible/roles/openstack-idr-instance/tasks/main.yml new file mode 100644 index 000000000..5ac758c5f --- /dev/null +++ b/ansible/roles/openstack-idr-instance/tasks/main.yml @@ -0,0 +1,27 @@ +--- +# Playbook for creating OpenStack IDR VMs + +# WARNING: Do not use set_facts in this role, since it'll create a hostvar +# on the host running the openstack client and not the VM created. +# This means multiple invocations of this role (for multiple VMs) will fail +# to work as expected since the hostvar is persistent across tasks. +# See defaults/main.yml for most of the logic. + +- name: idr vm | create VM + os_server: + name: "{{ idr_vm_name }}" + state: present + image: "{{ idr_vm_image }}" + key_name: "{{ idr_vm_keyname }}" + flavor: "{{ idr_vm_flavour }}" + nics: "{{ idr_vm_private_networks | default(omit) }}" + auto_ip: "{{ idr_vm_assign_floating_ip }}" + meta: + hostname: "{{ idr_vm_name }}" + groups: "{{ (idr_vm_groups + idr_vm_extra_groups) | join(',') }}" + security_groups: "{{ idr_vm_security_groups | join(',') }}" + register: vm + +- debug: + msg: "{{ idr_vm_name }} IP private:{{ vm.openstack.private_v4 | default('') }} floating:{{ vm.openstack.public_v4 | default('') }}" + verbosity: 1 diff --git a/ansible/roles/openstack-idr-security-groups/README.md b/ansible/roles/openstack-idr-security-groups/README.md new file mode 100644 index 000000000..ed99e73a2 --- /dev/null +++ b/ansible/roles/openstack-idr-security-groups/README.md @@ -0,0 +1,10 @@ +Openstack IDR Security Groups +============================= + +Create the security groups for the IDR + + +Author Information +------------------ + +ome-devel@lists.openmicroscopy.org.uk diff --git a/ansible/roles/openstack-idr-security-groups/tasks/main.yml b/ansible/roles/openstack-idr-security-groups/tasks/main.yml new file mode 100644 index 000000000..e7c65ef76 --- /dev/null +++ b/ansible/roles/openstack-idr-security-groups/tasks/main.yml @@ -0,0 +1,58 @@ +--- +# IDR security groups + +- name: OMERO external access security group + os_security_group: + description: External access to OMERO servers (managed by Ansible) + name: idr-omero-external + state: present + +- name: OMERO external access security group rules + os_security_group_rule: + direction: ingress + port_range_max: "{{ item }}" + port_range_min: "{{ item }}" + protocol: tcp + remote_ip_prefix: 0.0.0.0/0 + security_group: idr-omero-external + state: present + with_items: + - 4063 + - 4064 + +- name: Web external access security group + os_security_group: + description: External access to web servers (managed by Ansible) + name: idr-web-external + state: present + +- name: Web external access security group rules + os_security_group_rule: + direction: ingress + port_range_max: "{{ item }}" + port_range_min: "{{ item }}" + protocol: tcp + remote_ip_prefix: 0.0.0.0/0 + security_group: idr-web-external + state: present + with_items: + - 80 + - 443 + +- name: Bastion external access security group + os_security_group: + description: External access to bastion servers (managed by Ansible) + name: idr-bastion-external + state: present + +- name: Bastion external access security group rules + os_security_group_rule: + direction: ingress + port_range_max: "{{ item }}" + port_range_min: "{{ item }}" + protocol: tcp + remote_ip_prefix: 0.0.0.0/0 + security_group: idr-bastion-external + state: present + with_items: + - 22 diff --git a/ansible/roles/openstack-volume-storage/tasks/main.yml b/ansible/roles/openstack-volume-storage/tasks/main.yml index 28a6ff021..e212273e0 100644 --- a/ansible/roles/openstack-volume-storage/tasks/main.yml +++ b/ansible/roles/openstack-volume-storage/tasks/main.yml @@ -6,6 +6,10 @@ state: present size: "{{ openstack_volume_size }}" display_name: "{{ openstack_volume_vmname }}-{{ openstack_volume_name }}" + snapshot_id: "{{ openstack_volume_snapshot | default(omit) }}" + # TODO: copying from a volume may be quicker than from a snapshot, but this requires + # https://github.com/ansible/ansible-modules-core/pull/5176 + #volume_src: "{{ openstack_volume_source | default(omit) }}" - name: openstack volume | attach volume to host os_server_volume: