Add metadata service support to controller node

The metadata service is working as the following:
- metadata is being served by nova-api on controller at port 8775.
- VM instance requests metadata by 169.254.169.254 (eg, curl http://169.254.169.254/latest/meta-data)
- metadata request comes to neutron-ns-metadata-proxy on controller in dhcp network name space.
- neutron-ns-metadata-proxy forwards the request to neutron-metadata-agent through a unix domain socket (/var/lib/neutron/metadata_proxy).
- neutron-metadata-agent sends the request to nova-api on port 8775 to be serviced.

To support metadata service, neutron-ns-metadata-proxy is baked into the controller image. Also neutron-metadata-agent startup script
(/etc/init.d/neutron-metadata-agent) and config file (/etc/neutron/metadata_agent.ini) are added to start up metadata agent at system
initialization. dhcp_agent.ini and nova.conf are updated as well.

A README.metadata is added in the Documentation/ directory.

Signed-off-by: Andy Ning <andy.ning@windriver.com>
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>

8 files changed, 179 insertions, 2 deletions

diff --git a/meta-openstack/Documentation/README.metadata b/meta-openstack/Documentation/README.metadata
new file mode 100644
index 0000000..46a7c2a
--- /dev/null
+++ b/meta-openstack/Documentation/README.metadata
@@ -0,0 +1,117 @@
+Summary
+=======
+
+This document is intended to provide an overview of what metadata service is,
+how it works and how it is tested to ensure that metadata service is working correctly.
+
+Metadata Service Introduction
+=============================
+
+OpenStack Compute service uses a special metadata service to enable VM instances to retrieve instance-specific data (metadata). Instances access the metadata service at http://169.254.169.254. The metadata service supports two sets of APIs: an OpenStack metadata API and an EC2-compatible API. Each of the APIs is versioned by date.
+
+To retrieve a list of supported versions for the OpenStack metadata API, make a GET request to http://169.254.169.254/openstack.
+
+For example:
+$ curl http://169.254.169.254/openstack
+2012-08-10
+latest
+
+To list supported versions for the EC2-compatible metadata API, make a GET request to http://169.254.169.254.
+
+For example:
+$ curl http://169.254.169.254
+1.0
+2007-01-19
+2007-03-01
+2007-08-29
+2007-10-10
+2007-12-15
+2008-02-01
+2008-09-01
+2009-04-04
+latest
+
+If cloud-init is supported by the VM image, cloud-init can retrieve metadata from metadata service at instance initialization. 
+
+Metadata Service Implementation
+===============================
+
+Metadata service is provided by nova-api on controller at port 8775. VM instance requests metadata by 169.254.169.254 
+(eg, curl http://169.254.169.254/latest/meta-data). The requests from VM come to neutron-ns-metadata-proxy on controller
+in dhcp network name space, neutron-ns-metadata-proxy forwards the requests to neutron-metadata-agent through a unix domain 
+socket (/var/lib/neutron/metadata_proxy), and neutron-metadata-agent sends the request to nova-api on port 8775 to be serviced.
+
+Test Steps
+==========
+1. build controller and compute image as normal.
+2. setup a cloud with one controller and one compute on real hardware with a flat network.
+   - make sure controller and compute see each other by ping.
+3. on controller:
+   - checking metadata agent is running:
+     # ps -ef | grep neutron-metadata-agent
+   - create a network
+     example: 
+     # neutron net-create --provider:physical_network=ph-eth0 --provider:network_type=flat --router:external=True MY_NET
+   - create a subnet on the network just created
+     example: 
+     # neutron subnet-create MY_NET 128.224.149.0/24 --name MY_SUBNET --no-gateway --host-route destination=0.0.0.0/0,nexthop=128.224.149.1 --allocation-pool start=128.224.149.200,end=128.224.149.210
+   - create an image from cirros 0.3.2 (0.3.0 doesn't work properly due to a bug in it)
+     example: 
+     # glance image-create --name cirros-0.3.2 --is-public true --container-format bare --disk-format qcow2 --file cirros-0.3.2-x86_64-disk.img
+   - boot an instance from cirros-0.3.2 image
+     example: 
+     # nova boot --image cirros-0.3.2 --flavor 1 OpenStack_1 
+   - checking dhcp domain is created
+     # ip netns list
+     example output: 
+qdhcp-229dd93f-a3da-4a21-be22-49c3f3a5dbbd
+
+     # ip netns exec qdhcp-229dd93f-a3da-4a21-be22-49c3f3a5dbbd ip addr
+     example output:
+16: tap5dfe0d76-c5: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
+    link/ether fa:16:3e:c5:d9:65 brd ff:ff:ff:ff:ff:ff
+    inet 128.224.149.201/24 brd 128.224.149.255 scope global tap5dfe0d76-c5
+    inet 169.254.169.254/16 brd 169.254.255.255 scope global tap5dfe0d76-c5
+    inet6 fe80::f816:3eff:fec5:d965/64 scope link 
+       valid_lft forever preferred_lft forever
+17: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
+    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
+    inet 127.0.0.1/8 scope host lo
+    inet6 ::1/128 scope host 
+       valid_lft forever preferred_lft forever
+18: sit0: <NOARP> mtu 1480 qdisc noop state DOWN 
+    link/sit 0.0.0.0 brd 0.0.0.0
+
+     # ip netns exec qdhcp-229dd93f-a3da-4a21-be22-49c3f3a5dbbd netstat -anpe
+     - ensure 0.0.0.0:80 is in there
+     example output:
+Active Internet connections (servers and established)
+Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
+tcp        0      0 128.224.149.201:53      0.0.0.0:*               LISTEN      0          159928      8508/dnsmasq    
+tcp        0      0 169.254.169.254:53      0.0.0.0:*               LISTEN      0          159926      8508/dnsmasq    
+tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      0          164930      8522/python     
+tcp6       0      0 fe80::f816:3eff:fec5:53 :::*                    LISTEN      65534      161016      8508/dnsmasq    
+udp        0      0 128.224.149.201:53      0.0.0.0:*                           0          159927      8508/dnsmasq    
+udp        0      0 169.254.169.254:53      0.0.0.0:*                           0          159925      8508/dnsmasq    
+udp        0      0 0.0.0.0:67              0.0.0.0:*                           0          159918      8508/dnsmasq    
+udp6       0      0 fe80::f816:3eff:fec5:53 :::*                                65534      161015      8508/dnsmasq    
+Active UNIX domain sockets (servers and established)
+Proto RefCnt Flags       Type       State         I-Node   PID/Program name    Path
+unix  2      [ ]         DGRAM                    37016    8508/dnsmasq        
+
+4. on VM instance:
+   - check instance log, ensure the instance gets a dhcp IP address and a static route, as well as the instance-id 
+   - login to the instance, and do the following test
+     $ hostname
+       the host name should be the name specified in "nova boot" when instance is created.
+     $ ifconfig
+       it should have a valid IP on eth0 in the range specified in "neutron subnet-create" when subnet is created.
+     $ route
+       there should be an entry for "169.254.169.254 x.x.x.x 255.255.255.255 eth0"
+     $ curl http://169.254.169.254/latest/meta-data
+       it should return a list of metadata (hostname, instance-id, etc).
+     $ nova reboot <instance>, nova stop <instance>, nova start <instance>, nova rebuild <instance> <image>
+       metadata should be working
+     $ nova boot --user-data <userdata.txt> --image <image> --flavor 1 <instance>
+       curl http://169.254.169.254/latest/user-data should retrieve the userdata.txt
+
diff --git a/meta-openstack/classes/identity.bbclass b/meta-openstack/classes/identity.bbclass
index d1eb276..70191a5 100644
--- a/meta-openstack/classes/identity.bbclass
+++ b/meta-openstack/classes/identity.bbclass
@@ -4,3 +4,4 @@ SERVICE_PASSWORD = "password"
 SERVICE_TENANT_NAME = "service"
 DB_USER = "admin"
 DB_PASSWORD = "admin"
+METADATA_SHARED_SECRET = "password"
diff --git a/meta-openstack/recipes-devtools/python/python-neutron/dhcp_agent.ini b/meta-openstack/recipes-devtools/python/python-neutron/dhcp_agent.ini
index f9b1523..7cc648f 100644
--- a/meta-openstack/recipes-devtools/python/python-neutron/dhcp_agent.ini
+++ b/meta-openstack/recipes-devtools/python/python-neutron/dhcp_agent.ini
@@ -37,6 +37,7 @@ interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
 # be activated when the subnet does not contain any router port. The guest
 # instance must be configured to request host routes via DHCP (Option 121).
 # enable_isolated_metadata = False
+enable_isolated_metadata = True 
 
 # Allows for serving metadata requests coming from a dedicated metadata
 # access network whose cidr is 169.254.169.254/16 (or larger prefix), and
diff --git a/meta-openstack/recipes-devtools/python/python-neutron/metadata_agent.ini b/meta-openstack/recipes-devtools/python/python-neutron/metadata_agent.ini
new file mode 100644
index 0000000..98d958d
--- /dev/null
+++ b/meta-openstack/recipes-devtools/python/python-neutron/metadata_agent.ini
@@ -0,0 +1,29 @@
+[DEFAULT]
+# Show debugging output in log (sets DEBUG log level output)
+# debug = True
+
+# The Neutron user information for accessing the Neutron API.
+auth_url = http://localhost:5000/v2.0
+auth_region = RegionOne
+admin_tenant_name = %SERVICE_TENANT_NAME%
+admin_user = %SERVICE_USER%
+admin_password = %SERVICE_PASSWORD%
+
+# Network service endpoint type to pull from the keystone catalog
+# endpoint_type = adminURL
+
+# IP address used by Nova metadata server
+# nova_metadata_ip = 127.0.0.1
+
+# TCP Port used by Nova metadata server
+# nova_metadata_port = 8775
+
+# When proxying metadata requests, Neutron signs the Instance-ID header with a
+# shared secret to prevent spoofing.  You may select any string for a secret,
+# but it must match here and in the configuration used by the Nova Metadata
+# Server. NOTE: Nova uses a different key: neutron_metadata_proxy_shared_secret
+# metadata_proxy_shared_secret =
+metadata_proxy_shared_secret = %METADATA_SHARED_SECRET%
+
+# Location of Metadata Proxy UNIX domain socket
+# metadata_proxy_socket = $state_path/metadata_proxy
diff --git a/meta-openstack/recipes-devtools/python/python-neutron_git.bb b/meta-openstack/recipes-devtools/python/python-neutron_git.bb
index 4cf4751..2a581ca 100644
--- a/meta-openstack/recipes-devtools/python/python-neutron_git.bb
+++ b/meta-openstack/recipes-devtools/python/python-neutron_git.bb
@@ -14,6 +14,7 @@ SRC_URI = "git://github.com/openstack/${SRCNAME}.git;branch=stable/havana \
            file://neutron-agent.init \
            file://l3_agent.ini \
            file://dhcp_agent.ini \
+           file://metadata_agent.ini \
            file://neutron-dhcp-agent-netns-cleanup.cron \
           "
 SRCREV="a8417d78c2675a2ceb5dbfcee70948a88d69e747"
@@ -84,6 +85,22 @@ do_install_append() {
         install -m 0755 ${WORKDIR}/neutron-$AGENT.init.sh ${D}${sysconfdir}/init.d/neutron-$AGENT-agent
         install -m 600 ${WORKDIR}/${AGENT}_agent.ini ${NEUTRON_CONF_DIR}/
     fi
+    
+    AGENT=metadata
+    ARGS="--config-file=${sysconfdir}/${SRCNAME}/neutron.conf --config-file=${sysconfdir}/${SRCNAME}/metadata_agent.ini"
+    if ${@base_contains('DISTRO_FEATURES', 'sysvinit', 'true', 'false', d)}; then
+        sed "s:@suffix@:$AGENT:;s:@args@:$ARGS:" < ${WORKDIR}/neutron-agent.init >${WORKDIR}/neutron-$AGENT.init.sh
+        install -m 0755 ${WORKDIR}/neutron-$AGENT.init.sh ${D}${sysconfdir}/init.d/neutron-$AGENT-agent
+        install -m 600 ${WORKDIR}/${AGENT}_agent.ini ${NEUTRON_CONF_DIR}/
+    fi
+    sed -e "s:%SERVICE_TENANT_NAME%:${SERVICE_TENANT_NAME}:g" \
+        -i ${NEUTRON_CONF_DIR}/metadata_agent.ini
+    sed -e "s:%SERVICE_USER%:${SRCNAME}:g" \
+        -i ${NEUTRON_CONF_DIR}/metadata_agent.ini
+    sed -e "s:%SERVICE_PASSWORD%:${SERVICE_PASSWORD}:g" \
+        -i ${NEUTRON_CONF_DIR}/metadata_agent.ini
+    sed -e "s:%METADATA_SHARED_SECRET%:${METADATA_SHARED_SECRET}:g" \
+        -i ${NEUTRON_CONF_DIR}/metadata_agent.ini
 
     cp run_tests.sh ${NEUTRON_CONF_DIR}
 }
@@ -175,8 +192,9 @@ FILES_${SRCNAME}-l3-agent = "${bindir}/neutron-l3-agent \
     "
 
 FILES_${SRCNAME}-metadata-agent = "${bindir}/neutron-metadata-agent \
+    ${bindir}/neutron-ns-metadata-proxy \
     ${sysconfdir}/${SRCNAME}/metadata_agent.ini \
-    ${sysconfdir}/init.d/metadata_agent \
+    ${sysconfdir}/init.d/neutron-metadata-agent \
     "
 
 FILES_${SRCNAME}-extra-agents = "${bindir}/*"
@@ -226,7 +244,7 @@ RDEPENDS_${SRCNAME}-setup = "postgresql sudo"
 
 RRECOMMENDS_${SRCNAME}-server = "${SRCNAME}-plugin-openvswitch"
 
-INITSCRIPT_PACKAGES = "${SRCNAME}-server ${SRCNAME}-plugin-openvswitch ${SRCNAME}-plugin-linuxbridge ${SRCNAME}-dhcp-agent ${SRCNAME}-l3-agent"
+INITSCRIPT_PACKAGES = "${SRCNAME}-server ${SRCNAME}-plugin-openvswitch ${SRCNAME}-plugin-linuxbridge ${SRCNAME}-dhcp-agent ${SRCNAME}-l3-agent ${SRCNAME}-metadata-agent"
 INITSCRIPT_NAME_${SRCNAME}-server = "neutron-server"
 INITSCRIPT_PARAMS_${SRCNAME}-server = "${OS_DEFAULT_INITSCRIPT_PARAMS}"
 INITSCRIPT_NAME_${SRCNAME}-plugin-openvswitch = "neutron-openvswitch-agent"
@@ -237,3 +255,5 @@ INITSCRIPT_NAME_${SRCNAME}-dhcp-agent = "neutron-dhcp-agent"
 INITSCRIPT_PARAMS_${SRCNAME}-dhcp-agent = "${OS_DEFAULT_INITSCRIPT_PARAMS}"
 INITSCRIPT_NAME_${SRCNAME}-l3-agent = "neutron-l3-agent"
 INITSCRIPT_PARAMS_${SRCNAME}-l3-agent = "${OS_DEFAULT_INITSCRIPT_PARAMS}"
+INITSCRIPT_NAME_${SRCNAME}-metadata-agent = "neutron-metadata-agent"
+INITSCRIPT_PARAMS_${SRCNAME}-metadata-agent = "${OS_DEFAULT_INITSCRIPT_PARAMS}"
diff --git a/meta-openstack/recipes-devtools/python/python-nova/nova.conf b/meta-openstack/recipes-devtools/python/python-nova/nova.conf
index d763a8b..fadddca 100644
--- a/meta-openstack/recipes-devtools/python/python-nova/nova.conf
+++ b/meta-openstack/recipes-devtools/python/python-nova/nova.conf
@@ -69,6 +69,12 @@ libvirt_images_rbd_ceph_conf=/etc/ceph/ceph.conf
 rbd_user=cinder-volume
 #rbd_secret_uuid=
 
+# nova-api configuration for metadata
+# neutron_metadata_proxy_shared_secret must match
+# metadata_proxy_shared_secret in metadata_agent.ini
+service_neutron_metadata_proxy = true
+neutron_metadata_proxy_shared_secret=%METADATA_SHARED_SECRET%
+
 [spice]
 agent_enabled=True
 enabled=True
diff --git a/meta-openstack/recipes-devtools/python/python-nova_git.bb b/meta-openstack/recipes-devtools/python/python-nova_git.bb
index a1c7a53..9405123 100644
--- a/meta-openstack/recipes-devtools/python/python-nova_git.bb
+++ b/meta-openstack/recipes-devtools/python/python-nova_git.bb
@@ -83,6 +83,8 @@ do_install_append() {
     sed -e "s:%DB_USER%:${DB_USER}:g" -i ${NOVA_CONF_DIR}/nova.conf
     sed -e "s:%DB_PASSWORD%:${DB_PASSWORD}:g" -i ${NOVA_CONF_DIR}/nova.conf
 
+    sed -e "s:%METADATA_SHARED_SECRET%:${METADATA_SHARED_SECRET}:g" -i ${NOVA_CONF_DIR}/nova.conf
+
     sed -e "s:%CONTROLLER_IP%:${CONTROLLER_IP}:g" -i ${NOVA_CONF_DIR}/nova.conf
     sed -e "s:%CONTROLLER_HOST%:${CONTROLLER_HOST}:g" -i ${NOVA_CONF_DIR}/nova.conf
 
diff --git a/meta-openstack/recipes-extended/packagegroups/packagegroup-cloud-network.bb b/meta-openstack/recipes-extended/packagegroups/packagegroup-cloud-network.bb
index 5240aee..6134124 100644
--- a/meta-openstack/recipes-extended/packagegroups/packagegroup-cloud-network.bb
+++ b/meta-openstack/recipes-extended/packagegroups/packagegroup-cloud-network.bb
@@ -10,6 +10,7 @@ RDEPENDS_${PN} = " \
     neutron-plugin-openvswitch \
     neutron-dhcp-agent \
     neutron-l3-agent \
+    neutron-metadata-agent \
     openvswitch-switch \
     dhcp-server \
     dhcp-client \