Append bash history in real-time and change output to show date and time of the command execution

Bash history can be very useful to re-trace steps performed on the system, but one of the issues is that commands are not logged in real-time but after ending a session. Also history output does not include date and time which is really useful for debugging.

In order to solve this you will need to append 3 lines to your ~/.bashrc file:

shopt -s histappend
PROMPT_COMMAND="history -a;$PROMPT_COMMAND"
export HISTTIMEFORMAT='%F %T '

In order for this to become active you need either to log out and log back in our execute:

source ~/.bashrc

After this all your commands will be logged in real-time and output of history command will have date & time information in following format:

YYYY-MM-DD HH:MM:SS history

Run Waydroid in fullscreen on Linux Mint with Gnome

Waydroid looked like excellent option to run Android apps on my Lenovo X240 running Linux Mint.

Because Waydroid is reliant on Wayland you need to use either KDE, Gnome or Sway Desktop Environment.

I am running Gnome and Waydroid works perfectly, but I wanted it to run in fullscreen. Unfortunately Waydroid key mapping is conflicting with Gnome F11 key to go fullscreen, so first thing you need to do is to re-map Gnome fullscreen key to something like Shift+F11 which can easily be done with gnome tweaks.

But once you get your Android app to run “fullscreen” you will notice that actual size of the drawn surface is still the one window started with:

erol@x240mint:~$ sudo waydroid shell wm size 
Physical size: 1366x697

I wanted it shown in 1368×768 which is my native display resolution. To do this open terminal while Waydroid is running and do the following:

erol@x240mint:~$ waydroid prop set persist.waydroid.width 1366
erol@x240mint:~$ waydroid prop set persist.waydroid.height 768
erol@x240mint:~$ waydroid session stop

Now next time you run your Android app it is going to be rendered in native resolution.

erol@x240mint:~$ sudo waydroid shell wm size 
Physical size: 1366x768

Openstack orchestration

Colection of Openstack resources that work together as three-tier application is called a stack. Stack uses openstack resources like instances, networks, volumes and objects, and other elements. The application runs on a stack.

Stack can be set-up via CLI or Horizon GUI which will send API request to various Openstack services (Keystone, Nova, Neutron, Cinder, Swift, Glance …).

Heat is the Openstack orchestration service that sends these API requests automatically. Stack description is taken by heat and interpreted to API requests. Heat accepts AWS Cloudformation templates written in JSON, but heat has it’s own native format called Heat Orchestration Template (HOT) based on YAML.

Create simple stack with heat orchestration template:
heat_template_version: newton

resources:
   myserver:
      type: OS::Nova::Server
      properties:
         config_drive: true
         key_name: 'erol-keypair'
         image: 'cirros-image'
         flavor: 1
         networks:
           - network: 'private_network'
Validate
openstack orchestration template validate --template simple-stack.yaml
If everything is ok, there will be no errors shown:
Description: No description
Parameters: {}
Create stack:
openstack stack create -t simple-stack.yaml simple-stack
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| id                  | c0e028fd-6d10-4bd5-b3a0-5b0509ded846 |
| stack_name          | simple-stack                         |
| description         | No description                       |
| creation_time       | 2020-04-30T13:06:57Z                 |
| updated_time        | None                                 |
| stack_status        | CREATE_IN_PROGRESS                   |
| stack_status_reason | Stack CREATE started                 |
+---------------------+--------------------------------------+
List stacks to check if it has finished creating:
openstack stack list
+--------------------------------------+--------------+-----------------+----------------------+--------------+
| ID                                   | Stack Name   | Stack Status    | Creation Time        | Updated Time |
+--------------------------------------+--------------+-----------------+----------------------+--------------+
| c0e028fd-6d10-4bd5-b3a0-5b0509ded846 | simple-stack | CREATE_COMPLETE | 2020-04-30T13:06:57Z | None         |
+--------------------------------------+--------------+-----------------+----------------------+--------------+
List all resources belonging to the stack:
openstack stack resource list simple-stack
+---------------+--------------------------------------+------------------+-----------------+----------------------+
| resource_name | physical_resource_id                 | resource_type    | resource_status | updated_time         |
+---------------+--------------------------------------+------------------+-----------------+----------------------+
| myserver      | d6d86e6d-d904-4f90-90af-8efebf45aea2 | OS::Nova::Server | CREATE_COMPLETE | 2020-04-30T13:06:57Z |
+---------------+--------------------------------------+------------------+-----------------+----------------------+
Change server name in simple-stack.yaml:
myserver;
to:
server
Update the stack:
openstack stack update -t simple-stack.yaml.1 simple-stack*
* Changing the resource name of the server will delete old instance and create new instance with new resource_name.
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| id                  | c0e028fd-6d10-4bd5-b3a0-5b0509ded846 |
| stack_name          | simple-stack                         |
| description         | No description                       |
| creation_time       | 2020-04-30T13:06:57Z                 |
| updated_time        | 2020-04-30T13:14:59Z                 |
| stack_status        | UPDATE_IN_PROGRESS                   |
| stack_status_reason | Stack UPDATE started                 |
+---------------------+--------------------------------------+
Check if stack was updated successfuly:
openstack stack list
+--------------------------------------+--------------+-----------------+----------------------+----------------------+
| ID                                   | Stack Name   | Stack Status    | Creation Time        | Updated Time         |
+--------------------------------------+--------------+-----------------+----------------------+----------------------+
| c0e028fd-6d10-4bd5-b3a0-5b0509ded846 | simple-stack | UPDATE_COMPLETE | 2020-04-30T13:06:57Z | 2020-04-30T13:14:59Z |
+--------------------------------------+--------------+-----------------+----------------------+----------------------+

Check if resource list was updated:
openstack stack resource list simple-stack
+---------------+--------------------------------------+------------------+-----------------+----------------------+
| resource_name | physical_resource_id                 | resource_type    | resource_status | updated_time         |
+---------------+--------------------------------------+------------------+-----------------+----------------------+
| server        | 62d20902-ee6c-40a0-bdc6-e38ed81fdf7c | OS::Nova::Server | CREATE_COMPLETE | 2020-04-30T13:14:59Z |
+---------------+--------------------------------------+------------------+-----------------+----------------------+
Change network in simple-stack.yaml.1:
network: 'private_network'
to:
network: 'external_network'
openstack stack update -t simple-stack.yaml.2 simple-stack
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| id                  | c0e028fd-6d10-4bd5-b3a0-5b0509ded846 |
| stack_name          | simple-stack                         |
| description         | No description                       |
| creation_time       | 2020-04-30T13:06:57Z                 |
| updated_time        | 2020-04-30T13:39:28Z                 |
| stack_status        | UPDATE_IN_PROGRESS                   |
| stack_status_reason | Stack UPDATE started                 |
+---------------------+--------------------------------------+
openstack stack resource list simple-stack
+---------------+--------------------------------------+------------------+-----------------+----------------------+
| resource_name | physical_resource_id                 | resource_type    | resource_status | updated_time         |
+---------------+--------------------------------------+------------------+-----------------+----------------------+
| server        | 62d20902-ee6c-40a0-bdc6-e38ed81fdf7c | OS::Nova::Server | UPDATE_COMPLETE | 2020-04-30T13:39:31Z |
+---------------+--------------------------------------+------------------+-----------------+----------------------+

Retrieve all parameter functions supported by Openstack version:

openstack orchestration template version list
+--------------------------------------+------+
| version                              | type |
+--------------------------------------+------+
| AWSTemplateFormatVersion.2010-09-09  | cfn  |
| HeatTemplateFormatVersion.2012-12-12 | cfn  |
| heat_template_version.2013-05-23     | hot  |
| heat_template_version.2014-10-16     | hot  |
| heat_template_version.2015-04-30     | hot  |
| heat_template_version.2015-10-15     | hot  |
| heat_template_version.2016-04-08     | hot  |
| heat_template_version.2016-10-14     | hot  |
+--------------------------------------+------+
openstack orchestration template function list heat_template_version.2016-10-14
+-----------------+-------------------------------------------------------------------------+
| functions       | description                                                             |
+-----------------+-------------------------------------------------------------------------+
| list_join       | A function for joining one or more lists of strings.                    |
| if              | A function to return corresponding value based on condition evaluation. |
| yaql            | A function for executing a yaql expression.                             |
| digest          | A function for performing digest operations.                            |
| get_attr        | A function for resolving resource attributes.                           |
| repeat          | A function for iterating over a list of items.                          |
| resource_facade | A function for retrieving data in a parent provider template.           |
| map_replace     | A function for performing substitutions on maps.                        |
| str_replace     | A function for performing string substitutions.                         |
| get_resource    | A function for resolving resource references.                           |
| map_merge       | A function for merging maps.                                            |
| str_split       | A function for splitting delimited strings into a list.                 |
| get_param       | A function for resolving parameter references.                          |
| get_file        | A function for including a file inline.                                 |
+-----------------+-------------------------------------------------------------------------+

Parameters can be provided via command line with
--parameter imgname=cirros-image keyname=erol-keypair
Or combine several parameters under one option:
--parameter "imgname=cirros-image; keyname=erol-keypair"

If you have a lot of parameters it is probably better to put them in a file again in YAML format, with single parameters key and parameter key value pairs under that key. For example create myparameters.yaml file with following contents:

parameters:
   keyname: erol-keypair
   imgname: cirros-image

You can now use –enviroment option to include the parameter keypairs from your file to the heat orchestration template.

Contents of parameters-stack.yaml:

heat_template_version: newton

parameters:
  keyname:
    type: string
    default: erol-keypair
    description: .....
  imgname:
    type: string

resources:
   server:
      type: OS::Nova::Server
      properties:
         config_drive: true
         key_name: { get_param: keyname }
         image: { get_param: imgname }
         flavor: 1
         networks:
           - network: 'private_network'

You can create your new stack with new template and parameters file:

openstack stack create myfirststack --template parameters-stack.yaml -e myparameters.yaml
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| id                  | ccc6b0a3-f690-479b-87fd-d36b1077f53c |
| stack_name          | myfirststack                         |
| description         | No description                       |
| creation_time       | 2020-04-30T14:08:58Z                 |
| updated_time        | None                                 |
| stack_status        | CREATE_IN_PROGRESS                   |
| stack_status_reason | Stack CREATE started                 |
+---------------------+--------------------------------------+
[root@packstack ~(keystone_admin)]# openstack stack list
+--------------------------------------+--------------+-----------------+----------------------+----------------------+
| ID                                   | Stack Name   | Stack Status    | Creation Time        | Updated Time         |
+--------------------------------------+--------------+-----------------+----------------------+----------------------+
| ccc6b0a3-f690-479b-87fd-d36b1077f53c | myfirststack | CREATE_COMPLETE | 2020-04-30T14:08:58Z | None                 |
| c0e028fd-6d10-4bd5-b3a0-5b0509ded846 | simple-stack | UPDATE_COMPLETE | 2020-04-30T13:06:57Z | 2020-04-30T13:39:28Z |
+--------------------------------------+--------------+-----------------+----------------------+----------------------+
openstack stack resource list myfirststack
+---------------+--------------------------------------+------------------+-----------------+----------------------+
| resource_name | physical_resource_id                 | resource_type    | resource_status | updated_time         |
+---------------+--------------------------------------+------------------+-----------------+----------------------+
| server        | e7378fa5-ed43-48eb-b00e-17952d4aad97 | OS::Nova::Server | CREATE_COMPLETE | 2020-04-30T14:08:59Z |
+---------------+--------------------------------------+------------------+-----------------+----------------------+

Create a modified version of your HOT file called linking-stack.yaml:

heat_template_version: newton

parameters:
  keyname:
    type: string
    default: erol-keypair
    description: .....
  imgname:
    type: string

resources:
   fip:
     type: OS::Nova::FloatingIP
     properties:
       pool: external_network
   fip_assoc:
     type: OS::Nova::FloatingIPAssociation
     properties:
       server_id: { get_resource: server }
       floating_ip: { get_resource: fip }

   server:
      type: OS::Nova::Server
      properties:
         config_drive: true
         key_name: { get_param: keyname }
         image: { get_param: imgname }
         flavor: 1
         networks:
           - network: 'private_network'
outputs:
   fip_address:
      description: Floating IP address value
      value:
         get_attr: [ fip, ip ]

Create a modified version of your HOT file called linking-stack.yaml:

openstack stack create linkstack --template linking-stack.yaml -e myparameters.yaml
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| id                  | eb146b58-a00e-4d5c-897f-243d6c55c4ea |
| stack_name          | linkstack                            |
| description         | No description                       |
| creation_time       | 2020-04-30T14:22:51Z                 |
| updated_time        | None                                 |
| stack_status        | CREATE_IN_PROGRESS                   |
| stack_status_reason | Stack CREATE started                 |
+---------------------+--------------------------------------+
openstack stack list
+--------------------------------------+--------------+-----------------+----------------------+----------------------+
| ID                                   | Stack Name   | Stack Status    | Creation Time        | Updated Time         |
+--------------------------------------+--------------+-----------------+----------------------+----------------------+
| eb146b58-a00e-4d5c-897f-243d6c55c4ea | linkstack    | CREATE_COMPLETE | 2020-04-30T14:22:51Z | None                 |
| ccc6b0a3-f690-479b-87fd-d36b1077f53c | myfirststack | CREATE_COMPLETE | 2020-04-30T14:08:58Z | None                 |
| c0e028fd-6d10-4bd5-b3a0-5b0509ded846 | simple-stack | UPDATE_COMPLETE | 2020-04-30T13:06:57Z | 2020-04-30T13:39:28Z |
+--------------------------------------+--------------+-----------------+----------------------+----------------------+
openstack stack event list linkstack
2020-04-30 14:22:52Z [linkstack]: CREATE_IN_PROGRESS  Stack CREATE started
2020-04-30 14:22:54Z [linkstack.fip]: CREATE_IN_PROGRESS  state changed
2020-04-30 14:22:58Z [linkstack.link-server]: CREATE_IN_PROGRESS  state changed
2020-04-30 14:22:59Z [linkstack.fip]: CREATE_COMPLETE  state changed
2020-04-30 14:23:19Z [linkstack.link-server]: CREATE_COMPLETE  state changed
2020-04-30 14:23:24Z [linkstack.fip_assoc]: CREATE_IN_PROGRESS  state changed
2020-04-30 14:23:36Z [linkstack.fip_assoc]: CREATE_COMPLETE  state changed
2020-04-30 14:23:37Z [linkstack]: CREATE_COMPLETE  Stack CREATE completed successfully
openstack stack show linkstack
+-----------------------+----------------------------------------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                                                      |
+-----------------------+----------------------------------------------------------------------------------------------------------------------------+
| id                    | eb146b58-a00e-4d5c-897f-243d6c55c4ea                                                                                       |
| stack_name            | linkstack                                                                                                                  |
| description           | No description                                                                                                             |
| creation_time         | 2020-04-30T14:22:51Z                                                                                                       |
| updated_time          | None                                                                                                                       |
| stack_status          | CREATE_COMPLETE                                                                                                            |
| stack_status_reason   | Stack CREATE completed successfully                                                                                        |
| parameters            | OS::project_id: 7f64b208cc8b4a5988317789af7f827f                                                                           |
|                       | OS::stack_id: eb146b58-a00e-4d5c-897f-243d6c55c4ea                                                                         |
|                       | OS::stack_name: linkstack                                                                                                  |
|                       | imgname: cirros-image                                                                                                      |
|                       | keyname: erol-keypair                                                                                                      |
|                       |                                                                                                                            |
| outputs               | - description: Floating IP address value                                                                                   |
|                       |   output_key: fip_address                                                                                                  |
|                       |   output_value: 172.30.152.22                                                                                              |
|                       |                                                                                                                            |
| links                 | - href: http://172.30.152.4:8004/v1/7f64b208cc8b4a5988317789af7f827f/stacks/linkstack/eb146b58-a00e-4d5c-897f-243d6c55c4ea |
|                       |   rel: self                                                                                                                |
|                       |                                                                                                                            |
| parent                | None                                                                                                                       |
| disable_rollback      | True                                                                                                                       |
| deletion_time         | None                                                                                                                       |
| stack_user_project_id | 0d97b86e82234c028d9c5f785aeeb3c9                                                                                           |
| capabilities          | []                                                                                                                         |
| notification_topics   | []                                                                                                                         |
| stack_owner           | None                                                                                                                       |
| timeout_mins          | None                                                                                                                       |
| tags                  | null                                                                                                                       |
|                       | ...                                                                                                                        |
|                       |                                                                                                                            |
+-----------------------+----------------------------------------------------------------------------------------------------------------------------+
openstack stack output list linkstack
+-------------+---------------------------+
| output_key  | description               |
+-------------+---------------------------+
| fip_address | Floating IP address value |
+-------------+---------------------------+
openstack stack output show linkstack fip_address
+--------------+---------------------------+
| Field        | Value                     |
+--------------+---------------------------+
| description  | Floating IP address value |
| output_key   | fip_address               |
| output_value | 172.30.152.22             |
+--------------+---------------------------+
openstack stack resource list linkstack
+---------------+--------------------------------------+---------------------------------+-----------------+----------------------+
| resource_name | physical_resource_id                 | resource_type                   | resource_status | updated_time         |
+---------------+--------------------------------------+---------------------------------+-----------------+----------------------+
| fip           | 732de760-891d-4684-9853-874527c8dfdf | OS::Nova::FloatingIP            | CREATE_COMPLETE | 2020-04-30T14:22:53Z |
| fip_assoc     | 4                                    | OS::Nova::FloatingIPAssociation | CREATE_COMPLETE | 2020-04-30T14:22:53Z |
| link-server   | 4174ee39-6dce-4683-8d7c-8e9a00dc9907 | OS::Nova::Server                | CREATE_COMPLETE | 2020-04-30T14:22:53Z |
+---------------+--------------------------------------+---------------------------------+-----------------+----------------------+

Openstack telemetry: Ceilometer, Gnocchi and Aodh

In Openstack Ceilometer is the component that gathers data from the cloud and pre-processes it. It distinguishes between samples (CPU time) and events (creation of an instance). Resources, Meters and Samples are fundamental concepts in Ceilometer.

Samples are retrieved at regular intervals and if Ceilometar fails to get the sample it can be estimated by interpolation. Events are retrieved as they happen and cannot be estimated.

Ceilometer sends events to the a storage service, while samples are sent to a service named Gnocchi, which is optimized to handle large amount of time-series data.

Aodh gets measures from Gnocchi, checks whether certain conditions are met and triggers actions. This is the foundation for application auto-scaling.

Other uses for Gnocchi data are monitoring the health of the cloud and billing.

Ceilometer has three ways retrieving samples and events:

  • Services may voluntarily provide them by sending Ceilometer notification via Openstack’s messaging system. This is preferred way since it is based on internal knowledge that the service has about it’s resources and it is fast without much overhead and stress on the systems.
  • Ceilometer actively retrieves data via APIs which is a costly method for billing and alarming.
  • Ceilometer can get data by accessing sub-components of services such as the hypervisor that run the instances.

Second and third method are referred also as methods where Ceilometer “polls” the samples.

More details on Openstack telemetry can be found on this link:
https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.html

While Ceilometer has resources, meters and samples Gnocchi has resources, metrics and measures. Gnocchi resource corresponds to Ceilometer resource. Metric is roughly equivalent to a meter in Ceilometer. Gnocchi does not store every metric value it receives from Ceilomter, but rather it combines values and stores the results at regular intervals according to Archive policy.

Listing gnocchi resources, metrics, measures:
Resources:
gnocchi resource list
gnocchi resource show UUID
Metrics:
gnocchi metric list*
gnocchi metric show cpu --resource UUID
Measures:
gnocchi measures show cpu --resource UUID --start YYYY-MM-DDTHH:MM:SS+00:00
* Output will be empty for non-admin users.
Listing resources, metrics, measures with openstack client:
Resources:
openstack metric resource list
openstack metric resource show UUID
Metrics:
openstack metric metric list*
openstack metric metric show cpu --resource UUID
Measures:
openstack metric measures show cpu --resource UUID --start YYYY-MM-DDTHH:MM:SS+00:00
* Output will be empty for non-admin users.
Aggregation:
Server grouping:
openstack server create --property metering.server_group=Mail*
Metrics aggregation:
gnocchi measures aggregation --query server_group=Mail --resource-type=instance --aggregation mean -m cpu_util
* For gnocchi all servers with ‘–property metering.server_group=Mail’ can be considered tagged.
Listing Ceilometer Events:
Event types are defined in a YAML type:
/etc/ceilometer/event_definitions.yaml
List event types, events and event details:
ceilometer event-type-list
ceilometer event-list
ceilometer event-show EVENT_ID
* For gnocchi all servers with ‘–property metering.server_group=Mail’ can be considered tagged.
** There is no option in horizon GUI to view events or statictics, but gnocchi visualisation can be provided by Grafana.
Example generating CPU and disk load and showing gnocchi measures:
Create two instances:
openstack server create --image cirros-image --flavor 1 --nic net-id=... --user-data cpu.sh cpu-user*
openstack server create --image cirros-image --flavor 1 --nic net-id=... --user-data disk.sh disk-user*
Let the instances finish creating and leave them running for a while:
openstack server list
Show cpu usage measures from cpu-user server:
gnocchi measures show cpu.delta --resource-id SERVER_UUID
795 gnocchi measures show cpu_util --resource-id SERVER_UUID
Show disk usage measures from disk-user server:
gnocchi measures show disk.read.requests --resource-id SERVER_UUID
gnocchi measures show disk.read.requests.rate --resource-id SERVER_UUID
* Files cpu.sh and disk.sh are your bash scripts to generate CPU load and disk load.
** Don’t forget to stop cpu-server and disk-server after you have finished, since they will continue to generate CPU and disk load.
Alarms:
An alarm has:
Type
Condition: depends on type
Evaluation window
State: OK/Alarm/Insufficient Data
Actions for state transitions
Condition example:
mean cpu_util > 60
all resources tagged server_group=Mail
Single Resource Threshold Alarm:
openstack alarm create --name cpuhigh \
--type gnocchi_resources_threshold \
--aggregation-method mean --metric cpu_util \
--comparison-operator gt --threshold 30 \
--resource-type instance \
--resource-id INSTANCE_UUID \
--granularity 60 --evaluation-periods 2 \
--alarm-action http://127.0.0.1:1234 \
--ok-action http://127.0.0.1:1234
Alarm based on resource aggregates:
openstack alarm create --name cpuhigh \
--type gnocchi_aggregation_by_resources_threshold \
--aggregation-method mean --metric cpu_util \
--comparison-operator gt --threshold 30 \
--resource-type instance \
--query '{ "=": { "server_group" : "Mail" }}' \
--granularity 60 --evaluation-periods 2 \
--alarm-action http://127.0.0.1:1234 \
--ok-action http://127.0.0.1:1234
Alarm commands:
openstack alarm list
openstack alarm show ALARM_ID
openstack alarm-history show ALARM_ID
openstack alarm state get ALARM_ID
openstack alarm update ALARM_ID ...