erol.name

29. 04. 202030. 04. 2020

Openstack storage cookbook list of useful commands with examples

Here is the list of openstack storage commands and examples that I collected and found useful:

Openstack commands: Cinder commands:

Volume opearations: Cinder operations:

openstack volume create --size 1 [NAME] openstack volume create --size 1 --image [IMAGE] [NAME] openstack volume create --size 1 --source [VOLUME] [NAME] openstack volume create --size 1 --snapshot [SNAPSHOT] [NAME] openstack volume list openstack volume show [VOLUME] openstack volume set [VOLUME] --size 2 openstack volume set [VOLUME] --name [NEW_NAME] openstack volume set [VOLUME] --property [KEY]=[VALUE] openstack volume delete [VOLUME]* cinder create 1 cinder create 1 --image [IMAGE] cinder create 1 --source-volid [VOLUME_ID] cinder create 1 --snapshot-id [SNAPSHOT_ID] cinder list cinder show [VOLUME] cinder extend [VOLUME] 2 cinder rename [VOLUME] [NEW_NAME] cinder metadata [VOLUME] set [KEY]=[VALUE] cinder delete [VOLUME]*

Openstack commands:	Cinder commands:
Volume opearations:	Cinder operations:
`openstack volume create --size 1 [NAME] openstack volume create --size 1 --image [IMAGE] [NAME] openstack volume create --size 1 --source [VOLUME] [NAME] openstack volume create --size 1 --snapshot [SNAPSHOT] [NAME] openstack volume list openstack volume show [VOLUME] openstack volume set [VOLUME] --size 2 openstack volume set [VOLUME] --name [NEW_NAME] openstack volume set [VOLUME] --property [KEY]=[VALUE] openstack volume delete [VOLUME]*`	`cinder create 1 cinder create 1 --image [IMAGE] cinder create 1 --source-volid [VOLUME_ID] cinder create 1 --snapshot-id [SNAPSHOT_ID] cinder list cinder show [VOLUME] cinder extend [VOLUME] 2 cinder rename [VOLUME] [NEW_NAME] cinder metadata [VOLUME] set [KEY]=[VALUE] cinder delete [VOLUME]*`

Volumes can not be smaller than 1GB, that is why –size 1 is used.
* It is not posible to undo a delete operation
** Adding –type [VOLUME_TYPE] to create options

Volume types:
Volume operations:
`openstack volume type create thin --property volume_backend_name=lvm --property lvm:provisioning=thin` `openstack volume types openstack volume type show [NAME] openstack volume create --type thin --size 1 [thinvol] openstack volume list --long*`

* In reality lvm backend does not have parameter provisioning, actually it has no parameters at all, but this “fake” property “thin” is used to show how types are used
** In order to see parameter type use the “–long” option

Openstack commands:	Nova commands:
Attaching and detaching volumes:
`openstack server add volume SERVER_REF VOLUME_REF openstack server add volume --device /dev/vdc* SERVER_REF VOLUME_REF openstack server remove volume SERVER_REF VOLUME_REF`	`nova volume-attach SERVER_REF VOLUME_ID [DEVICE] nova volume-dettach SERVER_REF VOLUME_ID`
Boot from a volume CLI commands:
`openstack volume create --image IMAGE_REF mybootvol openstack volume create --image SNAPSHOT_REF mybootvol openstack server create --volume mybootvol -- flavor ... --nic ... myinstance`	`nova boot myinstance --nic .. --flavor .. --block-device source=volume,id=VOLUME_ID,dest=volume, size=SIZE,bootindex=0 nova boot myinstance --nic .. --flavor .. --block-device source=image,id=IMAGE_ID,dest=volume, size=SIZE,bootindex=0 nova boot myinstance --nic .. --flavor .. --block-device source=snapshot,id=SNAPSHOT_ID,dest=volume, size=SIZE,bootindex=0`

When instances are launched from images, instance’s internal disk is created from image and the instance boot from it’s that disk. Such internal disk is called ephemeral storage. It dissapears when instance is deleted.
* Libvirt (QEMU and KVM) ignore ‘–device’ parameter and you are stuck with whatever device filename Nova assigns

Openstack commands:	Cinder commands:
Create snapshot:
`openstack snapshot create --name myvol-snap4 myvol`*	`cinder snapshot-create --name myvol-snap4 myvol`*
Create volume from snapshot:
`openstack volume create --snapshot myvol-snap4 myvol-lastweek`
Backup create:	Backup restore:
`openstack volume backup create --name mybck myvol openstack volume backup create --name mybck --snapshot SNAP myvol *`	`openstack volume backup restore mybck myvol2`

*This command will fail on attached volumes, so ‘–force’ parameter must be specified.
** To backup attached volume add parameter ‘–force’ , or ‘–incremental’ which stores the difference between current volume and previous backup. Base for incremental backup is the backup with the most recent timestamp.
*** Cinder client has ‘backup-create’ and ‘backup-restore’ commands.
**** Other openstack commands include ‘delete/list/show/set’ to delete backup, list backups, show backup details and set backup properties respectively.

Recover deleted files from a snapshot:
Identify volume for backup:
`openstack volume list --long`
Indentify network where to launch new server:
`openstack network list`
Use clone of the image to launch an instance:
`openstack server create --volume myclonevol --nic net-id=... --flavor 1 --keyname mykey volserver`
To connect to this image security group and floating ip must be assigned:
`openstack server add security group volserver ssh openstack server add floating ip volserver 172.24.4.230 ssh -i mykey.pem username@ip_address`
Create some files to be recovered:
`cp /etc/passwd /home/username/file1 cp /etc/fstab /home/username/file2`
Create a snapshot of a still attached volume:
`openstack snapshot create --name myclonesnap myclonevol --force`
Remove a file:
`rm /home/username/file1 exit`
Since snapshot can not be attached to instance create volume first:
`openstack volume create --snapshot myclonesnap --size 1 tempvol`
Attach the new volume
`openstack server add volume volserver tempvol`
Log back to the server:
`ssh -i mykey.pem username@ip_address`
List block storage devices:
`lsblk`
Mount the new attached volume:
`mount /dev/vdb1 /mnt/temp`
List the same directory from backup volume, both files should be present:
`ls -la /mnt/temp/home/username/`
Copy the backup file from temporary mount to previous location:
`cp /mnt/temp/home/username/file1 /home/username/`
Unmount backup volume:
`umount /mnt/temp`
Remove and delete redundant copy of the snapshot data:
`openstack server remove ovlume volserver tempvol openstack volume delete tempvol`

Backup up and restoring volumes:
Identify the server:
`openstack server list`
Show volume details:
`openstack volume show myclonevol`
Create backup of this attached volume:
`openstack volume backup create --name myclonevol.backup.$date +%y%m%d) --force`
Check the backup progress*:
`openstack volume backup list`
Create another file on the instance:
`ssh -i mykey.pem username@ip_address cp /etc/group /home/username/file3`
Create an incremental backup:
`openstack volume backup create --name myclonevol.backup.$date +%y%m%d)-1 --force --incremental`
Check the incremental backup progress:
`openstack volume backup list`
Inspect the backups:
`openstack volume backup show myclonevol.backup.YYMMDD` `openstack volume backup show myclonevol.backup.YYMMDD-1*`
Add another file to the instance:
`ssh -i mykey.pem username@ip_address cp /etc/shadow/home/username/file4`
Create another incremental backup:
`openstack volume backup create --name myclonevol.backup.$date +%y%m%d)-2 --force --incremental`
Check the incremental backup progress:
`openstack volume backup list`
Inspect the incremental backups::
`openstack volume backup show myclonevol.backup.YYMMDD-1****`
Simulate failure by removing files from this instance*****:
`rm -rf /` `exit`
Identify backup to restore the instance from:
`openstack volume backup list`
Restore most recent backup to an empty volume******:
`openstack volume create --size 1 myclonevol2` `openstack volume backup restore myclonevol.backup.YYMMDD-2 myclonevol2`
Launch an instance from the restored volume:
`openstack server create --volume myclonevol2 --nic net-id=... --flavor 1 --key-name mykey volserver-restored`
Add a security group and floating IP address and login:
`openstack server add security group volserver-restored ssh openstack server add floating ip volserver 172.24.4.233 ssh -i mykey.pem username@172.24.4.233`
Check that all files are restored:
`ls -la /`

* Backup process takes some time, and while working it is shown as ‘creating’ when completed it will show ‘available’.
** Initial backup should have ‘has_dependant_backups’ set to ‘True’ and ‘is_incremental’ set to ‘False’.
*** Incremental backup should have ‘is_incremental’ set to ‘True’ and has no dependant backups so ‘has_dependant_backups’ is ‘False’.
**** Incremental backup is always created from the backup with the latest timestamp, which in this case is our previous incremental backup who should have now ‘has_dependant_backups’ set to ‘True’.
***** System is now broken beyond repair. Something that I always wanted to do 🙂
****** To restore to attached volume instance must be shutdown, that is why we are restoring to empty volume.
******* To enable volume backup option in the horizon dashboard ‘/etc/openstack-dashboard/local_settings’ should be edited and ‘OPENSTACK_CINDER_FEATURES = { ‘enable_backup’: False. }’ should be changed to ‘True’.

Openstack commands:	Swift commands:
Creating containers and objects:
`openstack container create myphotos` `openstack object create myphotos moon.jpg`	`swift post myphotos` `swift upload myphotos moon.jpg*`
Access data via URL:
`http://CLOUD-ADDRESS:8080/v1/ACCOUNT/myphotos/moon.jpg`
Object with ‘/’ in the name	Switch change object name:
`openstack object create myphotos localdir/moon.jpg**`	`swift upload myphotos localdir/moon.jpg --object-name=moon.jpg`
Show object details:
`openstack show object myphotos moon.jpg`	`swift stat myphotos moon.jpg -v`
Deleting an object:
`openstack object delete myphotos moon.jpg ***`
List containers and objects:
`openstack container list` `openstack object list myphotos --long`	`swift list --lh` `swift list myphotos --lh`
Downloading an object:
`openstack object save myphotos sun/2020.jpg****`
Downloading objects with wget:
`wget --user demo --password ******** $OBJECT_URL`
Setting Metadata:
`openstack object store account set --property category=astronomy openstack container set --property type=pictures myphotos opestack object set --property location=japan myphotos moon.jpg`	`swift post -m location:japan myphotos moon.jpg`
Deleting Metadata*****:
`openstack object store account unset --property category=astronomy openstack container unset --property type=pictures myphotos opestack object unset --property location=japan myphotos moon.jpg`	`swift post -H "X-Remove-Object-Meta-Location: x"****** myphotos moon.jpg`

* Swift upload command creates container if it does not exist.
** Name of the object created with openstack command can not be changed, while swift client can change the name.
*** Deleted objects can not be undeleted.
**** Will create direcotry ‘sun’ and store ‘2020.jpg’ in it. You could also specify an alternate local filename with ‘–file sun.jpg’ parameter.
*****Setting an empty metadata item also deletes it, but is not documented.
****** String ‘Location’ is actually the attribute you want to remove from object and ‘x’ is to satisfy HTTP syntax and it is ignored.

Access control lists:
Permissions based on PROJECT:USER
`demo:demo :admin :*`
Permissions based on referrer:
`.r:* .r:erol.name`
Set ACLs:
`swift post -r ACL CONTAINER swift post -w ACL CONTAINER`
Clear ACLs:
`swift post -r "" CONTAINER swift post -w "" CONTAINER`
Downloading object with wget using auth-token header:
`wget --header "x-auth-token: TOKEN_UUID*" $OBJECT_URL`
Allow any referrer to access file:
`swift post myvideos --read-acl '.referrer:'*`

* TOKEN_UUID value is ‘Auth Token’ value shown with ‘swift stat -v myphotos moon.jpg”
** In oreder to allow listings of a container add parameter ‘.referrer:*,.rlistings’ instead of just ‘.referrer:*’.

Temporary URLs:
Create TempURL key:
`openstack object store account set --property temp-url-key=abc123` `openstack container set --property temp-url-key=abc123`
Generate TempURL:
`swift tempurl GET 86400 /v1/AUTH_..../myvideos/vid.mp4 abc123`*
Generated URL:
`/v1/AUTH_..../myvideos/vid.mp4?temp_url_sig=fa28...&temp_url_expires=...`
Download TempURL with wget
`wget -O my-temp-titan.mp4 "http://CLOUD_IP:8080/v1/AUTH_..../myvideos/vid.mp4?temp_url_sig=fa28...&temp_url_expires=..."`

* To allow read access GET is used, 86400 is the validity of the temporary URL in seconds.

Large objects:
Upload object and segment it into smaller parts:
`swift upload --segment-size=100M mycontainer bigobject*`
Example:
`swift upload --segment-size=1M big-container myvideo.mp4`
Show object details:
`openstack object list big-container --long openstack object show big-container myvideo.mp4 openstack object list big-container_segments --long openstack object list big-container_segments`
Delete containers:
`openstack container delete big-container swift delete big-container*`

* Any size objects can be segmented, but since limitation on object size is 5GB larger objects than this must be split into segments. Container ‘mycontainer’ in this case is empty it only has metadata. The actual data goes into second container whose name is derived from original container f.e. ‘mycontainer_segments’ where one object per segment is stored.
** Openstack will not delete container while there are segments in it.
*** Swift client will delete the main container ‘big-container’ and all it’s segments, but it will not delete container ‘big-container_segments’.

27. 04. 202030. 04. 2020

Openstack cookbook list of useful commands with examples

I have decided to summarize in one page all useful Openstack commands. This post will be useful to anyone interested in learning Openstack cloud commands or use it as a daily reference.

Openstack commands:	Old equivalents:
General commands:
`openstack image list openstack flavor list openstack network list openstack server list`	`glance image-list nova flavor-list neutron net-list nova list`
Server commands:
`openstack server show [SERVER] openstack server stop [SERVER] openstack server start [SERVER] openstack server delete [SERVER] openstack server image create [IMAGE]`	`nova show [SERVER] nova stop [SERVER] nova start [SERVER] nova delete [SERVER] nova image-create [IMAGE]`
Server console commands:
`openstack console log show openstack console url show`	`nova console-log nova get-vnc-console`
SSH Keypairs*:
`openstack keypair create [KEYPAIR] > key.pem` `openstack keypair show[KEYPAIR] --public-key > key.pub` `openstack keypair show --public-key [KEYPAIR]`	`nova keypair-add [KEYPAIR] > key.pem` `nova keypair-show [KEYPAIR] \| grep Public \| awk -F': ' '{print $2}' > ./.ssh/key.pub` `nova keypair-show [KEYPAIR]`
Metadata**:
Key-value pairs: `openstack server create --property [KEY]=[VALUE]` Script (any language): `openstack server create --user-data [SCRIPT]` Cloud-config file (yaml) `openstack server create --user-data [FILE]` File injection `openstack server create --file [DEST_FILE]=[SRC_FILE]`	`nova boot --meta [KEY]=[VALUE]` `nova boot --user-data [SCRIPT]` `nova boot --user-data [FILE]` `nova boot --file [DEST_FILE]=[SRC_FILE]`

* Don’t forget to chmod 600 key.pem in order to protect it and make it usable by SSH command
** There can only be 1 user-data option, but many property and file options .
Metadata is retrieved by process cloud-int in cloud images .
Server requests with curl http://169.254.169.254 –> Metadata service (nova) which replies with
data in JSON format <– (Restful API).

Openstack network related commands:
Security group:
`openstack security group create [SEC_GROUP] openstack security group rule create --dst-port 12345 [SEC_GROUP]* openstack server add security group [SERVER] [SEC_GROUP] openstack security group rule list [SEC_GROUP] --long`
Network:
`openstack network list openstack network show [NETWORK] openstack network create [NETWORK] openstack network delete [NETWORK]`
Subnet:
`openstack subnet list openstack subnet show [SUBNET] openstack subnet create --network --subnet-range [CIDR] [SUBNET] openstack subnet delete [SUBNET]`
Port:
`openstack port list openstack port show [PORT] openstack port create --network [NETWORK] [PORT]** openstack port delete [PORT]`
Floating IP:
`openstack floating ip list openstack floating ip create [EXT_NET]* openstack server add floating ip [SERVER] [IP_ADDRESS] openstack server remove floating ip [SERVER] [IP_ADDRESS] openstack floating ip delete [IP_ADDRESS] openstack floating ip set [IP_ADDRESS]*`
Router:
`openstack router create [ROUTER] openstack router show [ROUTER] openstack router delete [ROUTER] openstack router list` set [ROUTER] gateway: `neutron router-gateway-set [ROUTER] public` clear [ROUTER] gateway: `neutron router-gateway-clear [ROUTER]` add private subnet to [ROUTER]: `openstack router add subnet [ROUTER] private_subnet` remove private subnet to [ROUTER]: `openstack router remove subnet [ROUTER] private_subnet` list [ROUTER] interfaces: `openstack port list --router [ROUTER]`

* You can add additional parameters here like –egress for outgoing traffic or –protocol [UDP|ICMP]
** You can associate floating ip with specific port by adding –port PORT
*** You could also add –fixed-ip [subnet=SUBNET|ip-address=IP_ADDRESS]

Metadata related commands

Server*:

Metadata related commands
Server*:
retrieve metadata info: `curl http://169.254.169.254/`retrieve latest metadata: `curl http://169.254.169/254/latest`meta-data/$: `curl http://169.254.169/254/latest/meta-data`retrieve floating ip: `curl http://169.254.169/254/latest/meta-data/public-ipv4`retrieve openstack metadata: `curl http://169.254.169.254/openstack`retrieve latest metadata: `curl http://169.254.169.254/openstack/metadata`retrieve meta_data.json: `curl http://169.254.169.254/openstack/meta_data.json`

retrieve metadata info:

curl http://169.254.169.254/

retrieve latest metadata:

curl http://169.254.169/254/latest

meta-data/$:

curl http://169.254.169/254/latest/meta-data

retrieve floating ip:

curl http://169.254.169/254/latest/meta-data/public-ipv4

retrieve openstack metadata:

curl http://169.254.169.254/openstack

retrieve latest metadata:

curl http://169.254.169.254/openstack/metadata

retrieve meta_data.json:
curl http://169.254.169.254/openstack/meta_data.json

* Curl commands are to be ran from running server

Openstack commands:	Old commands:
Server creation*:
`openstack server create \ --image [IMAGE] \ --flavor [FLAVOR] \ --nic [net-id=NETWORK_UUID\|port-id=PORT_UUID] \ --key-name [KEYPAIR] \ [SERVER]`	`nova boot --image [IMAGE] \ --flavor [FLAVOR] \ --nic [net-id=NETWORK_UUID\|port-id=PORT_UUID] \ --key-name [KEYPAIR] \ [SERVER]`

* You could use –nic net-id=NETWORK_UUID,v4-fixed-ip=IP_ADDRESS to specify fixed IP for SERVER

Few examples:
Create cirros-image:
`openstack image create --public --container-format=bare --disk-format=qcow2 --file cirros-0.3.4-x86_64-disk.img cirros-image`
Create network:
`neutron net-create external_network --provider:network_type flat --provider:physical_network extnet --router:external`
Create subnet:
`neutron subnet-create --name public_subnet --enable_dhcp=False --allocation-pool=start=172.30.152.10,end=172.30.152.40 --gateway=172.30.152.1 external_network 172.30.152.0/24`
Create server:
`openstack server create --image 9d904b2f-8774-4db8-b195-8aa47e3ebe3c --flavor 1 --nic net-id=4d75d36b-cc99-46b3-83aa-6ae5a96ae38e cirros`

Cloud config* examples:
Configure time zone:
`#cloud-config timezone: "Europe/Sarajevo"`
Install packages, specific versions**
`#cloud-config packages: - pwgen - pastebinit - [libpython2.7, 2.7.3-0ubuntu3.1]`
Upgrade all packages**:
`#cloud-config package_upgrade: true`
Install NTP if neccessary and configure NTP server map
`#cloud-config ntp: servers: - my.ntp.server - ntp.ubuntu.com - 192.168.23.2`
Configure CentOS users centos and root with password
`#cloud-config chpasswd: list: \| root:changeme centos:changeme expire: False ssh_pwauth: True`

* Yaml format text files that begin with #cloud-config and are executed by cloud-init process present in most cloud image files, but not in cirros.
** Works with all package managers if specific distro-package is not specified f.e. 2.7.3-0ubuntu3.1

21. 04. 202021. 04. 2020

Mount removable drives and network locations in the Windows Subsystem for Linux

You can mount a removable device or mounted network drive by using drvfs:

sudo mkdir /mnt/x
sudo mount -t drvfs X: /mnt/x

Or you could use UNC (Universal Naming Convention) format to mount:

sudo mkdir /mnt/folder
sudo mount -t '\\server\folder' /mnt/folder

20. 09. 201920. 09. 2019

Create swap file on Raspberry Pi Zero W

First check if you have swap configured:

swapon --show

IF empty you can create for example 256MB swap file with following commands:

fallocate -l 256M /var/swap
chmod 600 /var/swap
mkswap /var/swap
swapon /var/swap

You can again check if swap is properly configured:

#swapon --show
NAME      TYPE SIZE USED PRIO
/var/swap file 256M   0B   -2

#free -m
              total        used        free      shared  buff/cache   available
Mem:            480         109         140           9         229         312
Swap:           255           0         255

27. 07. 201927. 07. 2019

Download all files from website using wget

For example if you want to download rom files from specific site and to ignore all other file types you can use:

wget -A rom -m -p -E -k -K -np http://files.swos.eu/amiga/kickroms/

Additional extensions you can separate with comma (,).