Force a mongodb in a replica set to be a primary

There was one situation, where our production mongo server suddenly becomes secondary, causing any write and read to the server to fail. Searching in the mongo documentation, we found an easy solution. Below are the steps.

First step:
Check that your replica is running fine
mongo> rs.status()

Second step:
Freeze all mongo node in the replica that you do not want to be primary for lets say, 120 seconds. Access the mongo shell, and run below command, do this for all nodes that you do not want to be primary
mongo> rs.freeze(120) 

Third step:
Demote the current primary, so that other node that has not been frozen, will take over as primary. Run this in mongo shell, to demote the node from being a primary, for 120 seconds
mongo> rs.stepdown(120)

That's it, run rs.status() again to make sure that your desired server is now a primary.

Reference: https://docs.mongodb.com/manual/tutorial/force-member-to-be-primary/

Use centos repository in RHEL

Get the repository url in your country, by visiting http://mirror-status.centos.org/. In my case, since I am in Malaysia, I chose http://centos.ipserverone.com/centos/

Create a repo file in /etc/yum.repos.d
# cat > /etc/yum.repos.d/centos.repo
[centos]
name=CentOS-7
baseurl=http://centos.ipserverone.com/centos/7/os/x86_64/
enabled=1
gpgcheck=1
gpgkey=http://centos.ipserverone.com/centos/7/os/x86_64/RPM-GPG-KEY-CentOS-7

Press ctrl-d to exit and save

Check whether your new centos repo is listed and ready to be used
# yum repolist

Done

Backup RHEV/RHV/ovirt disk image to external drive

Login to RHV/RHEV/ovirt administrator interface

Search for your VM, lets say I want to search for windows VM

Right click on the VM name, and choose 'Create Snapshot'

Give a proper name, and tick "Save Memory" if you want to save current state of memory as well. Please be warned, that by choosing "Save Memory", the VM will be paused while the snapshooting is in progress. 

If you check the Snapshots tab, the status of the snapshot will be Locked, and the VM will be paused

Once completed, the status will be changed to Ok

Now, open a terminal inside the rhev machine, and search for the disk image snapshot by running below command:

# virsh -r dumpxml windows | grep "source file"

Your newly created snapshot is on the last line of the above command. Copy the file location, and put it into your external drive

# cp /rhev/data-center/00000002-0002-0002-0002-00000000035d/747b7d84-68d5-4436-98f5-baeec68381e1/images/8af6c761-94ac-4adb-bf33-f009e3dd3dc7/../8af6c761-94ac-4adb-bf33-f009e3dd3dc7/../8af6c761-94ac-4adb-bf33-f009e3dd3dc7/../8af6c761-94ac-4adb-bf33-f009e3dd3dc7/../8af6c761-94ac-4adb-bf33-f009e3dd3dc7/e42c8e85-9566-4953-b881-70c7ce97de0b /mnt/my_external_drive/windows-snapshot-20181219.raw

If you want a different format, use qemu-img command to change it. For example, you require this image to be vmware compatible

# qemu-img -f raw -O vmdk /mnt/my_external_drive/windows-snapshot-20181219.raw /mnt/my_external_drive/windows-snapshot-20181219.vmdk

Send postgresql log to syslog

Enable syslog to listen to udp port 514 (the port number can be changed to suit your need)

# vi /etc/rsyslog.conf

$ModLoad imudp

$UDPServerRun 514

Restart rsyslog

# systemctl restart rsyslog

Check if rsyslog is now listening to 514 udp port

# ss -tulpn | grep 514

udp    UNCONN     0      0                      *:514                   *:*      users:(("rsyslogd",15281,3))

udp    UNCONN     0      0                     :::514                  :::*      users:(("rsyslogd",15281,4))

Set postgres to log to syslog

# vi /var/lib/pgsql/9.5/data/postgresql.conf

log_destination = 'syslog'

Restart postgres

# systemctl restart postgresql-9.5

Check if your postgres log is now being logged by syslog

# tail -f /var/log/messages 

...

Dec 11 08:39:21 mypostgres postgres[21518]: [3-1] < 2018-12-11 08:39:21.932 UTC >LOG:  autovacuum launcher started

Install nvidia driver for Tesla card on ubuntu 16.04

Check what driver your card is currently using, in this case, the card is using the opensource nvidia drivers called nouveau
$ lspci -v | grep -A8 NVIDIA
3d:00.0 3D controller: NVIDIA Corporation Device 1bb3 (rev a1)
        Subsystem: NVIDIA Corporation Device 11d8
        Flags: bus master, fast devsel, latency 0, IRQ 40
        Memory at b7000000 (32-bit, non-prefetchable) [size=16M]
        Memory at 4bfe0000000 (64-bit, prefetchable) [size=256M]
        Memory at 4bff0000000 (64-bit, prefetchable) [size=32M]
        Capabilities:
        Kernel driver in use: nouveau
        Kernel modules: nvidiafb, nouveau

Download the driver for your card from nvidia download website

Choose the correct card model, and operating system, and click SEARCH

Click DOWNLOAD

Click AGREE & DOWNLOAD to download

If you want to download using wget, right click on the AGREE & DOWNLOAD button, and choose "Copy Link Address". Run wget against the link

$ wget http://us.download.nvidia.com/tesla/410.79/nvidia-diag-driver-local-repo-ubuntu1604-410.79_1.0-1_amd64.deb

Install the driver

$ sudo apt install ./nvidia-diag-driver-local-repo-ubuntu1604-410.79_1.0-1_amd64.deb

Make sure you have now nvidiafb in kernel modules 

$ lspci -v | grep -A15 NVIDIA

3d:00.0 3D controller: NVIDIA Corporation Device 1bb3 (rev a1)

        Subsystem: NVIDIA Corporation Device 11d8

        Flags: bus master, fast devsel, latency 0, IRQ 40

        Memory at b7000000 (32-bit, non-prefetchable) [size=16M]

        Memory at 4bfe0000000 (64-bit, prefetchable) [size=256M]

        Memory at 4bff0000000 (64-bit, prefetchable) [size=32M]

        Capabilities: [60] Power Management version 3

        Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+

        Capabilities: [78] Express Endpoint, MSI 00

        Capabilities: [100] Virtual Channel

        Capabilities: [250] Latency Tolerance Reporting

        Capabilities: [128] Power Budgeting

        Capabilities: [420] Advanced Error Reporting

        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024

        Capabilities: [900] #19

        Kernel driver in use: nouveau

        Kernel modules: nvidiafb, nouveau

Update

$ sudo apt update

Install cuda-drivers

$ sudo apt install cuda-drivers

Reboot your machine

$ sudo reboot

Check whether your card is using nvidia driver, and not nouveau

$ lspci -v | grep -A8 NVIDIA

3d:00.0 3D controller: NVIDIA Corporation Device 1bb3 (rev a1)

        Subsystem: NVIDIA Corporation Device 11d8

        Flags: bus master, fast devsel, latency 0, IRQ 88

        Memory at b7000000 (32-bit, non-prefetchable) [size=16M]

        Memory at 4bfe0000000 (64-bit, prefetchable) [size=256M]

        Memory at 4bff0000000 (64-bit, prefetchable) [size=32M]

        Capabilities:

        Kernel driver in use: nvidia

        Kernel modules: nvidiafb, nouveau, nvidia_410_drm, nvidia_410

Using tsocks with apt to bypass proxy

Recently, I encountered an error while trying to update ubuntu server 16.04. The error is as below:

E: Failed to fetch http://my.archive.ubuntu.com/ubuntu/dists/xenial/InRelease  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)

It seems there is a proxy somewhere in the network, that I do not know. Asking around, even the owner of the server does not even know any proxy server inside their network. So, what I did was, create a socks proxy on my own, to a server outside of the network, and tunnel the apt connection using an application called tsocks.

First, you need to install tsocks. This is a bit tricky since we cannot use the apt command to download and install it from the internet. So I downloaded the deb file, using wget.

$ wget -c http://archive.ubuntu.com/ubuntu/pool/universe/t/tsocks/tsocks_1.8beta5-9.3_amd64.deb

Once downloaded, install using apt on the local file.

$ sudo apt install ./tsocks*.deb

Once installed, create a socks proxy to a server outside of the network, on localhost port 8888 (this is just my favorite number for socks proxy, you can use any number above 1000). You need to be able to ssh to the outside server for this to be possible.

$ ssh foo@server.outside -D 8888

Change the tsocks config file, to suite your new socks proxy

$ sudo cat /etc/tsocks.conf

# You just need to change these 3 lines

server = 127.0.0.1

server_type = 5

server_port = 8888

$ sudo -i 

# tsocks apt update

# tsocks apt upgrade

P/S: You can also set an environment variable called http_proxy, if you do not want to install tsocks, but this setting will be only active on your current bash session. This can be accomplished by:

# export http_proxy='socks5://localhost:8888'

# apt update && apt upgrade