Use apt-get through http proxy

This issue happened when one day, my lovely company decided that they want to implement a proxy server, and without me realizing, not just browser will be affected, apt-get also will be affected.

How do you know that you apt-get command encounter proxy issue, when ypu received "401 authenticationrequired" error after running your apt-get command, like below:

$ sudo apt-get update
...
W: Failed to fetch http://my.archive.ubuntu.com/ubuntu/dists/raring-updates/universe/binary-i386/Packages  401  authenticationrequired
...

How to encounter this?

Method 1 (if you have GUI)

1. Simply open your browser, and the proxy will ask for authentication
2. Fill up your authentication.
3. Rerun your apt-get command

Method 2 (if you have GUI)

1. Go to System -> preferences -> Network Proxy
2. Under Proxy Configuration, put in you proxy details
3. Rerun apt-get

Method 3 (without GUI) - temporary proxy session

1. export the http_proxy environment variable using this command:

   $ sudo export http_proxy='http://myusername:mypassword@myproxyaddress:myproxyport'

2. Rerun apt-get

Method 4 (without GUI) - temporary proxy session

1. run the apt-get command with proxy in one line:

   $ sudo bash -c 'http_proxy="http://myusername:mypassword@myproxyaddress:myproxyport/" apt-get update'

Method 5 (without GUI) - permanent proxy setting on .bashrc

1. Put the settings into .bashrc:

   $ echo "http_proxy='http://myusername:mypassword@myproxyaddress:myproxyport'" >> .bashrc

2. Activate the change:

   $ source .bashrc

3. Rerun apt-get

Method 6 (without GUI) - permanent settings on apt.conf ~ need sudo

1. Append your proxy settings to /etc/apt/apt.conf (choose your proxy, either http, https, ftp, or socks:

   $ sudo echo -e 'Acquire::http::proxy "http://myusername:mypassword@myproxyaddress:myproxyport/";\nAcquire::https::proxy "https://myusername:mypassword@myproxyaddress:myproxyport/";\nAcquire::ftp::proxy "ftp://myusername:mypassword@myproxyaddress:myproxyport/";\nAcquire::socks::proxy "socks://myusername:mypassword@myproxyaddress:myproxyport/";' >> /etc/apt/apt.conf

2. Rerun apt

You can refer to here, on how to determine your proxy ip address and port using curl.

Using tar on the fly to efficiently transfer file over ssh (wondertar)

Have you been in the situation where you want to transfer a big file, and decided to tar it before transferring but being limited by the disk space available on the machine?

Well, worry no more as I will show you how you can do a tar on the fly while ssh'ing, to overcome that limitation.

Method 1:

ssh foo@machine-to-keep-the-data "tar czpf - /data/to/be/transferred" | tar xzpf - -C /the/data/new/place

What this command will do is to create a tar file (tar czpf), and untar it at the other side of the ssh (tar xvpf) command, where c is for create tar, z is to use gzip, p is for preserving permission, f is for file which is to be zipped, - is for stdin or stdout and x is for extract


Method 2:

tar cpf - /data/to/be/transferred | ssh foo@machine-to-keep-the-data tar xpf - -C /the/data/new/place" 

This command will tar the file, and untar it at the other end, same as above, but just different command arrangement


Method 3 (this is useful if you want to tar it. and keep it that way on the other end, without untarring it):

tar czf - -C /data/to/be/transferred | ssh foo@machine-to-keep-the-data "cat - > /the/data/new/place/backup.tar.gz"

Method 4 (add pv to the middle of the pipes to monitor the transfer speed):

sudo apt-get install pv; ssh foo@machine-to-keep-the-data "tar czpf - /data/to/be/transferred" | pv | tar xzpf - -C /the/data/new/place

That's all, hope you will find these useful.