Secwall's notes

A little brew helper to remove non needed formulas

2024-02-24T00:00:00+00:00

Problem</h1>
One of the annoying things about the Homebrew</a> package manager are garbage dependencies. Sometimes, a formula switches from a source-only version to a bottle, so its build dependencies are no longer required. Sometimes, a formula simply switches from one library to another.</p>
Let's take one example: the `include-what-you-use</code> formula depends on llvm</code>. If you run brew upgrade</code> at a time when llvm</code> already has version N, but include-what-you-use</code> still depends on version N-1, you will end up with two versions of llvm</code>. Some time will go by, and include-what-you-use</code> will depend on llvm</code> version N.</p>`
`So, what now? Do we need llvm</code> N-1</code>? There's an easy way to check: just try removing it and homebrew will tell you.</p>`
`But I like to see which formulas are not used as dependencies in my system using some simple command.</p>`

Helper</h1>

So I wrote a little helper</a> for this task. Here is how to use it:</p>
$ brew-helper list
</span>...
</span>llvm@16 <- here is an old version of llvm (at the moment of this writing the current version is 17)
</span>...
</span></code></pre>
Let's get rid of it:</p>
$ brew-helper rm-dep llvm@16
</span>Removing llvm@16
</span>Found new unused dep: six
</span>Removing six
</span></code></pre>
As you can see, after the removal of llvm@16</code>, a new garbage dependency has appeared (six</code>). And our helper has gotten rid of it too.</p>
Hope you find it usefull.</p>


Barman page-level incremental backups
2016-06-18T00:00:00+00:00
Warning</h2>
I present experimental fork</a> of
pgbarman</a> in this post. Some config parameters
could change in merging to upstream process. Stay tuned.</p>
TL;DR</h2>
I've added agent (barman-incr</code>) which implements parallel compressed
page-level incremental backups support to barman.</p>
Quick example</h1>
I'll use CentOS 7 in this example. If you are using different distro you
should probably change install phase.</p>
Install</h2>
We'll add some required repos:</p>
sudo</span> yum install epel-release
</span>sudo</span> yum install https://download.postgresql.org/pub/repos/yum/9.5/redhat/rhel-7-x86_64/pgdg-centos95-9.5-2.noarch.rpm
</span></code></pre>
Now we'll build barman</code> and barman-incr</code> packages from source (I assume that
you are in your home dir now):</p>
sudo</span> yum install git rpm-build
</span>git</span> clone https://github.com/secwall/barman
</span>mv</span> barman barman-1.6.2a1
</span>mkdir -p</span> rpmbuild/SOURCES
</span>tar -zcf</span> /home/vagrant/rpmbuild/SOURCES/barman-1.6.2a1.tar.gz barman-1.6.2a1
</span>rpmbuild -bb</span> barman-1.6.2a1/rpm/barman.spec
</span></code></pre>
In result we should get barman-1.6.2-0.1.a1.el7.centos.noarch.rpm</code> and
barman-incr-1.6.2-0.1.a1.el7.centos.noarch.rpm</code> in ~/rpmbuild/RPMS/noarch</code></p>
Installing pkgs:</p>
sudo</span> yum install rpmbuild/RPMS/noarch/barman-*
</span></code></pre>
Postgresql setup</h2>
We'll setup pgsql without any replicas on localhost (this is only for demo
purposes, you should never do this on production environment, always use
replication, strong passwords, ssl, and so on).</p>
sudo</span> yum install postgresql95-server postgresql95-contrib
</span>sudo</span> /usr/pgsql-9.5/bin/postgresql95-setup initdb
</span>Initializing</span> database ... OK
</span>
</span>sudo</span> systemctl restart postgresql-9.5
</span>
</span>sudo -u</span> postgres psql
</span>postgres</span>=</span># </span>create</span> user barman with encrypted password '</span>barman</span>' superuser;
</span>CREATE</span> ROLE
</span>
</span>cat </span><<</span>EOF </span>| </span>sudo</span> tee</span> --append</span> /var/lib/pgsql/9.5/data/postgresql.conf
</span>wal_level = hot_standby
</span>archive_mode = on
</span>archive_command = 'rsync %p barman@localhost:/var/lib/barman/test/incoming/%f'
</span>EOF
</span>
</span>sudo</span> sed</span> -i -e </span>'</span>s/ident/md5/g</span>' /var/lib/pgsql/9.5/data/pg_hba.conf
</span>sudo</span> grep</span> -v </span>'</span>^#</span>' /var/lib/pgsql/9.5/data/pg_hba.conf | </span>sed </span>'</span>/^\s*$/d</span>'
</span>local   </span>all             all                                     peer
</span>host</span>    all             all             127.0.0.1/32            md5
</span>host</span>    all             all             ::1/128                 md5
</span>
</span>sudo</span> systemctl restart postgresql-9.5
</span></code></pre>
Barman setup</h2>
Now we'll configure barman to backup our local postgresql</p>
cat </span><<</span>EOF </span>| </span>sudo</span> tee</span> --append</span> /etc/barman.conf
</span>[test]
</span>backup_method = incr
</span>description =  "Test PostgreSQL Database"
</span>ssh_command = ssh postgres@localhost
</span>conninfo = host=localhost user=barman dbname=postgres password=barman
</span>incr_compress = gzip-4
</span>incr_parallel = 4
</span>EOF
</span></code></pre>
Seems simple. We'll compress our backups with gzip -4, use 4 processes to make
backup. Refer to man 5 barman</code> for more info.</p>
OpenSSH Auth Setup</h2>
Gen ssh-keys for barman and postgres users. Add them into authorized_keys and
disable strict host checking.</p>
sudo -u</span> barman mkdir</span> -p</span> /var/lib/barman/.ssh
</span>sudo -u</span> barman ssh-keygen</span> -f</span> /var/lib/barman/.ssh/id_rsa</span> -N </span>''
</span>
</span>sudo -u</span> postgres mkdir</span> -p</span> /var/lib/pgsql/.ssh
</span>sudo -u</span> postgres ssh-keygen</span> -f</span> /var/lib/pgsql/.ssh/id_rsa</span> -N </span>''
</span>
</span>sudo</span> cat /var/lib/barman/.ssh/id_rsa.pub | </span>sudo</span> tee /var/lib/pgsql/.ssh/authorized_keys
</span>sudo</span> cat /var/lib/pgsql/.ssh/id_rsa.pub | </span>sudo</span> tee /var/lib/barman/.ssh/authorized_keys
</span>
</span>cat </span><<</span>EOF </span>| </span>sudo -u</span> barman tee /var/lib/barman/.ssh/config
</span>Host *
</span>    StrictHostKeyChecking no
</span>EOF
</span>
</span>cat </span><<</span>EOF </span>| </span>sudo -u</span> postgres tee /var/lib/pgsql/.ssh/config
</span>Host *
</span>    StrictHostKeyChecking no
</span>EOF
</span>
</span>sudo</span> chmod 700 /var/lib/barman/.ssh && </span>sudo</span> chown barman:barman /var/lib/barman/.ssh</span> -R
</span>sudo</span> chmod 700 /var/lib/pgsql/.ssh && </span>sudo</span> chown postgres:postgres /var/lib/pgsql/.ssh</span> -R
</span></code></pre>
Check that everything works as expected:</p>
sudo</span> barman check test
</span>Server</span> test:
</span>        </span>PostgreSQL:</span> OK
</span>        </span>superuser:</span> OK
</span>        </span>wal_level:</span> OK
</span>        </span>directories:</span> OK
</span>        </span>retention</span> policy settings: OK
</span>        </span>backup</span> maximum age: OK (no last_backup_maximum_age provided)
</span>        </span>compression</span> settings: OK
</span>        </span>failed</span> backups: OK (there are 0 failed backups)
</span>        </span>minimum</span> redundancy requirements: OK (have 0 backups, expected at least 0)
</span>        </span>ssh:</span> OK (barman-incr)
</span>        </span>version:</span> OK (ok)
</span>        </span>not</span> in recovery: OK
</span>        </span>archive_mode:</span> OK
</span>        </span>archive_command:</span> OK
</span>        </span>continuous</span> archiving: OK
</span>        </span>archiver</span> errors: OK
</span></code></pre>
If you see WAL archive check failure. Try to use pg_switch_xlog()</code> function on
postgres cluster and check the logs.</p>
Backup</h2>
We'll add some data with pgbench</code>, make full backup, change some data, and
finally make incremental backup.</p>
sudo -u</span> postgres /usr/pgsql-9.5/bin/pgbench</span> -i -s</span> 100</span> --foreign-keys
</span>
</span>sudo</span> du</span> -hs</span> /var/lib/pgsql/9.5/data/base/
</span>1.5G</span>    /var/lib/pgsql/9.5/data/base/
</span></code></pre>
Full backup:</p>
sudo</span> barman backup test
</span>barman</span> list-backup test
</span>test</span> 20160618T013124 - Sat Jun 18 01:32:55 2016 - Size: 84.3 MiB - WAL Size: 0 B
</span></code></pre>
Actually I cheat in some way here because pgbench data compressed really great.
You'll not see such impressive results on real data.</p>
Now we'll update 1000 rows:</p>
sudo -u</span> postgres psql
</span>postgres</span>=</span># </span>update</span> pgbench_accounts set abalance = 100 where aid % 10000 = 2;
</span>UPDATE</span> 1000
</span></code></pre>
Another backup (now it'll be really fast):</p>
sudo</span> barman backup test
</span>sudo</span> barman list-backup test
</span>test</span> 20160618T013605 - Sat Jun 18 01:37:41 2016 - Size: 170.1 KiB - WAL Size: 0 B
</span>test</span> 20160618T013124 - Sat Jun 18 01:32:55 2016 - Size: 84.3 MiB - WAL Size: 1.8 MiB
</span></code></pre>
Wow. This looks really good.</p>
What is next?</h1>
Read the docs (man 5 barman</code>) and adjust options to suit your requrements.
Important note: in real-world setup you'll need to install barman-incr</code> on
database host to make backup (ssh check will fail if you enable incr backups
without agent on host).</p>


QuickFix Log Pretty Print
2014-05-03T00:00:00+00:00
Problem</h1>
If you are using QuickFix</a> and don't want to
remember zillion of FIX tags you are in big trouble. Because messages.log
often looks like this:</p>
[2014-05-02 20:22:04.62] 8=FIX.4.4|9=263|35=X|34=10115815|49=CNTP|52=20140502-20:22:04.656|56=secwall|262=20|268=4|279=0|269=0|278=1716|55=CHF/JPY|270=116.398|271=1000000|346=1|279=2|269=0|278=1722|55=CHF/JPY|279=0|269=1|278=1837|55=CHF/JPY|270=116.462|271=1000000|346=1|279=2|269=1|278=1819|55=CHF/JPY|10=026|
</span></code></pre>
(Standard fields separator (001 - octal 1) is replaced here with "|")
So what? Do you remember what 268=4 means?</p>
If you just need to read several messages use
minifix</a> (Windows application, runs ok under
wine). You could just copy-paste part of log in it and click on particular
message.</p>
But what should we do if we want to grep log like in example below?</p>
cat</span> messages.log | <mygrep> NoMDEntries=4
</span></code></pre>
Solution</h1>
QuickFix has fix dictionary files (xml) with it. And
here</a> is simple python script for
translating numeric tags into their names.</p>
Simple example:</p>
cat</span> messages.log | </span>fixpp -d</span> /path/to/dictionary.xml | </span>grep</span> NoMDEntries=4
</span></code></pre>
But there are more. At first I don't like to use long paths. So there are
"quicklinks". Example usage:</p>
cat ~</span>/.config/fixpp/fixpp.conf
</span>[quicklink]
</span>fix44</span> = /opt/feed-fix-prod/selected/data/etc/FIX44.xml
</span>...
</span>
</span>cat</span> messages.log | </span>fixpp -d</span> fix44
</span></code></pre>
Another example. Using grep and long format (separated line for each tag):</p>
fixpp -d</span> fix44</span> -i</span> messages.log | </span>grep </span>'</span>Symbol=CHF/JPY</span>' | </span>fixpp -d</span> fix44</span> -s </span>\,</span> -l
</span></code></pre>
Code is real garbage but it works (I'll try to fix this is near future).</p>


Configuring ferm to allow connection only from cloudflare CDN
2014-03-22T00:00:00+00:00
Problem</h1>
VPS for this site is really slow. I wanted to speed up site load a bit.
So I signed up on Cloudflare</a> and set up CDN for it.</p>
What now? Site is still available via ip directly (So if someone want to find
it's real ip he/she could just scan entire internet and try to make GET / request
with HOST header secwall.me - not so easy task, but real).</p>
This is really THE issue if you are going to use cloudflare as DDoS protection.
Attacker task may be really simplier if he/she knows that you use specific hoster
(Leaseweb/Digital ocean/etc.).</p>
Solution</h1>
Warning! This makes your site unavailable without cloudflare.
You will need to disable firewall protection to make it work.</p>
I prefer to use ferm</a> for automatic iptables
configuration. Cloudflare gives us a list of it's ip addresses in
machine parseble format</a>.</p>
Here is simple shell script for downloading this list and checking if it was
changed:</p>
#!/bin/sh
</span>
</span>wget</span> https://www.cloudflare.com/ips-v4</span> -O</span> /tmp/cloudflare-ips-v4.list
</span>
</span>LOCAL</span>=`</span>md5sum</span> /etc/ferm/cloudflare-ipv4.list | </span>cut -d</span>\ </span> -f1</span>`
</span>CLOUD</span>=`</span>md5sum</span> /tmp/cloudflare-ips-v4.list | </span>cut -d</span>\ </span> -f1</span>`
</span>
</span>if </span>[ </span>"$</span>LOCAL</span>" != "$</span>CLOUD</span>" </span>]
</span>then
</span>    </span>mv</span> /tmp/cloudflare-ips-v4.list /etc/ferm/cloudflare-ipv4.list
</span>    </span>/etc/init.d/ferm</span> reload
</span>fi
</span></code></pre>
And example ferm configuraition:</p>
domain </span>ip </span>{
</span>    </span>table </span>filter </span>{
</span>        </span>chain </span>INPUT </span>{
</span>            </span>policy </span>DROP</span>;
</span>
</span>            </span>mod </span>state state </span>INVALID DROP</span>;
</span>            </span>mod </span>state state </span>(</span>ESTABLISHED RELATED</span>) </span>ACCEPT</span>;
</span>
</span>            </span>interface lo </span>ACCEPT</span>;
</span>
</span>            @</span>def </span>$</span>CLOUDFLARE </span>= `</span>cat /etc/ferm/cloudflare-ipv4.list</span>`;
</span>            </span>interface eth0 proto tcp saddr </span>$</span>CLOUDFLARE dport </span>80 ACCEPT</span>;
</span>
</span>            </span>proto icmp </span>ACCEPT</span>;
</span>
</span>            </span>proto tcp dport ssh </span>ACCEPT</span>;
</span>        }
</span>        </span>chain </span>OUTPUT </span>{
</span>            </span>policy </span>ACCEPT</span>;
</span>
</span>            </span>mod </span>state state </span>INVALID DROP</span>;
</span>            </span>mod </span>state state </span>(</span>ESTABLISHED RELATED</span>) </span>ACCEPT</span>;
</span>        }
</span>        </span>chain </span>FORWARD </span>{
</span>            </span>policy </span>ACCEPT</span>;
</span>
</span>            </span>mod </span>state state </span>INVALID DROP</span>;
</span>            </span>mod </span>state state </span>(</span>ESTABLISHED RELATED</span>) </span>ACCEPT</span>;
</span>        }
</span>    }
</span>}
</span>
</span>domain </span>ip6 </span>{
</span>    </span>table </span>filter </span>{
</span>        </span>chain </span>INPUT </span>{
</span>            </span>policy </span>DROP</span>;
</span>
</span>            </span>mod </span>state state </span>INVALID DROP</span>;
</span>            </span>mod </span>state state </span>(</span>ESTABLISHED RELATED</span>) </span>ACCEPT</span>;
</span>
</span>            </span>proto </span>(</span>ipv6</span>-</span>icmp </span>icmp</span>) </span>ACCEPT</span>;
</span>
</span>            </span>proto tcp dport ssh </span>ACCEPT</span>;
</span>        }
</span>        </span>chain </span>OUTPUT </span>{
</span>            </span>policy </span>ACCEPT</span>;
</span>
</span>            </span>mod </span>state state </span>INVALID DROP</span>;
</span>            </span>mod </span>state state </span>(</span>ESTABLISHED RELATED</span>) </span>ACCEPT</span>;
</span>        }
</span>        </span>chain </span>FORWARD </span>{
</span>            </span>policy </span>ACCEPT</span>;
</span>
</span>            </span>mod </span>state state </span>INVALID DROP</span>;
</span>            </span>mod </span>state state </span>(</span>ESTABLISHED RELATED</span>) </span>ACCEPT</span>;
</span>        }
</span>    }
</span>}
</span></code></pre>
Shell script should be placed is cron.hourly (or daily). This will make rules
update automatic.</p>

Secwall's notes

A little brew helper to remove non needed formulas

Barman page-level incremental backups

Warning</h2> I present experimental fork</a> of pgbarman</a> in this post. Some config parameters could change in merging to upstream process. Stay tuned.</p>

What is next?</h1> Read the docs (man 5 barman</code>) and adjust options to suit your requrements. Important note: in real-world setup you'll need to install barman-incr</code> on database host to make backup (ssh check will fail if you enable incr backups without agent on host).</p>

QuickFix Log Pretty Print

Configuring ferm to allow connection only from cloudflare CDN

Warning</h2>
I present experimental fork</a> of pgbarman</a> in this post. Some config parameters could change in merging to upstream process. Stay tuned.</p>

What is next?</h1>
Read the docs (`man 5 barman</code>) and adjust options to suit your requrements. Important note: in real-world setup you'll need to install barman-incr</code> on database host to make backup (ssh check will fail if you enable incr backups without agent on host).</p>`