Archives
git: reducing repository size (gc and destructive)
This article discusses two approaches to reducing the size of a Git repository: non-destructive garbage collection and destructive history rewriting. It provides a script for running git gc on multiple repositories and links to resources for more advanced techniques.
Garbage Collection (non-destructive)
This works especially well when removing a file added in the most recent unpushed commit. Git Garbage Collection automates some of these cleanup jobs.
I ran the following over my source folders:
AWS sync is not reliable!
While migrating from s3cmd to the AWS S3 CLI, I noticed that files did not reliably sync when using the AWS CLI.
I tested this behavior with different versions, and they all exhibited the same issue:
python2.7-awscli1.9.7python2.7-awscli1.15.47python3.6-awscli1.15.47
Test Setup
-
Set up the AWS CLI utility and configure your credentials.
-
Create a testing S3 bucket.
-
Set up some random files:
# Create 10 random files of 10MB each for i in {1..10}; do dd if=/dev/urandom of=multi/part-$i.out bs=1MB count=10; done; # Then copy the first 5 files over mkdir multi-changed cp -r multi/part-{1,2,3,4,5}.out multi-changed # And replace the content in the remaining 5 files (6-10) for i in {6..10}; do dd if=/dev/urandom of=multi-changed/part-$i.out bs=1MB count=10; done;
Testing S3 sync with AWS CLI
Cleanup
$ aws s3 rm s3://l3testing/multi --recursive
Inital sync
$ aws s3 sync multi s3://l3testing/multi
upload: multi/part-1.out to s3://l3testing/multi/part-1.out
upload: …Markdown Notetaking
This is a collection of interesting links and resources I came across in September 2018, covering topics like Kotlin, Python, Markdown, note-taking apps, web development, Linux, and more.
Kotlin
- Kotlin vs. Scala vs. Java
- What should I choose instead of Java?
- Super Kotlin Tutorial
- SQL with Kotlin Exposed
- Kotlin Playground
Python
Some notetaking apps you should give a try. At least Notion is very promising (yet you have to pay).
Connecting to CheckPoint VPN SNX in Linux
This guide explains how to connect to CheckPoint VPN using SNX (SSL Network Extender) on Linux systems.
Prerequisites
Ensure you have received the following information from your VPN administrator:
- VPN Certificate file (.p12)
- Your VPN password
- Your server username
Please use that information to replace placeholders in scripts found in this tutorial.
Installation Script
You can either download from their website (crappy and frustrating) or get it directly via http://gateway-ip.
Connecting to Checkpoint QVPN SXN in Linux
This tutorial provides a step-by-step guide on how to connect to a Checkpoint QVPN on a Linux system. It covers prerequisites, installation of the snx client, and automation scripts for connecting and disconnecting.
Prerequisites
Ensure you have received their E-Mail and following information:
- VPN Certificate file (.p12)
- Your VPN password
- Your server username
Please use that information to replace placeholders in scripts found in this tutorial.
Installation script
You can either download from their website (which can be frustrating) or get it directly via http://gateway-ip.
GitLab: checkout all available repositories
This guide provides a set of shell commands to automate the process of checking out all available repositories from one or more GitLab instances. It leverages the GitLab API, jq, and parallel to efficiently clone projects.
Generate a private token
https://<GITLAB-SERVER1>/profile/personal_access_tokens
https://<GITLAB-SERVER2>/profile/personal_access_tokens
Checkout a list of all available repositories
QUERY=".[] | .path_with_namespace + "\t" + .ssh_url_to_repo" # JQ Query
curl --request GET --header "PRIVATE-TOKEN: <PRIVATE-TOKEN>" "<GITLAB-SERVER1>/api/v4/projects?simple=true&per_page=65536" | jq -r "$QUERY" > repo.list
curl --request GET --header "PRIVATE-TOKEN: <PRIVATE-TOKEN>" "<GITLAB-SERVER2>/api/v3/projects?simple=true&per_page=65536" | jq -r "$QUERY" >> repo.list
Create directories for repositories
cat repo.list | cut -f1 | xargs mkdir -p
Checkout projects …
Infojunk August 2018
This is a collection of interesting links and resources I came across in August 2018, covering topics like Linux, Apache, hardware, coding, Python, and math.
Linux
- remove-kernel-scripts @ StackOverflow
- Optimizing Linux for slow computers
- Ananicy (ANother Auto NICe daemon) is a shell daemon created to manage processes’ IO and CPU priorities, with a community-driven set of rules.
It’s about responsiveness - not about the best performance!
Apache
Hardware
Coding
- Toy Benchmark for Java, Rust, Python, PHP, JavaScript
- Python3 Kurs (German)
- Law of Demeter
- Ada (programming language) - the first programming language
- Why Python does not have a switch-statement
Python
- Decorators
- Data Structures
- Collections
- Collections II
- PEP
- Multiprocessing Tutorial
- Non-Blocking stream reader
- Writing/Reading GZIP-files
- Async Subprocess
- Addict Python Lib
- Better Type Hinting
- dotmap
- Interactive python console
- Dynamically create stubs
Yes, a deep dive …
Decoding MySQL ~/.mylogin.cnf
Little tool to decode MySQL’s badly secured login-path. It does the same like the official MySQL server tools “my_print_defaults” (based on MySQL OSS python libs).
More security can be achieved by: https://www.percona.com/blog/2016/10/12/encrypt-defaults-file/
#!/usr/bin/env php
<?php
$fp = fopen(getenv('HOME') . '/.mylogin.cnf', "r");
if (!$fp) {
die("Cannot open .mylogin.cnf");
}
fseek($fp, 4);
$key = fread($fp, 20);
// generate real key
$rkey = "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0";
for ($i = 0; $i < strlen($key); $i++) {
$rkey[$i % 16] = ($rkey[$i % 16] ^ $key[$i]);
}
$section = null;
$settings = [];
while ($len = fread($fp, 4)) {
// as integer
$len = unpack("V", $len);
$len = $len[1];
// decrypt
$crypt = fread($fp, $len);
$plain = openssl_decrypt($crypt, 'aes-128-ecb', $rkey, true);
$decoded = preg_replace("/[^\\x32-\\xFFFF]/", "", $plain);
if (preg_match( …Albert Launcher 0.14: Switch Application Window Plugin
This post introduces a Python extension for the Albert Launcher that allows you to switch between application windows.
Since I really don’t like the Switcher Plugin for GNOME and I’m a fan of Albert Launcher, I created this extension. Maybe it will get accepted into their python extensions.
Drop the following code to ~/.local/share/albert/org.albert.extension.python/switch-app-window.py or other provided locations and activate it within the extensions menu:
import re
import subprocess
from albertv0 import *
__iid__ = "PythonInterface/v0.1"
__prettyname__ = "Switch App Window"
__version__ = "1.0"
__trigger__ = "w "
__author__ = "Markus Geiger <mg@evolution515.net>"
__id__ = "window"
__dependencies__ = []
iconPath = iconLookup("go-next")
def handleQuery(query):
stripped = query.string.strip()
if not query.isTriggered and not stripped:
return
results = []
process = subprocess.Popen([ …Git: Encrypt Credentials Within a Repository
This article explores the concept of encrypting credentials within a Git repository. It demonstrates a method using git smudge/clean filters but ultimately advises against it, advocating for the use of config servers instead.
Especially in the microservices era, you should use a config server and never store your credentials in a repository!
You should not use git smudge/clean filters for encryption. Why? Here’s an example!
Let’s create an example repository
% TMP=$(mktemp -d)
% cd $TMP
% git init
% echo 'Hello world!' > credentials
Add .gitattributes
/credentials filter=crypto
Add .git/config
[filter "crypto"]
smudge = openssl enc -aes-256-cbc -salt
clean = openssl enc -aes-256-cbc -salt
require
Note: require indicates that these commands need to exit with code 0, otherwise it could happen that these files are added without any encryption. You can test this by using smudge = gpg -d -q –batch –no-tty -r <SIGNATURE> and clean = gpg -ea -q –batch …