hur.st's bl.aagh

BSD, Ruby, Rust, Rambling

ioztat

ZFS dataset IO statistics

[python] [zfs]

ioztat is to OpenZFS datasets what zpool-iostat(8) is to ZFS pools and vdevs.

Where zpool-iostat provides IO statistics for accesses to the lower level components that make up a ZFS filesystem — the devices that underpin it and the mirrored, striped, and parity-combined virtual devices these are part of — ioztat does so for the filesystems and virtual block devices layered on top.

Continue Reading

faccess

Basic cross-platform file accessibility checks for Rust

[rust]

faccess adds file accessbility checks for file paths to Rust:

pub trait PathExt {
    fn access(&self, mode: AccessMode) -> std::io::Result<()>;
    fn readable(&self) -> bool;
    fn writable(&self) -> bool;
    fn executable(&self) -> bool;
}

impl PathExt for std::path::Path;
Continue Reading

Elite Shield Tester

Find the best-possible Elite Dangerous Shields

[rust] [javascript] [elite] [gaming]

In the popular space simulation game, Elite Dangerous, one of the challenges is determining how best to outfit your ship — if you’re going into combat, choosing the right shield, with the right boosters and the right engineering can be the difference between life or death.

So why not sap the fun out of it by using a computer to work out the best configuration for us?

Continue Reading

Compactor

A friendly user interface to Windows 10 filesystem compression

[rust] [windows]

One of the more notable features that shipped with Windows 10 is a new filesystem compression system based on the Windows Overlay Filesystem architecture, originally developed for transparent handling of disk images.

It’s intended for use in applications, in particular the base OS (in a feature referred to as CompactOS), but sadly this is only exposed to users via a command line program — compact.exe — and most users fail to take advantage of it.

Compactor offers this as an alternative:

Pretend there's a nice GUI here. Continue Reading

cw

Count Words - a Rust wc implementation

[rust]

cw is a fast Rust reimplementation of the classic Unix wc command, featuring fast paths for most common modes of operation, including SIMD-accelerated line and UTF-8 codepoint counting via the bytecount crate (closing issue #41 there in the process).

It also supports multithreading, because of course it does.

Even in single-threaded mode it is almost always much faster than either FreeBSD or GNU wc implementations.

Continue Reading

monotime

Rust-style monotonic clocks in Ruby

[ruby] [rust]

Monotime offers an alternative to the typical method of monotonic timekeeping in Ruby:

# Bog standard Ruby
start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
do_something
elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - start

# With monotime
require 'monotime'
include Monotime

start = Instant.now
do_something
elapsed = start.elapsed

# or
elapsed = Duration.measure { do_something }

It’s less fiddly, and uses proper types wrapping elapsed times in nanoseconds, which is about as close to the native representations as Ruby gets.

Continue Reading

Comlim

Ruby command runner with resource limits

[ruby]

Comlim is a command builder for Ruby focusing on an easy interface to restricted execution—limiting memory, runtime, and command output.

Runner = Comlim.memory(32.megabytes)
               .cputime(30.seconds)
               .runtime(60.seconds)
               .output(4.kilobytes)

RubyRunner = Runner.command('ruby').arg('-e')
RubyRunner.arg('loop { }').execute # killed after 30 seconds
Continue Reading

Blooming Ruby

Prototypical BitArray and BloomFilter library

[ruby]

Yet another Ruby bloom filter, and a new bit array implementation, including a speedy JRuby-specific version.

This was extracted from an experiment in leaked password list processing, before I moved on to Golomb Compressed Sets, and might be worth extracting into a proper gem at some point.

Continue Reading

pkg-cruft

Help keep FreeBSD tidy

[ruby] [bsd]

A small Ruby script for helping deal with cruft on pkgng-based systems like FreeBSD.

Perhaps most interesting is the implementation of checkrestart, a feature similar to the Debian-goodies program of the same name, which can find running processes which may need restarting following an upgrade.

# pkg-cruft checkrestart
[MISSING EXECUTABLE] (tmux-2.7)? running as 17319 (tmux)
[MISSING EXECUTABLE] (zsh-5.5.1)? running as 20115 (zsh)
[MISSING EXECUTABLE] (weechat-2.2)? running as 36747 (weechat)
/usr/local/bin/mosh-server (mosh-1.3.2_4) running as 53815 (mosh-server)
Continue Reading

Reattempt

A jittery Enumerable retry and backoff library

[ruby]

Reattempt is a simple application of Ruby’s Enumerators to provide a nice idiomatic interface to retries. At its most basic:

begin
  Reattempt::Retry.new.each do
    poke_remote_api
  end
rescue Reattempt::RetriesExceeded => e
  handle_repeated_failure(e.cause)
end
Continue Reading

rtss

Relative Timestamps for Stuff

[rust]

rtss annotates its output with relative durations between consecutive lines and since program start.

Inspired by Kevin Burke’s Golang tss, I thought it would be a fun exercise and a nice bit of Rust practice to implement something similar.

I’m fairly pleased with the result - it’s considerably faster, running at hundreds of MB per second in my tests - and somewhat more featureful, including pty support, allowing it to work similarly to expect’s unbuffer command.

Continue Reading

gcstool / ruby-gcs

Efficient set membership with Golomb Compressed Sets

[rust] [ruby]

Similar to Bloom Filters, Golomb Compressed sets allow for space-efficient probablistic storage of sets. In other words, you can ask a GCS if it’s seen an object, and retrieve either “absolutely not” or “probably not” in response.

gcstool was my first Rust project, developed primarily to play about with the haveibeenpwned.com pwned-passwords-2.0.txt database. It can store all half a billion items with a false-positive rate of 1-in-50 million in just 1.6GB, importing them in just a few minutes, though with fairly high memory requirements.

Continue Reading

run-one

Friendly wrapper around lockf/flock, unofficial clone of an Ubuntu package

[shell]

Wanting to hone my shell scripting skills a bit, I ended up reimplementing a cute script by Dustin Kirkland.

run-one is basically a thin wrapper around lockf(1), or on Linux flock(1):

-% lockf /var/tmp/sleep-5.lock sleep 5
# <runs sleep 5 under the named lock>
-% run-one sleep 5
# <runs sleep 5 under a lock named by a SHA256 of its arguments>
Continue Reading

FlashFind

High performance multithreaded Find redux

[ruby]

Following the development of FastFind, I wanted to make something with a more pleasing API and better internal design.

The result is interesting, but still in need of work before I’d consider it for production use.

Continue Reading

FastFind

High performance multithreaded Find

[ruby]

FastFind is a drop-in replacement for the standard library Find package.

I wrote this for FreshBSD, to speed up scanning CVS repositories - walking 300,000 files takes a while, especially with an API that basically forces you to File.stat twice for each one.

Continue Reading

💾 Scandisk

Scan.co.uk Disk Price List

[ruby] [web]

Sick of Scan.co.uk being a pain in the ass to browse for this sort of thing, I wrote a scraper and frontend to browse available storage devices and their prices.

Unauthorised, unofficial, and unsupported.

🖩 Bloom Filter Calculator 🖩

It has, like, graphs, and stuff.

[php] [javascript] [web]

Bloom filters are one of my favourite data structures, in no small part because of just how simple they are.

I first encountered them when I worked at Newzbin, when we needed a way of quickly determining if we’d seen a Usenet Message-ID before.

Their simplicity meant we were very quickly able to go from concept to a stable general-purpose microservice which served us well for many years.

One of the trickier aspects is knowing how to size one - typically you know how many items and what sort of false-positive rate you can stand, but how do you translate that to how many bits you need and how many hash functions to use?

Well, bam. The latest iteration of my calculator can calculate almost any valid set of parameters, and give you pretty graphs in real time, while offering server-side fallback. I hope you find it useful.

pqsort

Fast partial quicksort

[c]

pqsort is the sorting library I wrote for Newzbin’s custom search engine - it’s a lightly modified quicksort capable of partitioning and sorting only part of a list.

Quicksort lends itself very well to this kind of use due to how it works - it naturally partitions the list based on a pivot, and so it’s trivial to simply ignore any pivot side which cannot contain the desired results.

Continue Reading

k8temp

Command-line AMD K8 CPU thermal diode reader

[c] [bsd]

A foray into lower-level code, bashing PCI registers and using a bit of custom assembler, k8temp offers a command line interface to the thermal sensors on AMD’s old K8 CPUs.

Continue Reading

FreshBSD

BSD Commit Log Search

[ruby] [bsd] [web]

A search engine/viewer for source code commits, focused on projects of interest to the BSD community.

Currently into its fourth rewrite, backed by Elasticsearch, PostgreSQL and Redis, with the site itself running on JRuby and Roda, making use of concurrent-ruby to parallelise data retrieval and inserts.

Previous versions have used Rails, Sinatra, Padrino, Solr, MySQL, and memcached.

php-serialize

PHP's serialize()/unserialize() in Ruby

[ruby] [php]

Many, many years ago, I wrote this for unclear reasons. I don’t think I actually used it for anything. It implements the PHP serialize() and unserialize() functions in Ruby, allowing you to read/write PHP sessions and otherwise share objects between the two.

Now the various forks of it have nearly half a million downloads between them. php-serialize is the “official” version, with nearly half of those.

The exact incept date is uncertain, the earliest mention of it a quick search finds is ruby-talk:73669, 2003-06-16.