function* flat(iterable, depth) {
for (const item of iterable)
if (non_string_iterable(item) && depth > 1)
yield* flat(item, depth-1);
else
yield item;
} // flat
Jez Higgins
Freelance software generalist
software created
extended or repaired
Older posts are available in the archive or through tags.
Monday 10 December 2018 The Forest Road Reader, No 2.11: A Letter To Tharg
A couple of weeks ago, I wrote a letter to Tharg. Sadly, The Mighty One has not deemed it worthy to print in his illustrious organ so I humbly present it here.
Dear Mighty One,
As I stooped to gather Prog 2109 from my doormat last Saturday morning, I noticed it wasn’t contained in its usual Thrill-restraining plastic wrap. It was accompanied by its address label though, upon which I saw the handwritten note
Got posted with ours in Blenheim Road
I can only assume that two copies had, perhaps due to the high level of thrills therein, adhered to each other so strongly that the best efforts of the Post Office could not prise them apart, leading to my copy being inadvertently delivered to the wrong address.
I would beg Tharg give me the opportunity to publicly salute and thank my unknown fellow Squaxx who, on discovering this error, immediately hot-footed it to my house - not a huge distance, but not exactly next door - to deliver the misplaced prog before I had even noticed its absence. I would ask any graphic novel I might receive for printing this letter to instead be forwarded to them, my mysterious ally on Blenheim Road in Moseley, Birmingham.
Florix Grabundae,
Jez Higgins
Tagged 2000ad, and comics
Thursday 22 November 2018 The Forest Road Reader, No 2.10: Recreational Flattening
Spent a few minutes an evening or so back adding another couple of methods to Rillet, my JavaScript streams library. Sometimes it’s nice just to nurk around with code for no particular purpose.
The two new methods are flat and flatMap, and they have the same behaviour as the flatMap and flat Array methods currently going through the JavaScript standardisation process. The proposal’s currently at stage 3, which means they’re almost certain to go into the ES2019 standard.
-
Range.prototype.flat([depth])flattens any iterables in the sequence, down to the depth given. The depth is optional and if omitted or is less that 1, it defaults to 1. -
Range.prototype.flatMap(fn)appliesfnto each item in the sequence, then flattens the result into a new sequence
Rillet already had flatten() which is, essentially, flat(∞), which is good because that’s not a method call you can actually make. flatMap(fn) is equivalent to map(fn).flat(1), but in this modern age clearer and, not uncoincidentally, a little more efficient in both time and space too.
One of the pleasing aspects of working on this code is that the implementations of these two methods are really straightforward. Here’s flat -
Because we’re dealing with iterables, and so are lazily evaluating and only have to deal with the particular item at hand, we can lean on the magic of generators and yield and yield* to handle the book keeping. All we need to worry about is answering the does this one thing need flattening question.
Compare this with implementation for the proposed Array.prototype.flat. It has to exhaustively work through the array, examining every element within it, possibly flattening each one, examining elements within those, and so on and so on, until we’ve hit the desired depth limit. Here’s an (incomplete) example, lightly adapted from the core.js polyfill package.
function flat(depthArg = 1) {
const O = toObject(this);
const sourceLen = toLength(O.length);
const A = arraySpeciesCreate(O, 0);
A.length = flattenIntoArray(A, O, O, sourceLen, 0, depthArg);
return A;
}
function flattenIntoArray (target, original, source, sourceLen, start, depth) {
let targetIndex = start;
let sourceIndex = 0;
while (sourceIndex < sourceLen) {
if (sourceIndex in source) {
const element = source[sourceIndex];
if (depth > 0 && isArray(element)) {
targetIndex = flattenIntoArray(target, original, element, toLength(element.length), targetIndex, depth - 1) - 1;
} else {
target[targetIndex] = element;
}
targetIndex++;
}
sourceIndex++;
}
return targetIndex;
}This isn’t in anyway to suggest that the core.js code is at all bad. It isn’t. It’s simply the case that flat for Arrays is going to be wordier that for streams. The heart of our two implementations is essentially the same - if this thing needs flattening then recurse, otherwise use it - which we can express as a single if/else. All the rest of the Array version, ie about 75% of it, is managing state that yield manages for us in the Rillet generator. Sequences are great, I’ve been saying so for years 🙂
Tagged javascript, rillet, and code
Tuesday 13 November 2018 The Forest Road Reader, No 2.9 : Snakes Into Snake Oil
The ACCU Conference is one of the key events in my work year. It’s been an important part of my professional life for (counts on fingers, remembers when first child was born) nearly twenty years - for the things I’ve learned there, the conversations I’ve had, and the people I’ve met and who have become friends. I’ve had the privilege of giving sessions at the conference several times too - on countback I was surprised to discover it was eight times - and those occasions have been important for me too.
When this year’s Call for Papers opened, I didn’t know if I had anything to submit. This runs slightly counter to my conviction that everybody has something they can talk about, so I just left the knowledge that the Cfp was running to sit for a bit. One year, I didn’t even have an idea for a proposal until about ten minutes before the cutoff. Last year, something came to me during a Twitter conversation and Chris and I had nearly two hours to thrash out the details.
In truth, I did already have an idea but I was, frankly, extremely wary of having my name associated on the internet with the words blockchain and cryptocurrency. Eventually I resigned myself to it - I’m a big grown up boy, I can handle it - and submitted the following -
Snakes Into Snake Oil - What Blockchains Are And Why They’re Terrible
Since Bitcoin first rose to public awareness in the early 2010s,cryptocurrency enthusiasts have predicted a new world order. Central banks would fall away to be replaced with a consensus currency arising almost magically out of a worldwide network of independent computers, each transaction indelibly carved into the blockchain. The blockchain, immutable and permanent, stretching back in time to the genesis block, and growing, every few minutes, one block at a time, is, they say, the answer to all our economic ills.
And not just our economic ills: blockchains, perhaps running some kind of smart contract have been proposed as solutions for insurance markets, music distribution, land registries, voting, distributed file archiving, provenance of artworks and antiques, domain name resolution, human resources records, cross-border customs clearance, and more.
Snakes
In this session, we’ll have a look at what a blockchain is - how they’re implemented, and why they can indeed claim to be immutable. We’ll examine different consensus mechanisms, and how they allow new blocks to be formed without a central authority. That will lead into an overview of transaction mechanisms, and smart contracts. We might even write and a deploy a little smart contract of our own.
Into Snake Oil
Alongside establishing a baseline understanding of what blockchains are, we’ll also be looking at why they’re terrible.
The distributed nature of public blockchains purports to allow us to trust data produced by unknown and, indeed, unknowable third parties. This may not be the case, and if it isn’t you might never know. Blockchains are permanent and immutable, but is this feature or misfeature? While the ideas behind blockchains are all frightfully clever, is a blockchain basically a database with slow reads, really slow writes, and generally awful data throughput? Are they, in fact, ill-suited for many of the applications they are pitched to solve? And if you thought multi-threaded programming was hard, that’s just peanuts compared to smart contracts. Maybe we’ll also get a bit existential and consider whether a blockchain can die, and what happens if it does.
After all that doom and gloom we’ll try to end on a small positive note, with a brief look at a project which I believe is a good fit for a blockchain solution, and which might even make the world a slightly better place.
If the conference committee smile on me, I’ll be in Bristol in April to present it. If not, I’ll be in Bristol in April anyway.
Tagged blockchain, and accu-conference
Friday 09 November 2018 Talk: The Very Slow Time Machine
ARCHANGEL is a joint project involving University of Surrey, The National Archive, and The Open Data Institute, investigating how we might help ensure the long-term integrity of digital documents stored in public archives.
When an archive produces a physical artefact, its relatively easy to establish that it is indeed the original document, preserved unaltered since it was first deposited.
But a digital artefact?
A digital document can be infinitely copied without degradation, but can also be undetectably altered, inadvertently or deliberately, both with benign or malign intent.
How can we be confident that what we've presented with is, in fact, identical to the document that was first stored in archive?
ARCHANGEL is trying to address this problem, and in this talk I'll describe some of the approaches and technologies we're using.
Spoilers: Yes, it includes blockchains, but it's about the only blockchain application you'll hear of that doesn't immediately make you feel dirty. It might also include machine learning, but it's machine learning for justice.
I presented this session, which features a primer on archival practice, a bluffer's guide to blockchains, and a brief introduction to machine learning, last night at Nor(Dev), and I think it went pretty well. I certainly enjoyed presenting it. It was a pleasure to be there as the opening act for my friend Russel, who spoke about Me-TV, an honest to god broadcast TV client, and his journey picking up what was a dead project written in C and using the Xine video player to its current incarnation written in Rust using GStreamer. It was really good and even though Russel only showed a small amount of Rust code, it made me think about it very differently to the various introductions to that I've read. Real code always wins over toy examples.
- Slides with notes - it's a reveal.js deck, so press 'S' to get the speaker notes without which the slides will make no sense at all.
DVD Extras
-
Challenges in using blockchains to build trust in digital archiving - The ODI blog
-
ARCHANGEL: Trusted Archives of Digital Public Documents - Paper presented at ACM Document Engineering 2018.
-
ARCHANGEL GitHub Repositories - Smart contract, JavaScript and Ruby language bindings, Ethereum node support
-
Looking for Life on a Flat Earth - The astonishing New Yorker article I mention at the start of the talk
Tagged blockchain, archangel, machine learning, talk, and nordev
Wednesday 07 November 2018 The Forest Road Reader, No 2.8 : Elsewhere
In the course of what I laughingly call my career I’ve written many thousands of 'work words'. Not just the day-to-day emails or in whatever this year’s favourite chat messenger is, but formal 'documents' too. Design documents, the odd bit of policy, API descriptions, integration guides, all that kind of stuff. It’s all been for internal use, not for public consumption.
Well, the world is changing in so many ways, and somehow I managed to get my name on two articles, for two different clients, published on their respective websites two days apart.
As part of our work on the ARCHANGEL project, we - The ODI - have been looking at how blockchain technology could have a place in ensuring that our public archives are trusted, unaltered, and auditable. It’s a really interesting and, given the times we live in, potentially very important question.
In truth this article was written by my colleague Jared and my input was largely advisory, but I’m grateful to him for putting my name on it too. Technical Associate! Get me!
A great deal of the work we - West Midlands Fire Service - is built on Tymly, our state machine engine. Tymly’s state machines are defined using the Amazon State Language, and as we write more and more state machines, and they’re getting increasingly complex we thought, maybe, some validation might be useful. This little blog post introduces a couple of tools, why we ported them from Amazon’s own validation tools, and how the open source nature of what we and they are doing helped us all. It’s a bit of puff piece and I should probably write about that work in a bit more detail. There’s some untapped potential in some of the code I ported, I think.
I’m lucky to be working on both these. They’re interesting things to get stuck into, the people I work with are lovely, and the results will, hopefully, make the world a better place. You don’t always get to say that.
Tagged blockchain, archangel, javascript, ASL, validation, and porting